Differential D6377
wip: git: Group objects per type early to drop the packfile reference asap Authored by ardumont on Sep 30 2021, 9:27 AM.
Details
Prior to this commit, the loader git would keep the packfile reference and iterate over This should reduce the memory pressure on loading very large repository. Assuming the Related to T3625 tox
it does not. That kills the process almost immediately [1] [1] P1185
Diff Detail
Event TimelineComment Actions Build is green Patch application report for D6377 (id=23205)Rebasing onto 368674744c... Current branch diff-target is up to date. Changes applied before testcommit f17ccb5a810583c6d520b0da7ef7b644d4f88381
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Thu Sep 30 09:13:19 2021 +0200
Group objects per type early to drop the packfile reference asap
Prior to this commit, the loader git would keep the packfile reference and iterate over
it multiple times per object type. In this commit, we try to drop the packfile reference
earlier to release that reference as soon as possible.
This should reduce the memory pressure on loading very large repository. Assuming the
python dict takes less space than the packfile.See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/124/ for more details. Comment Actions Build is green Patch application report for D6377 (id=23206)Rebasing onto 368674744c... Current branch diff-target is up to date. Changes applied before testcommit 1f5fddb0e86162da1cc77b739ff4270fd4c2de51
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Thu Sep 30 09:13:19 2021 +0200
Group objects per type early to drop the packfile reference asap
Prior to this commit, the loader git would keep the packfile reference and iterate over
it multiple times per object type. In this commit, we try to drop the packfile reference
earlier to release that reference as soon as possible.
This should reduce the memory pressure on loading very large repository. Assuming the
python dict takes less space than the packfile.See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/125/ for more details. Comment Actions I truly doubt that proceeding like this will optimize the memory consumption of the loader as objects in a pack file are gzip compressed and usually deltified to optimize size. Your approach will store all git objects uncompressed in a dict so few chances that it will eat less memory than the pack file here, nevertheless it will optimize loading performance. Comment Actions
yes, I agree. That does reduce the i/o not the memory pressure. |