Page MenuHomeSoftware Heritage

dumb loader: stream pack files while downloading
ClosedPublic

Authored by vlorentz on Jan 10 2022, 5:49 PM.

Details

Summary

instead of writing them all at once, which partially defeats the point of
using a spooled buffer.

Depends on D6901.

Diff Detail

Repository
rDLDG Git loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D6902 (id=25022)

Could not rebase; Attempt merge onto 0cc96c25ab...

Updating 0cc96c2..b5f4837
Fast-forward
 swh/loader/git/dumb.py | 37 +++++++++++++------------------------
 1 file changed, 13 insertions(+), 24 deletions(-)
Changes applied before test
commit b5f4837a73773bec6c0adc57198ddf9436b88ce0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jan 10 17:49:13 2022 +0100

    dumb loader: stream pack files while downloading
    
    instead of writing them all at once, which partially negates the point of
    using a spooled buffer.

commit 2e43e1cafb5b0fdfdd6eb7601b13303cb9001a8d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jan 10 17:47:13 2022 +0100

    Remove unnecessary use of dulwich.client.HttpGitClient
    
    'requests' does the job just fine with less complexity.

commit c56f6126703aeaabd7b2a37e0960f39e5e7257d7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jan 10 17:01:34 2022 +0100

    Fix typing issue
    
    response.content_type is set by Dulwich, but isn't part of urllib3's
    HTTPResponse, so we shouldn't rely on it.
    (And it makes mypy complain when the 'types-urllib3' package is installed)

See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/165/ for more details.

Build is green

Patch application report for D6902 (id=25025)

Could not rebase; Attempt merge onto 0cc96c25ab...

Updating 0cc96c2..0f3a13e
Fast-forward
 swh/loader/git/dumb.py | 37 +++++++++++++------------------------
 1 file changed, 13 insertions(+), 24 deletions(-)
Changes applied before test
commit 0f3a13eb36155b4ae6c7806405c68832b55ef629
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jan 10 17:49:13 2022 +0100

    dumb loader: stream pack files while downloading
    
    instead of writing them all at once, which partially negates the point of
    using a spooled buffer.

commit 76739ece7af9313564dc5a1e7ffec10475da3e48
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jan 10 17:47:13 2022 +0100

    Remove unnecessary use of dulwich.client.HttpGitClient
    
    'requests' does the job just fine with less complexity.

commit d7481af6b33c3b7ddaad31f1801074d1525a7af2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jan 10 17:01:34 2022 +0100

    Fix typing issue
    
    response.content_type is set by Dulwich, but isn't part of urllib3's
    HTTPResponse, so we shouldn't rely on it.
    (And it makes mypy complain when the 'types-urllib3' package is installed)

See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/168/ for more details.

Build is green

Patch application report for D6902 (id=25027)

Could not rebase; Attempt merge onto 0cc96c25ab...

Updating 0cc96c2..4dfd2ed
Fast-forward
 swh/loader/git/dumb.py | 38 ++++++++++++++------------------------
 1 file changed, 14 insertions(+), 24 deletions(-)
Changes applied before test
commit 4dfd2ed12ffe2f94165527802a53c76c29f7a779
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jan 10 17:49:13 2022 +0100

    dumb loader: stream pack files while downloading
    
    instead of writing them all at once, which partially defeats the point of
    using a spooled buffer.

commit 705976f75bc30aa1451babe10b5449b91afce2bf
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jan 10 17:47:13 2022 +0100

    Remove unnecessary use of dulwich.client.HttpGitClient
    
    'requests' does the job just fine with less complexity.

commit d7481af6b33c3b7ddaad31f1801074d1525a7af2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jan 10 17:01:34 2022 +0100

    Fix typing issue
    
    response.content_type is set by Dulwich, but isn't part of urllib3's
    HTTPResponse, so we shouldn't rely on it.
    (And it makes mypy complain when the 'types-urllib3' package is installed)

See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/170/ for more details.

anlambert added a subscriber: anlambert.

Looks good to me.

This revision is now accepted and ready to land.Jan 10 2022, 6:07 PM