Page MenuHomeSoftware Heritage
Differential D6239

package/utils: Add FTP protocol support to download function
ClosedPublic
Actions

Authored by anlambert on Sep 10 2021, 4:07 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T2687: nixguix: Add support for downloads over FTP
Commits
rDLDBASEd5e54a5eea1e: package/utils: Add FTP protocol support to download function
Summary

Add support to download file using FTP protocol through the
use of the urllib.request.urlopen function from Python standard
library.

Related to T2687

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Branch
ftp-download-support
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 23508
Build 36673: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 36672: arc lint + arc unit

Event Timeline

anlambert created this revision.Sep 10 2021, 4:07 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptSep 10 2021, 4:07 PM
swh-public-ci added a comment.Sep 10 2021, 4:09 PM

Build is green

Patch application report for D6239 (id=22574)

Rebasing onto 50b062adc7...

Current branch diff-target is up to date.
Changes applied before test
commit 6f4d279e694cb37c9e0bd9cc7c7e82c51d2c64d1
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Sep 10 15:43:22 2021 +0200

    package/utils: Add FTP protocol support to download function
    
    Add support to download file using FTP protocol through the
    use of the urllib.request.urlopen function from Python standard
    library.
    
    Related to T2687

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/534/ for more details.

Harbormaster completed remote builds in B23508: Diff 22574.Sep 10 2021, 4:09 PM
anlambert requested review of this revision.Sep 10 2021, 4:09 PM
vlorentz added a subscriber: vlorentz.Sep 10 2021, 4:14 PM
vlorentz added inline comments.
swh/loader/package/utils.py
85–87

What about something like this to keep it streaming? (the response will be closed when decrefed anyway)

anlambert added inline comments.Sep 10 2021, 4:30 PM
swh/loader/package/utils.py
85–87

Nice, just tested in docker and it works great, will update the diff then.

anlambert updated this revision to Diff 22576.Sep 10 2021, 4:31 PM

Update: also stream FTP responses.

olasd added a subscriber: olasd.Sep 10 2021, 4:33 PM
olasd added inline comments.
swh/loader/package/utils.py
85–87

We should call response.close() in both cases so the connection gets properly returned to the pool. We can do that after we're done streaming it (before checking the hashes).

olasd added inline comments.Sep 10 2021, 4:36 PM
swh/loader/package/utils.py
106

(add response.close() here)

anlambert updated this revision to Diff 22577.Sep 10 2021, 4:36 PM

Fix test

swh-public-ci added a comment.Sep 10 2021, 4:36 PM

Build was aborted

Patch application report for D6239 (id=22576)

Rebasing onto 50b062adc7...

Current branch diff-target is up to date.
Changes applied before test
commit def3e4cbe4549db48a954eb1034da20b8bdd416c
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Sep 10 15:43:22 2021 +0200

    package/utils: Add FTP protocol support to download function
    
    Add support to download file using FTP protocol through the
    use of the urllib.request.urlopen function from Python standard
    library.
    
    Related to T2687

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/535/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/535/console

Harbormaster failed remote builds in B23510: Diff 22576!Sep 10 2021, 4:36 PM
swh-public-ci added a comment.Sep 10 2021, 4:38 PM

Build is green

Patch application report for D6239 (id=22577)

Rebasing onto 50b062adc7...

Current branch diff-target is up to date.
Changes applied before test
commit 1266b2e8c49a2fee32ef348e888f1b8134e62f51
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Sep 10 15:43:22 2021 +0200

    package/utils: Add FTP protocol support to download function
    
    Add support to download file using FTP protocol through the
    use of the urllib.request.urlopen function from Python standard
    library.
    
    Related to T2687

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/536/ for more details.

Harbormaster completed remote builds in B23511: Diff 22577.Sep 10 2021, 4:38 PM
anlambert updated this revision to Diff 22578.Sep 10 2021, 4:39 PM

Update: explicitely close response once processed

swh-public-ci added a comment.Sep 10 2021, 4:41 PM

Build is green

Patch application report for D6239 (id=22578)

Rebasing onto 50b062adc7...

Current branch diff-target is up to date.
Changes applied before test
commit 067450ab3784d9ff0aba6a66a8738ba90c7b86c5
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Sep 10 15:43:22 2021 +0200

    package/utils: Add FTP protocol support to download function
    
    Add support to download file using FTP protocol through the
    use of the urllib.request.urlopen function from Python standard
    library.
    
    Related to T2687

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/537/ for more details.

Harbormaster completed remote builds in B23512: Diff 22578.Sep 10 2021, 4:41 PM
ardumont accepted this revision.Sep 14 2021, 10:14 AM
This revision is now accepted and ready to land.Sep 14 2021, 10:14 AM
anlambert updated this revision to Diff 22631.Sep 14 2021, 10:41 AM

Rebase

swh-public-ci added a comment.Sep 14 2021, 10:43 AM

Build is green

Patch application report for D6239 (id=22631)

Rebasing onto 0efaf7a0ef...

Current branch diff-target is up to date.
Changes applied before test
commit d5e54a5eea1e0d6c6fac7e87892b684a5f53d911
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Sep 10 15:43:22 2021 +0200

    package/utils: Add FTP protocol support to download function
    
    Add support to download file using FTP protocol through the
    use of the urllib.request.urlopen function from Python standard
    library.
    
    Related to T2687

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/545/ for more details.

Harbormaster completed remote builds in B23566: Diff 22631.Sep 14 2021, 10:43 AM
Closed by commit rDLDBASEd5e54a5eea1e: package/utils: Add FTP protocol support to download function (authored by anlambert). · Explain WhySep 14 2021, 10:45 AM
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDLDBASEd5e54a5eea1e: package/utils: Add FTP protocol support to download function.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
loader/
package/
tests/
41 lines100%
Loading...
18 lines100%
Loading...

Diff 22574

View Options

swh/loader/package/tests/test_utils.py

Loading...
View Options

swh/loader/package/utils.py

Loading...
About · Developer information · Legal