Page MenuHomeSoftware Heritage

package/loader: Handle errors when retrieving package version info
ClosedPublic

Authored by anlambert on Sep 19 2022, 4:45 PM.

Details

Summary

Some package loaders might encounter errors while attempting to get
package info for a given version (HTTP error for instance).

So handle that case and ensure partial visit when such an error occurs.

Related to T4124

Depends on D8504

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Branch
handle-package-info-errors
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 31616
Build 49447: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 49446: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D8505 (id=30628)

Could not rebase; Attempt merge onto 0c599bdc81...

Updating 0c599bd..47c931d
Fast-forward
 swh/loader/package/loader.py                       |  61 +++++++---
 .../package_example_example-v1.0.tar.gz            | Bin 0 -> 181 bytes
 .../package_example_example-v2.0.tar.gz            | Bin 0 -> 181 bytes
 .../package_example_example-v3.0.tar.gz            | Bin 0 -> 181 bytes
 .../package_example_example-v4.0.tar.gz            | Bin 0 -> 182 bytes
 swh/loader/package/tests/test_loader.py            | 134 ++++++++++++++++++---
 6 files changed, 157 insertions(+), 38 deletions(-)
 create mode 100644 swh/loader/package/tests/data/https_example.org/package_example_example-v1.0.tar.gz
 create mode 100644 swh/loader/package/tests/data/https_example.org/package_example_example-v2.0.tar.gz
 create mode 100644 swh/loader/package/tests/data/https_example.org/package_example_example-v3.0.tar.gz
 create mode 100644 swh/loader/package/tests/data/https_example.org/package_example_example-v4.0.tar.gz
Changes applied before test
commit 47c931d1d6061353159d148791ef2a0d346ebbf8
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Sep 16 15:19:29 2022 +0200

    package/loader: Handle errors when retrieving package version info
    
    Some package loaders might encounter errors while attempting to get
    package info for a given version (HTTP error for instance).
    
    So handle that case and ensure partial visit when such an error occurs.
    
    Related to T4124

commit e12ca02f894e5769e576d3b799a7c1bd1645a695
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Sep 19 16:26:32 2022 +0200

    package/loader: Implement load_status and visit_status
    
    Those methods were not reimplemented and could return incorrect statuses
    once the load method called.
    
    As they are useful for tests, implement them in PackageLoader class.

commit 99e56d3f8746cf38c90e3e658e8937018c08a57a
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Sep 19 15:29:12 2022 +0200

    package/tests: Add a visit success test for StubPackageLoader
    
    Such test was missing so add a couple of tiny tarballs in tests
    data directory and simulate a successful visit for origin example.
    
    Also update origin URL and remove a couple of hardcoded litterals.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/878/ for more details.

This revision is now accepted and ready to land.Sep 20 2022, 11:37 AM

Build is green

Patch application report for D8505 (id=30637)

Could not rebase; Attempt merge onto c4370009d0...

Updating c437000..d471cca
Fast-forward
 swh/loader/package/loader.py                       |  61 +++++++---
 .../package_example_example-v1.0.tar.gz            | Bin 0 -> 181 bytes
 .../package_example_example-v2.0.tar.gz            | Bin 0 -> 181 bytes
 .../package_example_example-v3.0.tar.gz            | Bin 0 -> 181 bytes
 .../package_example_example-v4.0.tar.gz            | Bin 0 -> 182 bytes
 swh/loader/package/tests/test_loader.py            | 134 ++++++++++++++++++---
 6 files changed, 157 insertions(+), 38 deletions(-)
 create mode 100644 swh/loader/package/tests/data/https_example.org/package_example_example-v1.0.tar.gz
 create mode 100644 swh/loader/package/tests/data/https_example.org/package_example_example-v2.0.tar.gz
 create mode 100644 swh/loader/package/tests/data/https_example.org/package_example_example-v3.0.tar.gz
 create mode 100644 swh/loader/package/tests/data/https_example.org/package_example_example-v4.0.tar.gz
Changes applied before test
commit d471cca9f2a32b9e5ac2f0a752e64b593d6dfd8f
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Sep 16 15:19:29 2022 +0200

    package/loader: Handle errors when retrieving package version info
    
    Some package loaders might encounter errors while attempting to get
    package info for a given version (HTTP error for instance).
    
    So handle that case and ensure partial visit when such an error occurs.
    
    Related to T4124

commit 598873f00392e75f085aa0b9e957b7261127aaa2
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Sep 19 16:26:32 2022 +0200

    package/loader: Implement load_status and visit_status
    
    Those methods were not reimplemented and could return incorrect statuses
    once the load method called.
    
    As they are useful for tests, implement them in PackageLoader class.

commit 63ccbd9a3a671a4b218529be7d12d2fb1542b5cd
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Sep 19 15:29:12 2022 +0200

    package/tests: Add a visit success test for StubPackageLoader
    
    Such test was missing so add a couple of tiny tarballs in tests
    data directory and simulate a successful visit for origin example.
    
    Also update origin URL and remove a couple of hardcoded litterals.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/882/ for more details.