Page MenuHomeSoftware Heritage
Differential D5026

core: Mark visit status as failed when relevant
ClosedPublic
Actions

Authored by ardumont on Feb 5 2021, 1:00 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Maniphest Tasks
T3030: Improve loaders to deal with new visit status events
Commits
rDLDBASEfb7ee4332b0d: core: Mark visit status as failed when relevant
Summary

When a visit ends up with no snapshot within the exception block, mark the visit as
failed. Otherwise, let it be a partial visit.

Related to T3030

Diff Detail

Event Timeline

ardumont created this revision.Feb 5 2021, 1:00 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 5 2021, 1:00 PM
swh-public-ci added a comment.Feb 5 2021, 1:02 PM

Build is green

Patch application report for D5026 (id=17915)

Could not rebase; Attempt merge onto b332f4acd1...

Updating b332f4a..e8eef27
Fast-forward
 swh/loader/core/loader.py                  |  6 ++++--
 swh/loader/core/tests/test_loader.py       | 31 +++++++++++++++++++----------
 swh/loader/package/loader.py               |  9 +++++----
 swh/loader/package/npm/tests/test_npm.py   |  2 +-
 swh/loader/package/pypi/tests/test_pypi.py | 32 +++++++++++++++++++++++++++++-
 5 files changed, 62 insertions(+), 18 deletions(-)
Changes applied before test
commit e8eef2752226f9e6593c95d7589389dc7a35c8fe
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Feb 5 12:59:29 2021 +0100

    core: Mark visit status as failed when relevant
    
    When a visit ends up with no snapshot within the exception block, mark the visit as
    failed. Otherwise, let it be a partial visit.
    
    Related to T3030

commit c6a4f5f6c6fff953902fa097c75b57ebd8aac397
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Feb 5 12:25:12 2021 +0100

    package: Make visit status 'failed' when relevant
    
    When:
    - failure to communicate internally with the storage
    - absolutely no revision got loaded during a visit
    
    Related to T3030

commit 265d3b2b2ac6047cee40e26b145be500a9749a41
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Feb 5 11:41:41 2021 +0100

    loader: Make loader write the origin_visit_status' type
    
    Related to T3030

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/344/ for more details.

Harbormaster completed remote builds in B19034: Diff 17915.Feb 5 2021, 1:02 PM
ardumont requested review of this revision.Feb 5 2021, 1:02 PM
ardumont updated this revision to Diff 17937.Feb 5 2021, 3:33 PM

Rebase

swh-public-ci added a comment.Feb 5 2021, 3:36 PM

Build is green

Patch application report for D5026 (id=17937)

Could not rebase; Attempt merge onto 65a513578d...

Updating 65a5135..3d03374
Fast-forward
 swh/loader/core/loader.py                  |  4 +--
 swh/loader/core/tests/test_loader.py       | 31 +++++++++++------
 swh/loader/package/cran/tests/test_cran.py | 53 +++++++++++++++++++++++++++++-
 swh/loader/package/loader.py               | 14 ++++----
 swh/loader/package/npm/tests/test_npm.py   |  4 +--
 swh/loader/package/pypi/tests/test_pypi.py | 34 +++++++++++++++++--
 6 files changed, 116 insertions(+), 24 deletions(-)
Changes applied before test
commit 3d0337489b3e49d4ab6b19bd92e9eceacd946344
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Feb 5 12:59:29 2021 +0100

    core: Mark visit status as failed when relevant
    
    When a visit ends up with no snapshot within the exception block, mark the visit as
    failed. Otherwise, let it be a partial visit.
    
    Related to T3030

commit a77e748c184349c3ab3f4d674a6edb3f9f187e63
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Feb 5 12:25:12 2021 +0100

    package: Make visit status 'failed' when relevant
    
    When:
    - failure to communicate internally with the storage
    - absolutely no revision got loaded during a visit
    
    Related to T3030

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/348/ for more details.

Harbormaster completed remote builds in B19056: Diff 17937.Feb 5 2021, 3:36 PM
ardumont updated this revision to Diff 17945.Feb 5 2021, 5:46 PM

Rebase

swh-public-ci added a comment.Feb 5 2021, 5:49 PM

Build is green

Patch application report for D5026 (id=17945)

Could not rebase; Attempt merge onto 65a513578d...

Updating 65a5135..e2ea937
Fast-forward
 swh/loader/core/loader.py                  |  4 +--
 swh/loader/core/tests/test_loader.py       | 31 +++++++++++------
 swh/loader/package/cran/tests/test_cran.py | 53 +++++++++++++++++++++++++++++-
 swh/loader/package/loader.py               | 14 ++++----
 swh/loader/package/npm/tests/test_npm.py   |  4 +--
 swh/loader/package/pypi/tests/test_pypi.py | 34 +++++++++++++++++--
 6 files changed, 116 insertions(+), 24 deletions(-)
Changes applied before test
commit e2ea9373b0f554f9eb2f25d5a9869aa98b447b1f
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Feb 5 12:59:29 2021 +0100

    core: Mark visit status as failed when relevant
    
    When a visit ends up with no snapshot within the exception block, mark the visit as
    failed. Otherwise, let it be a partial visit.
    
    Related to T3030

commit 3b4a023468714a23f42eba4ef4e848ca209cc6b0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Feb 5 12:25:12 2021 +0100

    package: Mark visit status as failed when relevant
    
    When:
    - failure to communicate internally with the storage
    - absolutely no revision got loaded during a visit
    
    Related to T3030

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/350/ for more details.

Harbormaster completed remote builds in B19065: Diff 17945.Feb 5 2021, 5:49 PM
ardumont updated this revision to Diff 17948.Feb 5 2021, 7:17 PM

Rebase

swh-public-ci added a comment.Feb 5 2021, 7:20 PM

Build is green

Patch application report for D5026 (id=17948)

Could not rebase; Attempt merge onto 65a513578d...

Updating 65a5135..0673a2e
Fast-forward
 swh/loader/core/loader.py                  |  4 +--
 swh/loader/core/tests/test_loader.py       | 31 +++++++++++------
 swh/loader/package/cran/tests/test_cran.py | 53 +++++++++++++++++++++++++++++-
 swh/loader/package/loader.py               | 10 +++---
 swh/loader/package/npm/tests/test_npm.py   |  4 +--
 swh/loader/package/pypi/tests/test_pypi.py | 34 +++++++++++++++++--
 6 files changed, 114 insertions(+), 22 deletions(-)
Changes applied before test
commit 0673a2eb060c81a0293f8bc0548f5c8e3cde90c3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Feb 5 12:59:29 2021 +0100

    core: Mark visit status as failed when relevant
    
    When a visit ends up with no snapshot within the exception block, mark the visit as
    failed. Otherwise, let it be a partial visit.
    
    Related to T3030

commit 4500063a2fc5467fdd3ee2c0da25fa70a1fd919a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Feb 5 12:25:12 2021 +0100

    package: Mark visit status as failed when relevant
    
    When:
    - failure to communicate internally with the storage
    - absolutely no revision got loaded during a visit
    
    Related to T3030

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/353/ for more details.

Harbormaster completed remote builds in B19069: Diff 17948.Feb 5 2021, 7:20 PM
vlorentz added a subscriber: vlorentz.Feb 8 2021, 12:40 PM
vlorentz added inline comments.
swh/loader/core/loader.py
337–338

or failed

ardumont updated this revision to Diff 17968.Feb 8 2021, 2:06 PM

Adapt according to review, this adapts the exception log with the right status.

swh-public-ci added a comment.Feb 8 2021, 2:08 PM

Build is green

Patch application report for D5026 (id=17968)

Rebasing onto 4500063a2f...

Current branch diff-target is up to date.
Changes applied before test
commit fb7ee4332b0dc7889d950309dec9a4ecf57453b5
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Feb 5 12:59:29 2021 +0100

    core: Mark visit status as failed when relevant
    
    When a visit ends up with no snapshot within the exception block, mark the visit as
    failed. Otherwise, let it be a partial visit.
    
    Related to T3030

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/358/ for more details.

Harbormaster completed remote builds in B19089: Diff 17968.Feb 8 2021, 2:09 PM
ardumont added a child revision: D5035: package: Mark visit as not_found when relevant.Feb 11 2021, 8:59 AM
douardda accepted this revision.Feb 11 2021, 10:04 AM
douardda added a subscriber: douardda.

lgtm

This revision is now accepted and ready to land.Feb 11 2021, 10:04 AM
Closed by commit rDLDBASEfb7ee4332b0d: core: Mark visit status as failed when relevant (authored by ardumont). · Explain WhyFeb 11 2021, 10:50 AM
This revision was automatically updated to reflect the committed changes.
ardumont marked an inline comment as done.
ardumont added a commit: rDLDBASEfb7ee4332b0d: core: Mark visit status as failed when relevant.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
loader/
core/
8 lines80%
Loading...
tests/
31 lines97%
Loading...

Diff 17968

View Options

swh/loader/core/loader.py

Loading...
View Options

swh/loader/core/tests/test_loader.py

Loading...
About · Developer information · Legal