Page MenuHomeSoftware Heritage

Add a flag to not enable origins listed by a lister
ClosedPublic

Authored by olasd on Dec 5 2022, 2:22 PM.

Details

Summary

This cuts down one more manual step in the add forge now validation
process: we can add the relevant origins to the staging scheduler
without enabling them at all.

Test Plan

new test added

Diff Detail

Repository
rDLS Listers
Branch
detached-head
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 33077
Build 51848: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 51847: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D8916 (id=32124)

Could not rebase; Attempt merge onto f4aafe026b...

Updating f4aafe0..51c573b
Fast-forward
 swh/lister/pattern.py            | 31 +++++++++++++++-
 swh/lister/rubygems/lister.py    | 18 +++++++++-
 swh/lister/tests/test_pattern.py | 77 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 124 insertions(+), 2 deletions(-)
Changes applied before test
commit 51c573b5d319c365e5818872e19abc3eda80cbb7
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Dec 5 14:20:31 2022 +0100

    Add a flag to not enable origins listed by a lister
    
    This cuts down one more manual step in the add forge now validation
    process: we can add the relevant origins to the staging scheduler
    without enabling them at all.

commit 851b8e180e3841eebc5f1440b92add24f816d051
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Dec 5 14:20:19 2022 +0100

    Add built-in page and origin count limit to listers
    
    This will allow more automation of the staging add forge now process:
    for known-good listers, we can limit the number of origins being
    processed and reduce the amount of manual steps taken for each instance.

commit a66e24bfa2af7de8cacf4c4525f0eb5103aae0eb
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Dec 5 13:43:32 2022 +0100

    Ignore psqlrc when loading the rubygems database dump
    
    The SQL dump contains ownership instructions that can't be run if you
    don't have the right users in your database clusters. When someone has a
    psqlrc with ON_ERROR_STOP, this fails the load of the dump.
    
    Use the opportunity to trigger an exception when psql returns a non-zero
    exit code, rather than continue with an empty/inconsistent database.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/870/ for more details.

olasd requested review of this revision.Dec 5 2022, 2:27 PM
This revision is now accepted and ready to land.Dec 5 2022, 2:32 PM

Build is green

Patch application report for D8916 (id=32127)

Could not rebase; Attempt merge onto f4aafe026b...

Updating f4aafe0..64267f8
Fast-forward
 swh/lister/pattern.py            |  31 +++++++++++-
 swh/lister/rubygems/lister.py    |  18 ++++++-
 swh/lister/tests/test_pattern.py | 101 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 148 insertions(+), 2 deletions(-)
Changes applied before test
commit 64267f8f50a801302cde9aadd20dc8036a382d1f
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Dec 5 14:20:31 2022 +0100

    Add a flag to not enable origins listed by a lister
    
    This cuts down one more manual step in the add forge now validation
    process: we can add the relevant origins to the staging scheduler
    without enabling them at all.

commit b815737054acdbebf0d71036af3b713e866beed6
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Dec 5 14:20:19 2022 +0100

    Add built-in page and origin count limit to listers
    
    This will allow more automation of the staging add forge now process:
    for known-good listers, we can limit the number of origins being
    processed and reduce the amount of manual steps taken for each instance.

commit a66e24bfa2af7de8cacf4c4525f0eb5103aae0eb
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Dec 5 13:43:32 2022 +0100

    Ignore psqlrc when loading the rubygems database dump
    
    The SQL dump contains ownership instructions that can't be run if you
    don't have the right users in your database clusters. When someone has a
    psqlrc with ON_ERROR_STOP, this fails the load of the dump.
    
    Use the opportunity to trigger an exception when psql returns a non-zero
    exit code, rather than continue with an empty/inconsistent database.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/872/ for more details.