Page MenuHomeSoftware Heritage
Differential D3423

Introduce a get_listed_origins endpoint
ClosedPublic
Actions

Authored by olasd on Jul 6 2020, 9:52 AM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDSCH5b373ceb36e4: Introduce a get_listed_origins endpoint
Summary

This paginated endpoint allows retrieving information about the origins recorded
by listers.

Depends on D3422.

Test Plan

tox tests added

Diff Detail

Repository
rDSCH Scheduling utilities
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 13350
Build 20403: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 20402: arc lint + arc unit

Event Timeline

olasd created this revision.Jul 6 2020, 9:52 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptJul 6 2020, 9:52 AM
swh-public-ci added a comment.Jul 6 2020, 9:57 AM

Build is green

Patch application report for D3423 (id=12123)

Could not rebase; Attempt merge onto cc8fa7f4c0...

Updating cc8fa7f..014c775
Fast-forward
 swh/scheduler/api/serializers.py       |  2 +-
 swh/scheduler/backend.py               | 69 ++++++++++++++++++++++++++++++++--
 swh/scheduler/exc.py                   |  5 +++
 swh/scheduler/interface.py             | 24 +++++++++++-
 swh/scheduler/model.py                 | 33 +++++++++++++++-
 swh/scheduler/sql/60-swh-indexes.sql   |  3 ++
 swh/scheduler/tests/test_api_client.py |  1 +
 swh/scheduler/tests/test_scheduler.py  | 57 +++++++++++++++++++++++++++-
 8 files changed, 186 insertions(+), 8 deletions(-)
Changes applied before test
commit 014c775f337d13072554654d25f1b37fb0657650
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jul 6 09:49:44 2020 +0200

    Introduce a get_listed_origins endpoint
    
    This paginated endpoint allows retrieving information about the origins recorded
    by listers.

commit aefc5c97f80ae869447165c12620df5ff1775afc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jul 6 09:48:29 2020 +0200

    Don't recurse into attrs objects when serializing
    
    We need to use our serialization hook recursively to make sure that we can
    deserialize nested data structures.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/34/ for more details.

Harbormaster completed remote builds in B13344: Diff 12123.Jul 6 2020, 9:57 AM
olasd mentioned this in D3425: Introduce a simpler base pattern for lister implementations..Jul 6 2020, 10:39 AM
vlorentz accepted this revision.Jul 6 2020, 10:54 AM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/scheduler/backend.py
255–256

We don't usually limit page sizes in backends (only swh-web). Any specific reason to do it here?

This revision is now accepted and ready to land.Jul 6 2020, 10:54 AM
olasd added inline comments.Jul 6 2020, 10:55 AM
swh/scheduler/backend.py
255–256

Not really, no.

olasd updated this revision to Diff 12128.Jul 6 2020, 11:09 AM

Add db upgrade script

swh-public-ci added a comment.Jul 6 2020, 11:12 AM

Build is green

Patch application report for D3423 (id=12128)

Could not rebase; Attempt merge onto cc8fa7f4c0...

Updating cc8fa7f..5054d05
Fast-forward
 sql/updates/17.sql                     |  4 ++
 swh/scheduler/api/serializers.py       |  2 +-
 swh/scheduler/backend.py               | 69 ++++++++++++++++++++++++++++++++--
 swh/scheduler/exc.py                   |  5 +++
 swh/scheduler/interface.py             | 24 +++++++++++-
 swh/scheduler/model.py                 | 33 +++++++++++++++-
 swh/scheduler/sql/30-swh-schema.sql    |  2 +-
 swh/scheduler/sql/60-swh-indexes.sql   |  3 ++
 swh/scheduler/tests/test_api_client.py |  1 +
 swh/scheduler/tests/test_scheduler.py  | 57 +++++++++++++++++++++++++++-
 10 files changed, 191 insertions(+), 9 deletions(-)
 create mode 100644 sql/updates/17.sql
Changes applied before test
commit 5054d0516d44dce907ffdfc2a23ac1158314adfe
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jul 6 09:49:44 2020 +0200

    Introduce a get_listed_origins endpoint
    
    This paginated endpoint allows retrieving information about the origins recorded
    by listers.

commit aefc5c97f80ae869447165c12620df5ff1775afc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jul 6 09:48:29 2020 +0200

    Don't recurse into attrs objects when serializing
    
    We need to use our serialization hook recursively to make sure that we can
    deserialize nested data structures.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/35/ for more details.

Harbormaster completed remote builds in B13349: Diff 12128.Jul 6 2020, 11:12 AM
olasd updated this revision to Diff 12129.Jul 6 2020, 11:32 AM

Drop limits checking

swh-public-ci added a comment.Jul 6 2020, 11:34 AM

Build is green

Patch application report for D3423 (id=12129)

Could not rebase; Attempt merge onto cc8fa7f4c0...

Updating cc8fa7f..9d48fe4
Fast-forward
 sql/updates/17.sql                     |  4 +++
 swh/scheduler/api/serializers.py       |  2 +-
 swh/scheduler/backend.py               | 64 ++++++++++++++++++++++++++++++++--
 swh/scheduler/interface.py             | 24 ++++++++++++-
 swh/scheduler/model.py                 | 33 +++++++++++++++++-
 swh/scheduler/sql/30-swh-schema.sql    |  2 +-
 swh/scheduler/sql/60-swh-indexes.sql   |  3 ++
 swh/scheduler/tests/test_api_client.py |  1 +
 swh/scheduler/tests/test_scheduler.py  | 47 ++++++++++++++++++++++++-
 9 files changed, 173 insertions(+), 7 deletions(-)
 create mode 100644 sql/updates/17.sql
Changes applied before test
commit 9d48fe4e7e7dc653f32423902d8c34a475a5c60a
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jul 6 09:49:44 2020 +0200

    Introduce a get_listed_origins endpoint
    
    This paginated endpoint allows retrieving information about the origins recorded
    by listers.

commit aefc5c97f80ae869447165c12620df5ff1775afc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jul 6 09:48:29 2020 +0200

    Don't recurse into attrs objects when serializing
    
    We need to use our serialization hook recursively to make sure that we can
    deserialize nested data structures.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/36/ for more details.

Harbormaster completed remote builds in B13350: Diff 12129.Jul 6 2020, 11:34 AM
olasd updated this revision to Diff 12130.Jul 6 2020, 11:51 AM

Add test for get_listed_origins()

swh-public-ci added a comment.Jul 6 2020, 11:54 AM

Build is green

Patch application report for D3423 (id=12130)

Could not rebase; Attempt merge onto cc8fa7f4c0...

Updating cc8fa7f..5b373ce
Fast-forward
 sql/updates/17.sql                     |  4 +++
 swh/scheduler/api/serializers.py       |  2 +-
 swh/scheduler/backend.py               | 64 ++++++++++++++++++++++++++++++++--
 swh/scheduler/interface.py             | 24 ++++++++++++-
 swh/scheduler/model.py                 | 33 +++++++++++++++++-
 swh/scheduler/sql/30-swh-schema.sql    |  2 +-
 swh/scheduler/sql/60-swh-indexes.sql   |  3 ++
 swh/scheduler/tests/test_api_client.py |  1 +
 swh/scheduler/tests/test_scheduler.py  | 54 +++++++++++++++++++++++++++-
 9 files changed, 180 insertions(+), 7 deletions(-)
 create mode 100644 sql/updates/17.sql
Changes applied before test
commit 5b373ceb36e4796db3ac3f7cea5c7abcdc1c0a09
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jul 6 09:49:44 2020 +0200

    Introduce a get_listed_origins endpoint
    
    This paginated endpoint allows retrieving information about the origins recorded
    by listers.

commit aefc5c97f80ae869447165c12620df5ff1775afc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jul 6 09:48:29 2020 +0200

    Don't recurse into attrs objects when serializing
    
    We need to use our serialization hook recursively to make sure that we can
    deserialize nested data structures.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/37/ for more details.

Harbormaster completed remote builds in B13352: Diff 12130.Jul 6 2020, 11:54 AM
Closed by commit rDSCH5b373ceb36e4: Introduce a get_listed_origins endpoint (authored by olasd). · Explain WhyJul 6 2020, 12:16 PM
This revision was automatically updated to reflect the committed changes.
olasd added a commit: rDSCH5b373ceb36e4: Introduce a get_listed_origins endpoint.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
sql/
updates/
4 lines--
swh/
scheduler/
64 lines89%
Loading...
24 lines65%
Loading...
33 lines100%
Loading...
sql/
2 lines--
3 lines--
tests/
1 line100%
Loading...
47 lines98%
Loading...

Diff 12129

View Options

sql/updates/17.sql

Loading...
View Options

swh/scheduler/backend.py

Loading...
View Options

swh/scheduler/interface.py

Loading...
View Options

swh/scheduler/model.py

Loading...
View Options

swh/scheduler/sql/30-swh-schema.sql

Loading...
View Options

swh/scheduler/sql/60-swh-indexes.sql

Loading...
View Options

swh/scheduler/tests/test_api_client.py

Loading...
View Options

swh/scheduler/tests/test_scheduler.py

Loading...
About · Developer information · Legal