Page MenuHomeSoftware Heritage
Differential D4890

journal_client: Read visit_stats entries by batch out of the loop
ClosedPublic
Actions

Authored by ardumont on Jan 19 2021, 6:52 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Maniphest Tasks
T2967: Write journal client subcribed to origin_visit_status topics
Commits
rDSCH9fb0dd6c7cc3: journal_client: Read visit_stats entries by batch out of the loop
Summary

Related to T2967
Depends on D4888

Diff Detail

Repository
rDSCH Scheduling utilities
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 18503
Build 28619: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 28618: arc lint + arc unit

Event Timeline

ardumont created this revision.Jan 19 2021, 6:52 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJan 19 2021, 6:52 PM
swh-public-ci added a comment.Jan 19 2021, 6:54 PM

Build is green

Patch application report for D4890 (id=17366)

Could not rebase; Attempt merge onto 0a32a31195...

Merge made by the 'recursive' strategy.
 swh/scheduler/backend.py                   |  29 +++--
 swh/scheduler/interface.py                 |   6 +-
 swh/scheduler/journal_client.py            |  14 ++-
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 179 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  33 +++---
 6 files changed, 140 insertions(+), 126 deletions(-)
Changes applied before test
commit 4640511f740df579ccbd73bf54fff1321b0d7d91
Merge: 0a32a31 85ad949
Author: Jenkins user <jenkins@localhost>
Date:   Tue Jan 19 17:52:46 2021 +0000

    Merge branch 'diff-target' into HEAD

commit 85ad94906dac77b167531d7d63a31db2e64405dc
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit cc3fe724526a65116ed0dfb97ad084eb120ebad2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/175/ for more details.

Harbormaster completed remote builds in B18502: Diff 17366.Jan 19 2021, 6:54 PM
ardumont requested review of this revision.Jan 19 2021, 6:54 PM
ardumont updated this revision to Diff 17367.Jan 19 2021, 7:07 PM

Keep only interesting visits to ask for their visit_stats informations

swh-public-ci added a comment.Jan 19 2021, 7:09 PM

Build is green

Patch application report for D4890 (id=17367)

Could not rebase; Attempt merge onto 0a32a31195...

Merge made by the 'recursive' strategy.
 swh/scheduler/backend.py                   |  29 +++--
 swh/scheduler/interface.py                 |   6 +-
 swh/scheduler/journal_client.py            |  22 ++--
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 179 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  33 +++---
 6 files changed, 145 insertions(+), 129 deletions(-)
Changes applied before test
commit 4241e0c14cfd984de3015491b54361c88f44b33a
Merge: 0a32a31 fc7b738
Author: Jenkins user <jenkins@localhost>
Date:   Tue Jan 19 18:07:53 2021 +0000

    Merge branch 'diff-target' into HEAD

commit fc7b7382e7e5877d35b320277205d1225363873b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit cc3fe724526a65116ed0dfb97ad084eb120ebad2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/176/ for more details.

Harbormaster completed remote builds in B18503: Diff 17367.Jan 19 2021, 7:09 PM
olasd accepted this revision.Jan 20 2021, 10:38 AM
olasd added a subscriber: olasd.
olasd added inline comments.
swh/scheduler/journal_client.py
53

Maybe make this a list(set(...))? shouldn't change much in practice.

This revision is now accepted and ready to land.Jan 20 2021, 10:38 AM
ardumont updated this revision to Diff 17392.Jan 20 2021, 3:07 PM

Rebase

swh-public-ci added a comment.Jan 20 2021, 3:07 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/188/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/188/console

Harbormaster failed remote builds in B18528: Diff 17392!Jan 20 2021, 3:07 PM
ardumont edited the summary of this revision. (Show Details)Jan 20 2021, 3:09 PM
ardumont added a parent revision: D4888: scheduler: Make origin_visit_stats_get read multiple entries.
swh-public-ci added a comment.Jan 20 2021, 3:09 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/189/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/189/console

ardumont updated this revision to Diff 17393.Jan 20 2021, 3:14 PM

Adapt according to suggestion

swh-public-ci added a comment.Jan 20 2021, 3:14 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/190/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/190/console

Harbormaster failed remote builds in B18529: Diff 17393!Jan 20 2021, 3:14 PM
ardumont updated this revision to Diff 17395.Jan 20 2021, 3:19 PM

Rebase

swh-public-ci added a comment.Jan 20 2021, 3:23 PM

Build is green

Patch application report for D4890 (id=17395)

Could not rebase; Attempt merge onto b03d978241...

Updating b03d978..1a8f81d
Fast-forward
 swh/scheduler/backend.py                   |  22 ++--
 swh/scheduler/interface.py                 |  14 ++-
 swh/scheduler/journal_client.py            |  22 ++--
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 189 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  41 +++----
 6 files changed, 157 insertions(+), 136 deletions(-)
Changes applied before test
commit 1a8f81d80aed1e231fe7f3aed516e29798d1f50b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit ffa2b43153137b4a0b51690f4179309688c867b7
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/192/ for more details.

Harbormaster completed remote builds in B18531: Diff 17395.Jan 20 2021, 3:23 PM
ardumont updated this revision to Diff 17436.Jan 20 2021, 5:50 PM

Rebase

swh-public-ci added a comment.Jan 20 2021, 5:58 PM

Build has FAILED

Patch application report for D4890 (id=17436)

Could not rebase; Attempt merge onto 898820fac5...

Updating 898820f..1c2c436
Fast-forward
 swh/scheduler/backend.py                   |  22 ++--
 swh/scheduler/interface.py                 |  14 ++-
 swh/scheduler/journal_client.py            |  22 ++--
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 189 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  41 +++----
 6 files changed, 157 insertions(+), 136 deletions(-)
Changes applied before test
commit 1c2c436eaec3be99e525ed8e5efd91d572cc016c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit 1fd92d0b378da82ec605de36f21c428df6236515
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/200/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/200/console

Harbormaster failed remote builds in B18573: Diff 17436!Jan 20 2021, 5:58 PM
ardumont updated this revision to Diff 17441.Jan 20 2021, 6:46 PM

Rebase

swh-public-ci added a comment.Jan 20 2021, 6:50 PM

Build is green

Patch application report for D4890 (id=17441)

Could not rebase; Attempt merge onto c7b740cafa...

Auto-merging swh/scheduler/tests/test_cli_journal.py
Merge made by the 'recursive' strategy.
 swh/scheduler/backend.py                   |  22 ++--
 swh/scheduler/interface.py                 |  14 ++-
 swh/scheduler/journal_client.py            |  22 ++--
 swh/scheduler/simulator/origins.py         |   4 +-
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 189 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  41 +++----
 7 files changed, 160 insertions(+), 137 deletions(-)
Changes applied before test
commit 7136a8a12f1a10ff35515b526349ea0d3719c848
Merge: c7b740c e124699
Author: Jenkins user <jenkins@localhost>
Date:   Wed Jan 20 17:48:42 2021 +0000

    Merge branch 'diff-target' into HEAD

commit e124699d2fd69272c6fc708d24bd354b1054cf01
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit c4250eb896816c88ad720f6dd2817a4bd1dc1b04
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/204/ for more details.

Harbormaster completed remote builds in B18578: Diff 17441.Jan 20 2021, 6:50 PM
ardumont updated this revision to Diff 17442.Jan 20 2021, 7:10 PM

Rebase

swh-public-ci added a comment.Jan 20 2021, 7:12 PM

Build is green

Patch application report for D4890 (id=17442)

Could not rebase; Attempt merge onto c7b740cafa...

Updating c7b740c..ca8dcd7
Fast-forward
 swh/scheduler/backend.py                   |  22 ++--
 swh/scheduler/interface.py                 |  14 ++-
 swh/scheduler/journal_client.py            |  22 ++--
 swh/scheduler/simulator/origins.py         |   4 +-
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 189 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  41 +++----
 7 files changed, 160 insertions(+), 137 deletions(-)
Changes applied before test
commit ca8dcd7c4710065437aee4e59ed02a61c2ff1ecf
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit 1cbc3f6f39c6dd12d162f8be7bdbc4689f86188d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/205/ for more details.

Harbormaster completed remote builds in B18579: Diff 17442.Jan 20 2021, 7:12 PM
ardumont updated this revision to Diff 17450.Jan 21 2021, 10:57 AM

Rebase

swh-public-ci added a comment.Jan 21 2021, 11:01 AM

Build is green

Patch application report for D4890 (id=17450)

Could not rebase; Attempt merge onto ffe2aed2fa...

Updating ffe2aed..9fb0dd6
Fast-forward
 swh/scheduler/backend.py                   | 22 +++----
 swh/scheduler/interface.py                 | 14 +++--
 swh/scheduler/journal_client.py            | 22 ++++---
 swh/scheduler/simulator/origins.py         |  4 +-
 swh/scheduler/tests/test_cli_journal.py    |  5 +-
 swh/scheduler/tests/test_journal_client.py | 95 +++++++++++++++---------------
 swh/scheduler/tests/test_scheduler.py      | 41 +++++++------
 7 files changed, 109 insertions(+), 94 deletions(-)
Changes applied before test
commit 9fb0dd6c7cc39af775af749959fef40f529942fd
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit d464b4cc1f9ae6a5c5c94a534826eff5cc27f12f
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/208/ for more details.

Harbormaster completed remote builds in B18586: Diff 17450.Jan 21 2021, 11:01 AM
Closed by commit rDSCH9fb0dd6c7cc3: journal_client: Read visit_stats entries by batch out of the loop (authored by ardumont). · Explain WhyJan 21 2021, 11:02 AM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDSCH9fb0dd6c7cc3: journal_client: Read visit_stats entries by batch out of the loop.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
scheduler/
22 lines100%
Loading...

Diff 17367

View Options

swh/scheduler/journal_client.py

Loading...
About · Developer information · Legal