Page MenuHomeSoftware Heritage
Differential D4890

journal_client: Read visit_stats entries by batch out of the loop
ClosedPublic
Actions

Authored by ardumont on Jan 19 2021, 6:52 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Maniphest Tasks
T2967: Write journal client subcribed to origin_visit_status topics
Commits
rDSCH9fb0dd6c7cc3: journal_client: Read visit_stats entries by batch out of the loop
Summary

Related to T2967
Depends on D4888

Diff Detail

Repository
rDSCH Scheduling utilities
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Jan 19 2021, 6:52 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJan 19 2021, 6:52 PM
swh-public-ci added a comment.Jan 19 2021, 6:54 PM

Build is green

Patch application report for D4890 (id=17366)

Could not rebase; Attempt merge onto 0a32a31195...

Merge made by the 'recursive' strategy.
 swh/scheduler/backend.py                   |  29 +++--
 swh/scheduler/interface.py                 |   6 +-
 swh/scheduler/journal_client.py            |  14 ++-
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 179 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  33 +++---
 6 files changed, 140 insertions(+), 126 deletions(-)
Changes applied before test
commit 4640511f740df579ccbd73bf54fff1321b0d7d91
Merge: 0a32a31 85ad949
Author: Jenkins user <jenkins@localhost>
Date:   Tue Jan 19 17:52:46 2021 +0000

    Merge branch 'diff-target' into HEAD

commit 85ad94906dac77b167531d7d63a31db2e64405dc
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit cc3fe724526a65116ed0dfb97ad084eb120ebad2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/175/ for more details.

Harbormaster completed remote builds in B18502: Diff 17366.Jan 19 2021, 6:54 PM
ardumont requested review of this revision.Jan 19 2021, 6:54 PM
ardumont updated this revision to Diff 17367.Jan 19 2021, 7:07 PM

Keep only interesting visits to ask for their visit_stats informations

swh-public-ci added a comment.Jan 19 2021, 7:09 PM

Build is green

Patch application report for D4890 (id=17367)

Could not rebase; Attempt merge onto 0a32a31195...

Merge made by the 'recursive' strategy.
 swh/scheduler/backend.py                   |  29 +++--
 swh/scheduler/interface.py                 |   6 +-
 swh/scheduler/journal_client.py            |  22 ++--
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 179 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  33 +++---
 6 files changed, 145 insertions(+), 129 deletions(-)
Changes applied before test
commit 4241e0c14cfd984de3015491b54361c88f44b33a
Merge: 0a32a31 fc7b738
Author: Jenkins user <jenkins@localhost>
Date:   Tue Jan 19 18:07:53 2021 +0000

    Merge branch 'diff-target' into HEAD

commit fc7b7382e7e5877d35b320277205d1225363873b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit cc3fe724526a65116ed0dfb97ad084eb120ebad2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/176/ for more details.

Harbormaster completed remote builds in B18503: Diff 17367.Jan 19 2021, 7:09 PM
olasd accepted this revision.Jan 20 2021, 10:38 AM
olasd added a subscriber: olasd.
olasd added inline comments.
swh/scheduler/journal_client.py
57

Maybe make this a list(set(...))? shouldn't change much in practice.

This revision is now accepted and ready to land.Jan 20 2021, 10:38 AM
ardumont updated this revision to Diff 17392.Jan 20 2021, 3:07 PM

Rebase

swh-public-ci added a comment.Jan 20 2021, 3:07 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/188/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/188/console

Harbormaster failed remote builds in B18528: Diff 17392!Jan 20 2021, 3:07 PM
ardumont edited the summary of this revision. (Show Details)Jan 20 2021, 3:09 PM
ardumont added a parent revision: D4888: scheduler: Make origin_visit_stats_get read multiple entries.
swh-public-ci added a comment.Jan 20 2021, 3:09 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/189/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/189/console

ardumont updated this revision to Diff 17393.Jan 20 2021, 3:14 PM

Adapt according to suggestion

swh-public-ci added a comment.Jan 20 2021, 3:14 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/190/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/190/console

Harbormaster failed remote builds in B18529: Diff 17393!Jan 20 2021, 3:14 PM
ardumont updated this revision to Diff 17395.Jan 20 2021, 3:19 PM

Rebase

swh-public-ci added a comment.Jan 20 2021, 3:23 PM

Build is green

Patch application report for D4890 (id=17395)

Could not rebase; Attempt merge onto b03d978241...

Updating b03d978..1a8f81d
Fast-forward
 swh/scheduler/backend.py                   |  22 ++--
 swh/scheduler/interface.py                 |  14 ++-
 swh/scheduler/journal_client.py            |  22 ++--
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 189 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  41 +++----
 6 files changed, 157 insertions(+), 136 deletions(-)
Changes applied before test
commit 1a8f81d80aed1e231fe7f3aed516e29798d1f50b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit ffa2b43153137b4a0b51690f4179309688c867b7
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/192/ for more details.

Harbormaster completed remote builds in B18531: Diff 17395.Jan 20 2021, 3:23 PM
ardumont updated this revision to Diff 17436.Jan 20 2021, 5:50 PM

Rebase

swh-public-ci added a comment.Jan 20 2021, 5:58 PM

Build has FAILED

Patch application report for D4890 (id=17436)

Could not rebase; Attempt merge onto 898820fac5...

Updating 898820f..1c2c436
Fast-forward
 swh/scheduler/backend.py                   |  22 ++--
 swh/scheduler/interface.py                 |  14 ++-
 swh/scheduler/journal_client.py            |  22 ++--
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 189 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  41 +++----
 6 files changed, 157 insertions(+), 136 deletions(-)
Changes applied before test
commit 1c2c436eaec3be99e525ed8e5efd91d572cc016c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit 1fd92d0b378da82ec605de36f21c428df6236515
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/200/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/200/console

Harbormaster failed remote builds in B18573: Diff 17436!Jan 20 2021, 5:58 PM
ardumont updated this revision to Diff 17441.Jan 20 2021, 6:46 PM

Rebase

swh-public-ci added a comment.Jan 20 2021, 6:50 PM

Build is green

Patch application report for D4890 (id=17441)

Could not rebase; Attempt merge onto c7b740cafa...

Auto-merging swh/scheduler/tests/test_cli_journal.py
Merge made by the 'recursive' strategy.
 swh/scheduler/backend.py                   |  22 ++--
 swh/scheduler/interface.py                 |  14 ++-
 swh/scheduler/journal_client.py            |  22 ++--
 swh/scheduler/simulator/origins.py         |   4 +-
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 189 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  41 +++----
 7 files changed, 160 insertions(+), 137 deletions(-)
Changes applied before test
commit 7136a8a12f1a10ff35515b526349ea0d3719c848
Merge: c7b740c e124699
Author: Jenkins user <jenkins@localhost>
Date:   Wed Jan 20 17:48:42 2021 +0000

    Merge branch 'diff-target' into HEAD

commit e124699d2fd69272c6fc708d24bd354b1054cf01
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit c4250eb896816c88ad720f6dd2817a4bd1dc1b04
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/204/ for more details.

Harbormaster completed remote builds in B18578: Diff 17441.Jan 20 2021, 6:50 PM
ardumont updated this revision to Diff 17442.Jan 20 2021, 7:10 PM

Rebase

swh-public-ci added a comment.Jan 20 2021, 7:12 PM

Build is green

Patch application report for D4890 (id=17442)

Could not rebase; Attempt merge onto c7b740cafa...

Updating c7b740c..ca8dcd7
Fast-forward
 swh/scheduler/backend.py                   |  22 ++--
 swh/scheduler/interface.py                 |  14 ++-
 swh/scheduler/journal_client.py            |  22 ++--
 swh/scheduler/simulator/origins.py         |   4 +-
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 189 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  41 +++----
 7 files changed, 160 insertions(+), 137 deletions(-)
Changes applied before test
commit ca8dcd7c4710065437aee4e59ed02a61c2ff1ecf
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit 1cbc3f6f39c6dd12d162f8be7bdbc4689f86188d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/205/ for more details.

Harbormaster completed remote builds in B18579: Diff 17442.Jan 20 2021, 7:12 PM
ardumont updated this revision to Diff 17450.Jan 21 2021, 10:57 AM

Rebase

swh-public-ci added a comment.Jan 21 2021, 11:01 AM

Build is green

Patch application report for D4890 (id=17450)

Could not rebase; Attempt merge onto ffe2aed2fa...

Updating ffe2aed..9fb0dd6
Fast-forward
 swh/scheduler/backend.py                   | 22 +++----
 swh/scheduler/interface.py                 | 14 +++--
 swh/scheduler/journal_client.py            | 22 ++++---
 swh/scheduler/simulator/origins.py         |  4 +-
 swh/scheduler/tests/test_cli_journal.py    |  5 +-
 swh/scheduler/tests/test_journal_client.py | 95 +++++++++++++++---------------
 swh/scheduler/tests/test_scheduler.py      | 41 +++++++------
 7 files changed, 109 insertions(+), 94 deletions(-)
Changes applied before test
commit 9fb0dd6c7cc39af775af749959fef40f529942fd
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit d464b4cc1f9ae6a5c5c94a534826eff5cc27f12f
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/208/ for more details.

Harbormaster completed remote builds in B18586: Diff 17450.Jan 21 2021, 11:01 AM
Closed by commit rDSCH9fb0dd6c7cc3: journal_client: Read visit_stats entries by batch out of the loop (authored by ardumont). · Explain WhyJan 21 2021, 11:02 AM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDSCH9fb0dd6c7cc3: journal_client: Read visit_stats entries by batch out of the loop.

Revision Contents
Changeset List

PathSize
swh/
scheduler/
22 lines

Diff 17452

View Options

swh/scheduler/journal_client.py

Loading...
About · Developer information · Legal