Page MenuHomeSoftware Heritage

journal_client: Read visit_stats entries by batch out of the loop
ClosedPublic

Authored by ardumont on Jan 19 2021, 6:52 PM.

Diff Detail

Repository
rDSCH Scheduling utilities
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 18528
Build 28663: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 28662: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D4890 (id=17366)

Could not rebase; Attempt merge onto 0a32a31195...

Merge made by the 'recursive' strategy.
 swh/scheduler/backend.py                   |  29 +++--
 swh/scheduler/interface.py                 |   6 +-
 swh/scheduler/journal_client.py            |  14 ++-
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 179 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  33 +++---
 6 files changed, 140 insertions(+), 126 deletions(-)
Changes applied before test
commit 4640511f740df579ccbd73bf54fff1321b0d7d91
Merge: 0a32a31 85ad949
Author: Jenkins user <jenkins@localhost>
Date:   Tue Jan 19 17:52:46 2021 +0000

    Merge branch 'diff-target' into HEAD

commit 85ad94906dac77b167531d7d63a31db2e64405dc
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit cc3fe724526a65116ed0dfb97ad084eb120ebad2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/175/ for more details.

Keep only interesting visits to ask for their visit_stats informations

Build is green

Patch application report for D4890 (id=17367)

Could not rebase; Attempt merge onto 0a32a31195...

Merge made by the 'recursive' strategy.
 swh/scheduler/backend.py                   |  29 +++--
 swh/scheduler/interface.py                 |   6 +-
 swh/scheduler/journal_client.py            |  22 ++--
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 179 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  33 +++---
 6 files changed, 145 insertions(+), 129 deletions(-)
Changes applied before test
commit 4241e0c14cfd984de3015491b54361c88f44b33a
Merge: 0a32a31 fc7b738
Author: Jenkins user <jenkins@localhost>
Date:   Tue Jan 19 18:07:53 2021 +0000

    Merge branch 'diff-target' into HEAD

commit fc7b7382e7e5877d35b320277205d1225363873b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit cc3fe724526a65116ed0dfb97ad084eb120ebad2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/176/ for more details.

olasd added a subscriber: olasd.
olasd added inline comments.
swh/scheduler/journal_client.py
52

Maybe make this a list(set(...))? shouldn't change much in practice.

This revision is now accepted and ready to land.Jan 20 2021, 10:38 AM

Adapt according to suggestion

Build is green

Patch application report for D4890 (id=17395)

Could not rebase; Attempt merge onto b03d978241...

Updating b03d978..1a8f81d
Fast-forward
 swh/scheduler/backend.py                   |  22 ++--
 swh/scheduler/interface.py                 |  14 ++-
 swh/scheduler/journal_client.py            |  22 ++--
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 189 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  41 +++----
 6 files changed, 157 insertions(+), 136 deletions(-)
Changes applied before test
commit 1a8f81d80aed1e231fe7f3aed516e29798d1f50b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit ffa2b43153137b4a0b51690f4179309688c867b7
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/192/ for more details.

Build has FAILED

Patch application report for D4890 (id=17436)

Could not rebase; Attempt merge onto 898820fac5...

Updating 898820f..1c2c436
Fast-forward
 swh/scheduler/backend.py                   |  22 ++--
 swh/scheduler/interface.py                 |  14 ++-
 swh/scheduler/journal_client.py            |  22 ++--
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 189 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  41 +++----
 6 files changed, 157 insertions(+), 136 deletions(-)
Changes applied before test
commit 1c2c436eaec3be99e525ed8e5efd91d572cc016c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit 1fd92d0b378da82ec605de36f21c428df6236515
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/200/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/200/console

Build is green

Patch application report for D4890 (id=17441)

Could not rebase; Attempt merge onto c7b740cafa...

Auto-merging swh/scheduler/tests/test_cli_journal.py
Merge made by the 'recursive' strategy.
 swh/scheduler/backend.py                   |  22 ++--
 swh/scheduler/interface.py                 |  14 ++-
 swh/scheduler/journal_client.py            |  22 ++--
 swh/scheduler/simulator/origins.py         |   4 +-
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 189 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  41 +++----
 7 files changed, 160 insertions(+), 137 deletions(-)
Changes applied before test
commit 7136a8a12f1a10ff35515b526349ea0d3719c848
Merge: c7b740c e124699
Author: Jenkins user <jenkins@localhost>
Date:   Wed Jan 20 17:48:42 2021 +0000

    Merge branch 'diff-target' into HEAD

commit e124699d2fd69272c6fc708d24bd354b1054cf01
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit c4250eb896816c88ad720f6dd2817a4bd1dc1b04
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/204/ for more details.

Build is green

Patch application report for D4890 (id=17442)

Could not rebase; Attempt merge onto c7b740cafa...

Updating c7b740c..ca8dcd7
Fast-forward
 swh/scheduler/backend.py                   |  22 ++--
 swh/scheduler/interface.py                 |  14 ++-
 swh/scheduler/journal_client.py            |  22 ++--
 swh/scheduler/simulator/origins.py         |   4 +-
 swh/scheduler/tests/test_cli_journal.py    |   5 +-
 swh/scheduler/tests/test_journal_client.py | 189 +++++++++++++++--------------
 swh/scheduler/tests/test_scheduler.py      |  41 +++----
 7 files changed, 160 insertions(+), 137 deletions(-)
Changes applied before test
commit ca8dcd7c4710065437aee4e59ed02a61c2ff1ecf
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit 1cbc3f6f39c6dd12d162f8be7bdbc4689f86188d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/205/ for more details.

Build is green

Patch application report for D4890 (id=17450)

Could not rebase; Attempt merge onto ffe2aed2fa...

Updating ffe2aed..9fb0dd6
Fast-forward
 swh/scheduler/backend.py                   | 22 +++----
 swh/scheduler/interface.py                 | 14 +++--
 swh/scheduler/journal_client.py            | 22 ++++---
 swh/scheduler/simulator/origins.py         |  4 +-
 swh/scheduler/tests/test_cli_journal.py    |  5 +-
 swh/scheduler/tests/test_journal_client.py | 95 +++++++++++++++---------------
 swh/scheduler/tests/test_scheduler.py      | 41 +++++++------
 7 files changed, 109 insertions(+), 94 deletions(-)
Changes applied before test
commit 9fb0dd6c7cc39af775af749959fef40f529942fd
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:45:36 2021 +0100

    journal_client: Read visit_stats entries by batch out of the loop
    
    Related to T2967

commit d464b4cc1f9ae6a5c5c94a534826eff5cc27f12f
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jan 19 18:12:33 2021 +0100

    scheduler: Make origin_visit_stats_get read multiple entries
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/208/ for more details.