Page MenuHomeSoftware Heritage

Start handling of recurrent loading tasks in scheduler
ClosedPublic

Authored by ardumont on Jun 23 2021, 6:11 PM.

Details

Summary
  • This deals with the next_position_offset update done by the scheduler journal client.
  • Introduce storage endpoint for the recurrent visit scheduler queue position read/write.

TODO:

  • update the journal client so it computes the queue position per visit type (D5950)

Related to T2345

Test Plan

tox

Diff Detail

Repository
rDSCH Scheduling utilities
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 22337
Build 34772: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 34771: arc lint + arc unit

Unit TestsFailed

TimeTest
37 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_journal_client::test_journal_client_origin_visit_status_duplicated_messages
swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f09406027b8> def test_journal_client_origin_visit_status_duplicated_messages(swh_scheduler):
39 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_journal_client::test_journal_client_origin_visit_status_from_journal_last_eventful
swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f0948141cf8> def test_journal_client_origin_visit_status_from_journal_last_eventful(swh_scheduler):
43 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_journal_client::test_journal_client_origin_visit_status_from_journal_last_failed
swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f0949c5e860> def test_journal_client_origin_visit_status_from_journal_last_failed(swh_scheduler):
43 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_journal_client::test_journal_client_origin_visit_status_from_journal_last_failed2
swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f094a99a390> def test_journal_client_origin_visit_status_from_journal_last_failed2(swh_scheduler):
39 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_journal_client::test_journal_client_origin_visit_status_from_journal_last_notfound
swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f0949c8a3c8> def test_journal_client_origin_visit_status_from_journal_last_notfound(swh_scheduler):
View Full Test Results (37 Failed · 272 Passed · 1 Skipped)

Event Timeline

Build has FAILED

Patch application report for D5919 (id=21246)

Could not rebase; Attempt merge onto ed818702c4...

Updating ed81870..49e9549
Fast-forward
 swh/scheduler/backend.py                   | 31 ++++++++---
 swh/scheduler/cli/task.py                  |  6 ++-
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 56 ++++++++++++++------
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 7 files changed, 163 insertions(+), 60 deletions(-)
Changes applied before test
commit 49e9549a6112b38544c894e4401e055415a493ed
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit 5ce02958b2d5981bac7367f57af312320dbeef07
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    wip: Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

commit 1006f0aee494b96e333eef13331d0fad1fcc0b6e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 15:37:40 2021 +0200

    journal_client: Auto-generate the empty object from model fields
    
    This will help us when adding new fields to the table.

commit 6400cc2b95cb1afdfc54594ea2c059b69c78f6ef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 15:28:55 2021 +0200

    backend: Auto-generate origin visit stats upsert query
    
    This will help us when adding new fields to the table.

commit 3762c34024027dd4377302736bc804ce8a6da8b0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:49:12 2021 +0200

    cli/task: Ensure cli output is always in the same order

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/385/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/385/console

  • Fix sql typo
  • Add missing entrypoint in the api endpoint listing test

Build is green

Patch application report for D5919 (id=21247)

Could not rebase; Attempt merge onto ed818702c4...

Updating ed81870..54e412b
Fast-forward
 swh/scheduler/backend.py                   | 31 ++++++++---
 swh/scheduler/cli/task.py                  |  6 ++-
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 56 ++++++++++++++------
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 8 files changed, 165 insertions(+), 60 deletions(-)
Changes applied before test
commit 54e412b9f5fa242f72967f59e620389848a69c61
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit 5ce02958b2d5981bac7367f57af312320dbeef07
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    wip: Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

commit 1006f0aee494b96e333eef13331d0fad1fcc0b6e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 15:37:40 2021 +0200

    journal_client: Auto-generate the empty object from model fields
    
    This will help us when adding new fields to the table.

commit 6400cc2b95cb1afdfc54594ea2c059b69c78f6ef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 15:28:55 2021 +0200

    backend: Auto-generate origin visit stats upsert query
    
    This will help us when adding new fields to the table.

commit 3762c34024027dd4377302736bc804ce8a6da8b0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:49:12 2021 +0200

    cli/task: Ensure cli output is always in the same order

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/386/ for more details.

  • Update diff with the right commit range
  • rebase

Build is green

Patch application report for D5919 (id=21284)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..7bbeccd
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 35 ++++++++++---
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 7 files changed, 143 insertions(+), 42 deletions(-)
Changes applied before test
commit 7bbeccd4a338ed94232294aa592e9f1cf3fa0350
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit 63ad8b9ebe0772269fb63cb608177fd28c6b0e20
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/387/ for more details.

Add coverage on new backend endpoint

Build is green

Patch application report for D5919 (id=21346)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..6ca999c
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 35 ++++++++++---
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 163 insertions(+), 42 deletions(-)
Changes applied before test
commit 6ca999c1009e7d61519275e32c4d4032da7bb548
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit 63ad8b9ebe0772269fb63cb608177fd28c6b0e20
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/388/ for more details.

Explicit the (ovs) journal client docstring (drop fixmes)

Build has FAILED

Patch application report for D5919 (id=21350)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..4ed6283
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 57 ++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 182 insertions(+), 45 deletions(-)
Changes applied before test
commit 4ed62839099fd1da35456850f9d3e65a79670b8a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit bcbc732f3c8c3858df63a952933cf72cbe03e64b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 4e3bffd0e53643c999053733335b616faddccf1d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/389/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/389/console

Build has FAILED

Patch application report for D5919 (id=21350)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..4ed6283
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 57 ++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 182 insertions(+), 45 deletions(-)
Changes applied before test
commit 4ed62839099fd1da35456850f9d3e65a79670b8a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit bcbc732f3c8c3858df63a952933cf72cbe03e64b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 4e3bffd0e53643c999053733335b616faddccf1d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/390/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/390/console

Make the doc build pass (hopefully)

Build is green

Patch application report for D5919 (id=21351)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..4fa0b3a
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 58 +++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 183 insertions(+), 45 deletions(-)
Changes applied before test
commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/391/ for more details.

  • journal_client: Compute next position for origin visit

(build will fail now)

Build has FAILED

Patch application report for D5919 (id=21357)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..f3f5410
Fast-forward
 swh/scheduler/backend.py                   |  18 +++++
 swh/scheduler/interface.py                 |  19 +++++
 swh/scheduler/journal_client.py            |  94 ++++++++++++++++++++---
 swh/scheduler/model.py                     |   8 ++
 swh/scheduler/sql/30-schema.sql            |  19 +++++
 swh/scheduler/tests/test_api_client.py     |   2 +
 swh/scheduler/tests/test_journal_client.py | 115 ++++++++++++++++++++---------
 swh/scheduler/tests/test_scheduler.py      |  20 +++++
 8 files changed, 248 insertions(+), 47 deletions(-)
Changes applied before test
commit f3f54100ab547d75bc53e198c428351b85f5ff0c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 16:00:01 2021 +0200

    journal_client: Compute next position for origin visit

commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/392/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/392/console

swh/scheduler/journal_client.py
201–202

¯\_(ツ)_/¯

Fix typo (build will still fail)

Build has FAILED

Patch application report for D5919 (id=21358)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..73c8c51
Fast-forward
 swh/scheduler/backend.py                   |  18 +++++
 swh/scheduler/interface.py                 |  19 +++++
 swh/scheduler/journal_client.py            |  94 ++++++++++++++++++++---
 swh/scheduler/model.py                     |   8 ++
 swh/scheduler/sql/30-schema.sql            |  19 +++++
 swh/scheduler/tests/test_api_client.py     |   2 +
 swh/scheduler/tests/test_journal_client.py | 115 ++++++++++++++++++++---------
 swh/scheduler/tests/test_scheduler.py      |  20 +++++
 8 files changed, 248 insertions(+), 47 deletions(-)
Changes applied before test
commit 73c8c511470bd8fa600b2e3c61e73268eee4eb73
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 16:00:01 2021 +0200

    journal_client: Compute next position for origin visit

commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/393/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/393/console

Drop the last commit [1] so the build stays green and the perimeter does not grow out of
hand.

[1] moved to D5950

Build has FAILED

Patch application report for D5919 (id=21362)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..73c8c51
Fast-forward
 swh/scheduler/backend.py                   |  18 +++++
 swh/scheduler/interface.py                 |  19 +++++
 swh/scheduler/journal_client.py            |  94 ++++++++++++++++++++---
 swh/scheduler/model.py                     |   8 ++
 swh/scheduler/sql/30-schema.sql            |  19 +++++
 swh/scheduler/tests/test_api_client.py     |   2 +
 swh/scheduler/tests/test_journal_client.py | 115 ++++++++++++++++++++---------
 swh/scheduler/tests/test_scheduler.py      |  20 +++++
 8 files changed, 248 insertions(+), 47 deletions(-)
Changes applied before test
commit 73c8c511470bd8fa600b2e3c61e73268eee4eb73
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 16:00:01 2021 +0200

    journal_client: Compute next position for origin visit

commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/395/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/395/console

Build is green

Patch application report for D5919 (id=21363)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..4fa0b3a
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 58 +++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 183 insertions(+), 45 deletions(-)
Changes applied before test
commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/396/ for more details.

Build is green

Patch application report for D5919 (id=21379)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..cb1edf1
Fast-forward
 sql/updates/29.sql                         | 27 ++++++++++
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 58 +++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 21 +++++++-
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 9 files changed, 211 insertions(+), 46 deletions(-)
 create mode 100644 sql/updates/29.sql
Changes applied before test
commit cb1edf1ab24d1c8db5821578a7fb2633fab50ff4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit ec6e69f6415a007611c46f25e7c48e909a793d53
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/397/ for more details.

ardumont published this revision for review.Jul 6 2021, 3:58 PM
ardumont retitled this revision from wip: Start handling of recurrent loading tasks in scheduler to Start handling of recurrent loading tasks in scheduler.
This revision is now accepted and ready to land.Jul 22 2021, 11:29 AM