Page MenuHomeSoftware Heritage
Differential D5919

Start handling of recurrent loading tasks in scheduler
ClosedPublic
Actions

Authored by ardumont on Jun 23 2021, 6:11 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Maniphest Tasks
T2345: Improve handling of recurrent loading tasks in scheduler
Commits
rDSCHcb1edf1ab24d: Introduce storage for the recurrent visit scheduler queue position
rDSCHec6e69f6415a: Start handling of recurrent loading tasks in scheduler
rDSCHc486b28ece7c: journal_client: Explicit docstring
Summary
  • This deals with the next_position_offset update done by the scheduler journal client.
  • Introduce storage endpoint for the recurrent visit scheduler queue position read/write.

TODO:

  • update the journal client so it computes the queue position per visit type (D5950)

Related to T2345

Test Plan

tox

Diff Detail

Repository
rDSCH Scheduling utilities
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 22337
Build 34772: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 34771: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
37 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_journal_client::test_journal_client_origin_visit_status_duplicated_messages
swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f09406027b8> def test_journal_client_origin_visit_status_duplicated_messages(swh_scheduler):
39 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_journal_client::test_journal_client_origin_visit_status_from_journal_last_eventful
swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f0948141cf8> def test_journal_client_origin_visit_status_from_journal_last_eventful(swh_scheduler):
43 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_journal_client::test_journal_client_origin_visit_status_from_journal_last_failed
swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f0949c5e860> def test_journal_client_origin_visit_status_from_journal_last_failed(swh_scheduler):
43 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_journal_client::test_journal_client_origin_visit_status_from_journal_last_failed2
swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f094a99a390> def test_journal_client_origin_visit_status_from_journal_last_failed2(swh_scheduler):
39 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_journal_client::test_journal_client_origin_visit_status_from_journal_last_notfound
swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f0949c8a3c8> def test_journal_client_origin_visit_status_from_journal_last_notfound(swh_scheduler):
View Full Test Results (37 Failed · 272 Passed · 1 Skipped)

Event Timeline

ardumont created this revision.Jun 23 2021, 6:11 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 23 2021, 6:11 PM
ardumont edited the summary of this revision. (Show Details)Jun 23 2021, 6:13 PM
swh-public-ci added a comment.Jun 23 2021, 6:13 PM

Build has FAILED

Patch application report for D5919 (id=21246)

Could not rebase; Attempt merge onto ed818702c4...

Updating ed81870..49e9549
Fast-forward
 swh/scheduler/backend.py                   | 31 ++++++++---
 swh/scheduler/cli/task.py                  |  6 ++-
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 56 ++++++++++++++------
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 7 files changed, 163 insertions(+), 60 deletions(-)
Changes applied before test
commit 49e9549a6112b38544c894e4401e055415a493ed
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit 5ce02958b2d5981bac7367f57af312320dbeef07
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    wip: Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

commit 1006f0aee494b96e333eef13331d0fad1fcc0b6e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 15:37:40 2021 +0200

    journal_client: Auto-generate the empty object from model fields
    
    This will help us when adding new fields to the table.

commit 6400cc2b95cb1afdfc54594ea2c059b69c78f6ef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 15:28:55 2021 +0200

    backend: Auto-generate origin visit stats upsert query
    
    This will help us when adding new fields to the table.

commit 3762c34024027dd4377302736bc804ce8a6da8b0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:49:12 2021 +0200

    cli/task: Ensure cli output is always in the same order

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/385/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/385/console

Harbormaster failed remote builds in B22230: Diff 21246!Jun 23 2021, 6:13 PM
ardumont updated this revision to Diff 21247.Jun 24 2021, 9:31 AM
  • Fix sql typo
  • Add missing entrypoint in the api endpoint listing test
swh-public-ci added a comment.Jun 24 2021, 9:34 AM

Build is green

Patch application report for D5919 (id=21247)

Could not rebase; Attempt merge onto ed818702c4...

Updating ed81870..54e412b
Fast-forward
 swh/scheduler/backend.py                   | 31 ++++++++---
 swh/scheduler/cli/task.py                  |  6 ++-
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 56 ++++++++++++++------
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 8 files changed, 165 insertions(+), 60 deletions(-)
Changes applied before test
commit 54e412b9f5fa242f72967f59e620389848a69c61
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit 5ce02958b2d5981bac7367f57af312320dbeef07
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    wip: Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

commit 1006f0aee494b96e333eef13331d0fad1fcc0b6e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 15:37:40 2021 +0200

    journal_client: Auto-generate the empty object from model fields
    
    This will help us when adding new fields to the table.

commit 6400cc2b95cb1afdfc54594ea2c059b69c78f6ef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 15:28:55 2021 +0200

    backend: Auto-generate origin visit stats upsert query
    
    This will help us when adding new fields to the table.

commit 3762c34024027dd4377302736bc804ce8a6da8b0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:49:12 2021 +0200

    cli/task: Ensure cli output is always in the same order

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/386/ for more details.

Harbormaster completed remote builds in B22231: Diff 21247.Jun 24 2021, 9:34 AM
ardumont updated this revision to Diff 21284.Jun 25 2021, 3:38 PM
  • Update diff with the right commit range
  • rebase
swh-public-ci added a comment.Jun 25 2021, 3:41 PM

Build is green

Patch application report for D5919 (id=21284)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..7bbeccd
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 35 ++++++++++---
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 7 files changed, 143 insertions(+), 42 deletions(-)
Changes applied before test
commit 7bbeccd4a338ed94232294aa592e9f1cf3fa0350
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit 63ad8b9ebe0772269fb63cb608177fd28c6b0e20
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/387/ for more details.

Harbormaster completed remote builds in B22261: Diff 21284.Jun 25 2021, 3:41 PM
ardumont mentioned this in D5917: journal_client: Only check last_* fields for some permutation tests.Jun 25 2021, 5:38 PM
ardumont updated this revision to Diff 21346.Jun 29 2021, 2:20 PM

Add coverage on new backend endpoint

swh-public-ci added a comment.Jun 29 2021, 2:23 PM

Build is green

Patch application report for D5919 (id=21346)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..6ca999c
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 35 ++++++++++---
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 163 insertions(+), 42 deletions(-)
Changes applied before test
commit 6ca999c1009e7d61519275e32c4d4032da7bb548
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit 63ad8b9ebe0772269fb63cb608177fd28c6b0e20
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/388/ for more details.

Harbormaster completed remote builds in B22325: Diff 21346.Jun 29 2021, 2:23 PM
ardumont edited the summary of this revision. (Show Details)Jun 29 2021, 2:45 PM
ardumont added a project: Scheduling utilities.
ardumont updated this revision to Diff 21350.Jun 29 2021, 2:46 PM

Explicit the (ovs) journal client docstring (drop fixmes)

swh-public-ci added a comment.Jun 29 2021, 2:49 PM

Build has FAILED

Patch application report for D5919 (id=21350)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..4ed6283
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 57 ++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 182 insertions(+), 45 deletions(-)
Changes applied before test
commit 4ed62839099fd1da35456850f9d3e65a79670b8a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit bcbc732f3c8c3858df63a952933cf72cbe03e64b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 4e3bffd0e53643c999053733335b616faddccf1d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/389/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/389/console

Harbormaster failed remote builds in B22329: Diff 21350!Jun 29 2021, 2:49 PM
swh-public-ci added a comment.Jun 29 2021, 2:54 PM

Build has FAILED

Patch application report for D5919 (id=21350)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..4ed6283
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 57 ++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 182 insertions(+), 45 deletions(-)
Changes applied before test
commit 4ed62839099fd1da35456850f9d3e65a79670b8a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit bcbc732f3c8c3858df63a952933cf72cbe03e64b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 4e3bffd0e53643c999053733335b616faddccf1d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/390/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/390/console

ardumont updated this revision to Diff 21351.Jun 29 2021, 3:16 PM

Make the doc build pass (hopefully)

swh-public-ci added a comment.Jun 29 2021, 3:19 PM

Build is green

Patch application report for D5919 (id=21351)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..4fa0b3a
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 58 +++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 183 insertions(+), 45 deletions(-)
Changes applied before test
commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/391/ for more details.

Harbormaster completed remote builds in B22331: Diff 21351.Jun 29 2021, 3:19 PM
ardumont updated this revision to Diff 21357.Jun 29 2021, 5:23 PM
  • journal_client: Compute next position for origin visit

(build will fail now)

swh-public-ci added a comment.Jun 29 2021, 5:25 PM

Build has FAILED

Patch application report for D5919 (id=21357)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..f3f5410
Fast-forward
 swh/scheduler/backend.py                   |  18 +++++
 swh/scheduler/interface.py                 |  19 +++++
 swh/scheduler/journal_client.py            |  94 ++++++++++++++++++++---
 swh/scheduler/model.py                     |   8 ++
 swh/scheduler/sql/30-schema.sql            |  19 +++++
 swh/scheduler/tests/test_api_client.py     |   2 +
 swh/scheduler/tests/test_journal_client.py | 115 ++++++++++++++++++++---------
 swh/scheduler/tests/test_scheduler.py      |  20 +++++
 8 files changed, 248 insertions(+), 47 deletions(-)
Changes applied before test
commit f3f54100ab547d75bc53e198c428351b85f5ff0c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 16:00:01 2021 +0200

    journal_client: Compute next position for origin visit

commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/392/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/392/console

Harbormaster failed remote builds in B22336: Diff 21357!Jun 29 2021, 5:25 PM
ardumont added inline comments.Jun 29 2021, 5:33 PM
swh/scheduler/journal_client.py
201–202

¯\_(ツ)_/¯

ardumont updated this revision to Diff 21358.Jun 29 2021, 5:39 PM

Fix typo (build will still fail)

swh-public-ci added a comment.Jun 29 2021, 5:43 PM

Build has FAILED

Patch application report for D5919 (id=21358)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..73c8c51
Fast-forward
 swh/scheduler/backend.py                   |  18 +++++
 swh/scheduler/interface.py                 |  19 +++++
 swh/scheduler/journal_client.py            |  94 ++++++++++++++++++++---
 swh/scheduler/model.py                     |   8 ++
 swh/scheduler/sql/30-schema.sql            |  19 +++++
 swh/scheduler/tests/test_api_client.py     |   2 +
 swh/scheduler/tests/test_journal_client.py | 115 ++++++++++++++++++++---------
 swh/scheduler/tests/test_scheduler.py      |  20 +++++
 8 files changed, 248 insertions(+), 47 deletions(-)
Changes applied before test
commit 73c8c511470bd8fa600b2e3c61e73268eee4eb73
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 16:00:01 2021 +0200

    journal_client: Compute next position for origin visit

commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/393/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/393/console

Harbormaster failed remote builds in B22337: Diff 21358!Jun 29 2021, 5:43 PM
ardumont updated this revision to Diff 21362.Jun 30 2021, 10:03 AM
ardumont added a child revision: D5950: journal_client: Compute next position for origin visit.

Drop the last commit [1] so the build stays green and the perimeter does not grow out of
hand.

[1] moved to D5950

ardumont updated this revision to Diff 21363.Jun 30 2021, 10:05 AM

Actually drop the commit

ardumont edited the summary of this revision. (Show Details)Jun 30 2021, 10:08 AM
swh-public-ci added a comment.Jun 30 2021, 10:09 AM

Build has FAILED

Patch application report for D5919 (id=21362)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..73c8c51
Fast-forward
 swh/scheduler/backend.py                   |  18 +++++
 swh/scheduler/interface.py                 |  19 +++++
 swh/scheduler/journal_client.py            |  94 ++++++++++++++++++++---
 swh/scheduler/model.py                     |   8 ++
 swh/scheduler/sql/30-schema.sql            |  19 +++++
 swh/scheduler/tests/test_api_client.py     |   2 +
 swh/scheduler/tests/test_journal_client.py | 115 ++++++++++++++++++++---------
 swh/scheduler/tests/test_scheduler.py      |  20 +++++
 8 files changed, 248 insertions(+), 47 deletions(-)
Changes applied before test
commit 73c8c511470bd8fa600b2e3c61e73268eee4eb73
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 16:00:01 2021 +0200

    journal_client: Compute next position for origin visit

commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/395/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/395/console

Harbormaster failed remote builds in B22341: Diff 21362!Jun 30 2021, 10:09 AM
swh-public-ci added a comment.Jun 30 2021, 10:11 AM

Build is green

Patch application report for D5919 (id=21363)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..4fa0b3a
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 58 +++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 183 insertions(+), 45 deletions(-)
Changes applied before test
commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/396/ for more details.

Harbormaster completed remote builds in B22342: Diff 21363.Jun 30 2021, 10:11 AM
ardumont edited the test plan for this revision. (Show Details)Jul 1 2021, 10:25 AM
ardumont updated this revision to Diff 21379.Jul 1 2021, 10:39 AM

Add missing upgrade script

swh-public-ci added a comment.Jul 1 2021, 10:42 AM

Build is green

Patch application report for D5919 (id=21379)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..cb1edf1
Fast-forward
 sql/updates/29.sql                         | 27 ++++++++++
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 58 +++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 21 +++++++-
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 9 files changed, 211 insertions(+), 46 deletions(-)
 create mode 100644 sql/updates/29.sql
Changes applied before test
commit cb1edf1ab24d1c8db5821578a7fb2633fab50ff4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit ec6e69f6415a007611c46f25e7c48e909a793d53
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/397/ for more details.

Harbormaster completed remote builds in B22358: Diff 21379.Jul 1 2021, 10:42 AM
ardumont published this revision for review.Jul 6 2021, 3:58 PM
ardumont retitled this revision from wip: Start handling of recurrent loading tasks in scheduler to Start handling of recurrent loading tasks in scheduler.
ardumont mentioned this in T2345: Improve handling of recurrent loading tasks in scheduler.Jul 8 2021, 12:30 PM
olasd accepted this revision.Jul 22 2021, 11:29 AM
This revision is now accepted and ready to land.Jul 22 2021, 11:29 AM
Closed by commit rDSCHc486b28ece7c: journal_client: Explicit docstring (authored by ardumont). · Explain WhyJul 22 2021, 2:22 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDSCHc486b28ece7c: journal_client: Explicit docstring.
ardumont added a commit: rDSCHec6e69f6415a: Start handling of recurrent loading tasks in scheduler.
ardumont added a commit: rDSCHcb1edf1ab24d: Introduce storage for the recurrent visit scheduler queue position.

Revision Contents
Changeset List

PathSize
swh/
scheduler/
18 lines
19 lines
94 lines
8 lines
sql/
19 lines
tests/
2 lines
93 lines
20 lines

Diff 21358

View Options

swh/scheduler/backend.py

Loading...
View Options

swh/scheduler/interface.py

Loading...
View Options

swh/scheduler/journal_client.py

Loading...
View Options

swh/scheduler/model.py

Loading...
View Options

swh/scheduler/sql/30-schema.sql

Loading...
View Options

swh/scheduler/tests/test_api_client.py

Loading...
View Options

swh/scheduler/tests/test_journal_client.py

Loading...
View Options

swh/scheduler/tests/test_scheduler.py

Loading...
About · Developer information · Legal