Details

Reviewers

Group Reviewers

Maniphest Tasks

T2345: Improve handling of recurrent loading tasks in scheduler

Commits

rDSCHcb1edf1ab24d: Introduce storage for the recurrent visit scheduler queue position
rDSCHec6e69f6415a: Start handling of recurrent loading tasks in scheduler
rDSCHc486b28ece7c: journal_client: Explicit docstring

Summary

This deals with the next_position_offset update done by the scheduler journal client.
Introduce storage endpoint for the recurrent visit scheduler queue position read/write.

TODO:

update the journal client so it computes the queue position per visit type (D5950)

Related to T2345

Test Plan

tox

Diff Detail

Repository

rDSCH Scheduling utilities

Branch

master

Lint

Lint Skipped

Unit

Unit Tests Skipped

Build Status

Buildable 22341
Build 34779: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 34778: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	39 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_journal_client::test_journal_client_origin_visit_status_duplicated_messages swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f4f1a8ab4a8> def test_journal_client_origin_visit_status_duplicated_messages(swh_scheduler):
	41 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_journal_client::test_journal_client_origin_visit_status_from_journal_last_eventful swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f4f1a7f2b38> def test_journal_client_origin_visit_status_from_journal_last_eventful(swh_scheduler):
	43 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_journal_client::test_journal_client_origin_visit_status_from_journal_last_failed swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f4f1a744860> def test_journal_client_origin_visit_status_from_journal_last_failed(swh_scheduler):
	45 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_journal_client::test_journal_client_origin_visit_status_from_journal_last_failed2 swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f4f1a828860> def test_journal_client_origin_visit_status_from_journal_last_failed2(swh_scheduler):
	48 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_journal_client::test_journal_client_origin_visit_status_from_journal_last_notfound swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f4f1a8bd128> def test_journal_client_origin_visit_status_from_journal_last_notfound(swh_scheduler):
		View Full Test Results (37 Failed · 272 Passed · 1 Skipped)

Event Timeline

ardumont created this revision.Jun 23 2021, 6:11 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJun 23 2021, 6:11 PM

ardumont edited the summary of this revision. (Show Details)Jun 23 2021, 6:13 PM

Build has FAILED

Patch application report for D5919 (id=21246)

Could not rebase; Attempt merge onto ed818702c4...

Updating ed81870..49e9549
Fast-forward
 swh/scheduler/backend.py                   | 31 ++++++++---
 swh/scheduler/cli/task.py                  |  6 ++-
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 56 ++++++++++++++------
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 7 files changed, 163 insertions(+), 60 deletions(-)

Changes applied before test

commit 49e9549a6112b38544c894e4401e055415a493ed
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit 5ce02958b2d5981bac7367f57af312320dbeef07
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    wip: Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

commit 1006f0aee494b96e333eef13331d0fad1fcc0b6e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 15:37:40 2021 +0200

    journal_client: Auto-generate the empty object from model fields
    
    This will help us when adding new fields to the table.

commit 6400cc2b95cb1afdfc54594ea2c059b69c78f6ef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 15:28:55 2021 +0200

    backend: Auto-generate origin visit stats upsert query
    
    This will help us when adding new fields to the table.

commit 3762c34024027dd4377302736bc804ce8a6da8b0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:49:12 2021 +0200

    cli/task: Ensure cli output is always in the same order

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/385/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/385/console

Harbormaster failed remote builds in B22230: Diff 21246!Jun 23 2021, 6:13 PM

Fix sql typo
Add missing entrypoint in the api endpoint listing test

Build is green

Patch application report for D5919 (id=21247)

Could not rebase; Attempt merge onto ed818702c4...

Updating ed81870..54e412b
Fast-forward
 swh/scheduler/backend.py                   | 31 ++++++++---
 swh/scheduler/cli/task.py                  |  6 ++-
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 56 ++++++++++++++------
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 8 files changed, 165 insertions(+), 60 deletions(-)

Changes applied before test

commit 54e412b9f5fa242f72967f59e620389848a69c61
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit 5ce02958b2d5981bac7367f57af312320dbeef07
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    wip: Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

commit 1006f0aee494b96e333eef13331d0fad1fcc0b6e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 15:37:40 2021 +0200

    journal_client: Auto-generate the empty object from model fields
    
    This will help us when adding new fields to the table.

commit 6400cc2b95cb1afdfc54594ea2c059b69c78f6ef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 15:28:55 2021 +0200

    backend: Auto-generate origin visit stats upsert query
    
    This will help us when adding new fields to the table.

commit 3762c34024027dd4377302736bc804ce8a6da8b0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:49:12 2021 +0200

    cli/task: Ensure cli output is always in the same order

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/386/ for more details.

Harbormaster completed remote builds in B22231: Diff 21247.Jun 24 2021, 9:34 AM

Update diff with the right commit range
rebase

Build is green

Patch application report for D5919 (id=21284)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..7bbeccd
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 35 ++++++++++---
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 7 files changed, 143 insertions(+), 42 deletions(-)

Changes applied before test

commit 7bbeccd4a338ed94232294aa592e9f1cf3fa0350
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit 63ad8b9ebe0772269fb63cb608177fd28c6b0e20
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/387/ for more details.

Harbormaster completed remote builds in B22261: Diff 21284.Jun 25 2021, 3:41 PM

ardumont mentioned this in D5917: journal_client: Only check last_* fields for some permutation tests.Jun 25 2021, 5:38 PM

Add coverage on new backend endpoint

Build is green

Patch application report for D5919 (id=21346)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..6ca999c
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 35 ++++++++++---
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 163 insertions(+), 42 deletions(-)

Changes applied before test

commit 6ca999c1009e7d61519275e32c4d4032da7bb548
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit 63ad8b9ebe0772269fb63cb608177fd28c6b0e20
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/388/ for more details.

Harbormaster completed remote builds in B22325: Diff 21346.Jun 29 2021, 2:23 PM

ardumont edited the summary of this revision. (Show Details)Jun 29 2021, 2:45 PM

ardumont added a project: Scheduling utilities.

Explicit the (ovs) journal client docstring (drop fixmes)

Build has FAILED

Patch application report for D5919 (id=21350)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..4ed6283
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 57 ++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 182 insertions(+), 45 deletions(-)

Changes applied before test

commit 4ed62839099fd1da35456850f9d3e65a79670b8a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit bcbc732f3c8c3858df63a952933cf72cbe03e64b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 4e3bffd0e53643c999053733335b616faddccf1d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/389/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/389/console

Harbormaster failed remote builds in B22329: Diff 21350!Jun 29 2021, 2:49 PM

Build has FAILED

Patch application report for D5919 (id=21350)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..4ed6283
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 57 ++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 182 insertions(+), 45 deletions(-)

Changes applied before test

commit 4ed62839099fd1da35456850f9d3e65a79670b8a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit bcbc732f3c8c3858df63a952933cf72cbe03e64b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 4e3bffd0e53643c999053733335b616faddccf1d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/390/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/390/console

Make the doc build pass (hopefully)

Build is green

Patch application report for D5919 (id=21351)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..4fa0b3a
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 58 +++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 183 insertions(+), 45 deletions(-)

Changes applied before test

commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/391/ for more details.

Harbormaster completed remote builds in B22331: Diff 21351.Jun 29 2021, 3:19 PM

journal_client: Compute next position for origin visit

(build will fail now)

Build has FAILED

Patch application report for D5919 (id=21357)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..f3f5410
Fast-forward
 swh/scheduler/backend.py                   |  18 +++++
 swh/scheduler/interface.py                 |  19 +++++
 swh/scheduler/journal_client.py            |  94 ++++++++++++++++++++---
 swh/scheduler/model.py                     |   8 ++
 swh/scheduler/sql/30-schema.sql            |  19 +++++
 swh/scheduler/tests/test_api_client.py     |   2 +
 swh/scheduler/tests/test_journal_client.py | 115 ++++++++++++++++++++---------
 swh/scheduler/tests/test_scheduler.py      |  20 +++++
 8 files changed, 248 insertions(+), 47 deletions(-)

Changes applied before test

commit f3f54100ab547d75bc53e198c428351b85f5ff0c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 16:00:01 2021 +0200

    journal_client: Compute next position for origin visit

commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/392/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/392/console

Harbormaster failed remote builds in B22336: Diff 21357!Jun 29 2021, 5:25 PM

ardumont added inline comments.Jun 29 2021, 5:33 PM

swh/scheduler/journal_client.py
201–202	¯\_(ツ)_/¯

Fix typo (build will still fail)

Build has FAILED

Patch application report for D5919 (id=21358)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..73c8c51
Fast-forward
 swh/scheduler/backend.py                   |  18 +++++
 swh/scheduler/interface.py                 |  19 +++++
 swh/scheduler/journal_client.py            |  94 ++++++++++++++++++++---
 swh/scheduler/model.py                     |   8 ++
 swh/scheduler/sql/30-schema.sql            |  19 +++++
 swh/scheduler/tests/test_api_client.py     |   2 +
 swh/scheduler/tests/test_journal_client.py | 115 ++++++++++++++++++++---------
 swh/scheduler/tests/test_scheduler.py      |  20 +++++
 8 files changed, 248 insertions(+), 47 deletions(-)

Changes applied before test

commit 73c8c511470bd8fa600b2e3c61e73268eee4eb73
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 16:00:01 2021 +0200

    journal_client: Compute next position for origin visit

commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/393/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/393/console

Harbormaster failed remote builds in B22337: Diff 21358!Jun 29 2021, 5:43 PM

Drop the last commit [1] so the build stays green and the perimeter does not grow out of
hand.

[1] moved to D5950

Actually drop the commit

ardumont edited the summary of this revision. (Show Details)Jun 30 2021, 10:08 AM

Build has FAILED

Patch application report for D5919 (id=21362)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..73c8c51
Fast-forward
 swh/scheduler/backend.py                   |  18 +++++
 swh/scheduler/interface.py                 |  19 +++++
 swh/scheduler/journal_client.py            |  94 ++++++++++++++++++++---
 swh/scheduler/model.py                     |   8 ++
 swh/scheduler/sql/30-schema.sql            |  19 +++++
 swh/scheduler/tests/test_api_client.py     |   2 +
 swh/scheduler/tests/test_journal_client.py | 115 ++++++++++++++++++++---------
 swh/scheduler/tests/test_scheduler.py      |  20 +++++
 8 files changed, 248 insertions(+), 47 deletions(-)

Changes applied before test

commit 73c8c511470bd8fa600b2e3c61e73268eee4eb73
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 16:00:01 2021 +0200

    journal_client: Compute next position for origin visit

commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/395/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/395/console

Harbormaster failed remote builds in B22341: Diff 21362!Jun 30 2021, 10:09 AM

Build is green

Patch application report for D5919 (id=21363)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..4fa0b3a
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 58 +++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 183 insertions(+), 45 deletions(-)

Changes applied before test

commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/396/ for more details.

Harbormaster completed remote builds in B22342: Diff 21363.Jun 30 2021, 10:11 AM

ardumont edited the test plan for this revision. (Show Details)Jul 1 2021, 10:25 AM

Add missing upgrade script

Build is green

Patch application report for D5919 (id=21379)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..cb1edf1
Fast-forward
 sql/updates/29.sql                         | 27 ++++++++++
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 58 +++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 21 +++++++-
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 9 files changed, 211 insertions(+), 46 deletions(-)
 create mode 100644 sql/updates/29.sql

Changes applied before test

commit cb1edf1ab24d1c8db5821578a7fb2633fab50ff4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit ec6e69f6415a007611c46f25e7c48e909a793d53
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/397/ for more details.

Harbormaster completed remote builds in B22358: Diff 21379.Jul 1 2021, 10:42 AM

ardumont published this revision for review.Jul 6 2021, 3:58 PM

ardumont retitled this revision from wip: Start handling of recurrent loading tasks in scheduler to Start handling of recurrent loading tasks in scheduler.

ardumont mentioned this in T2345: Improve handling of recurrent loading tasks in scheduler.Jul 8 2021, 12:30 PM

olasd accepted this revision.Jul 22 2021, 11:29 AM

This revision is now accepted and ready to land.Jul 22 2021, 11:29 AM

Closed by commit rDSCHc486b28ece7c: journal_client: Explicit docstring (authored by ardumont). · Explain WhyJul 22 2021, 2:22 PM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDSCHc486b28ece7c: journal_client: Explicit docstring.

ardumont added a commit: rDSCHec6e69f6415a: Start handling of recurrent loading tasks in scheduler.

ardumont added a commit: rDSCHcb1edf1ab24d: Introduce storage for the recurrent visit scheduler queue position.

Start handling of recurrent loading tasks in scheduler
ClosedPublic
Actions

Details

Diff Detail

Unit TestsFailed
View All

Event Timeline

Patch application report for D5919 (id=21246)

Changes applied before test

Patch application report for D5919 (id=21247)

Changes applied before test

Patch application report for D5919 (id=21284)

Changes applied before test

Patch application report for D5919 (id=21346)

Changes applied before test

Patch application report for D5919 (id=21350)

Changes applied before test

Patch application report for D5919 (id=21350)

Changes applied before test

Patch application report for D5919 (id=21351)

Changes applied before test

Patch application report for D5919 (id=21357)

Changes applied before test

Patch application report for D5919 (id=21358)

Changes applied before test

Patch application report for D5919 (id=21362)

Changes applied before test

Patch application report for D5919 (id=21363)

Changes applied before test

Patch application report for D5919 (id=21379)

Changes applied before test

Revision Contents
Changeset List

Diff 21362

swh/scheduler/backend.py

swh/scheduler/interface.py

swh/scheduler/journal_client.py

swh/scheduler/model.py

swh/scheduler/sql/30-schema.sql

swh/scheduler/tests/test_api_client.py

swh/scheduler/tests/test_journal_client.py

swh/scheduler/tests/test_scheduler.py

Start handling of recurrent loading tasks in schedulerClosedPublicActions

Details

Diff Detail

Unit TestsFailedView All

Event Timeline

Patch application report for D5919 (id=21246)

Changes applied before test

Patch application report for D5919 (id=21247)

Changes applied before test

Patch application report for D5919 (id=21284)

Changes applied before test

Patch application report for D5919 (id=21346)

Changes applied before test

Patch application report for D5919 (id=21350)

Changes applied before test

Patch application report for D5919 (id=21350)

Changes applied before test

Patch application report for D5919 (id=21351)

Changes applied before test

Patch application report for D5919 (id=21357)

Changes applied before test

Patch application report for D5919 (id=21358)

Changes applied before test

Patch application report for D5919 (id=21362)

Changes applied before test

Patch application report for D5919 (id=21363)

Changes applied before test

Patch application report for D5919 (id=21379)

Changes applied before test

Revision ContentsChangeset List

Diff 21362

swh/scheduler/backend.py

swh/scheduler/interface.py

swh/scheduler/journal_client.py

swh/scheduler/model.py

swh/scheduler/sql/30-schema.sql

swh/scheduler/tests/test_api_client.py

swh/scheduler/tests/test_journal_client.py

swh/scheduler/tests/test_scheduler.py

Start handling of recurrent loading tasks in scheduler
ClosedPublic
Actions

Unit TestsFailed
View All

Revision Contents
Changeset List