Page MenuHomeSoftware Heritage
Differential D5919

Start handling of recurrent loading tasks in scheduler
ClosedPublic
Actions

Authored by ardumont on Jun 23 2021, 6:11 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Maniphest Tasks
T2345: Improve handling of recurrent loading tasks in scheduler
Commits
rDSCHcb1edf1ab24d: Introduce storage for the recurrent visit scheduler queue position
rDSCHec6e69f6415a: Start handling of recurrent loading tasks in scheduler
rDSCHc486b28ece7c: journal_client: Explicit docstring
Summary
  • This deals with the next_position_offset update done by the scheduler journal client.
  • Introduce storage endpoint for the recurrent visit scheduler queue position read/write.

TODO:

  • update the journal client so it computes the queue position per visit type (D5950)

Related to T2345

Test Plan

tox

Diff Detail

Event Timeline

ardumont created this revision.Jun 23 2021, 6:11 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 23 2021, 6:11 PM
ardumont edited the summary of this revision. (Show Details)Jun 23 2021, 6:13 PM
swh-public-ci added a comment.Jun 23 2021, 6:13 PM

Build has FAILED

Patch application report for D5919 (id=21246)

Could not rebase; Attempt merge onto ed818702c4...

Updating ed81870..49e9549
Fast-forward
 swh/scheduler/backend.py                   | 31 ++++++++---
 swh/scheduler/cli/task.py                  |  6 ++-
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 56 ++++++++++++++------
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 7 files changed, 163 insertions(+), 60 deletions(-)
Changes applied before test
commit 49e9549a6112b38544c894e4401e055415a493ed
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit 5ce02958b2d5981bac7367f57af312320dbeef07
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    wip: Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

commit 1006f0aee494b96e333eef13331d0fad1fcc0b6e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 15:37:40 2021 +0200

    journal_client: Auto-generate the empty object from model fields
    
    This will help us when adding new fields to the table.

commit 6400cc2b95cb1afdfc54594ea2c059b69c78f6ef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 15:28:55 2021 +0200

    backend: Auto-generate origin visit stats upsert query
    
    This will help us when adding new fields to the table.

commit 3762c34024027dd4377302736bc804ce8a6da8b0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:49:12 2021 +0200

    cli/task: Ensure cli output is always in the same order

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/385/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/385/console

Harbormaster failed remote builds in B22230: Diff 21246!Jun 23 2021, 6:13 PM
ardumont updated this revision to Diff 21247.Jun 24 2021, 9:31 AM
  • Fix sql typo
  • Add missing entrypoint in the api endpoint listing test
swh-public-ci added a comment.Jun 24 2021, 9:34 AM

Build is green

Patch application report for D5919 (id=21247)

Could not rebase; Attempt merge onto ed818702c4...

Updating ed81870..54e412b
Fast-forward
 swh/scheduler/backend.py                   | 31 ++++++++---
 swh/scheduler/cli/task.py                  |  6 ++-
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 56 ++++++++++++++------
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 8 files changed, 165 insertions(+), 60 deletions(-)
Changes applied before test
commit 54e412b9f5fa242f72967f59e620389848a69c61
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit 5ce02958b2d5981bac7367f57af312320dbeef07
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    wip: Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

commit 1006f0aee494b96e333eef13331d0fad1fcc0b6e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 15:37:40 2021 +0200

    journal_client: Auto-generate the empty object from model fields
    
    This will help us when adding new fields to the table.

commit 6400cc2b95cb1afdfc54594ea2c059b69c78f6ef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 15:28:55 2021 +0200

    backend: Auto-generate origin visit stats upsert query
    
    This will help us when adding new fields to the table.

commit 3762c34024027dd4377302736bc804ce8a6da8b0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:49:12 2021 +0200

    cli/task: Ensure cli output is always in the same order

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/386/ for more details.

Harbormaster completed remote builds in B22231: Diff 21247.Jun 24 2021, 9:34 AM
ardumont updated this revision to Diff 21284.Jun 25 2021, 3:38 PM
  • Update diff with the right commit range
  • rebase
swh-public-ci added a comment.Jun 25 2021, 3:41 PM

Build is green

Patch application report for D5919 (id=21284)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..7bbeccd
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 35 ++++++++++---
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 7 files changed, 143 insertions(+), 42 deletions(-)
Changes applied before test
commit 7bbeccd4a338ed94232294aa592e9f1cf3fa0350
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit 63ad8b9ebe0772269fb63cb608177fd28c6b0e20
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/387/ for more details.

Harbormaster completed remote builds in B22261: Diff 21284.Jun 25 2021, 3:41 PM
ardumont mentioned this in D5917: journal_client: Only check last_* fields for some permutation tests.Jun 25 2021, 5:38 PM
ardumont updated this revision to Diff 21346.Jun 29 2021, 2:20 PM

Add coverage on new backend endpoint

swh-public-ci added a comment.Jun 29 2021, 2:23 PM

Build is green

Patch application report for D5919 (id=21346)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..6ca999c
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 35 ++++++++++---
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 163 insertions(+), 42 deletions(-)
Changes applied before test
commit 6ca999c1009e7d61519275e32c4d4032da7bb548
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit 63ad8b9ebe0772269fb63cb608177fd28c6b0e20
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/388/ for more details.

Harbormaster completed remote builds in B22325: Diff 21346.Jun 29 2021, 2:23 PM
ardumont edited the summary of this revision. (Show Details)Jun 29 2021, 2:45 PM
ardumont added a project: Scheduling utilities.
ardumont updated this revision to Diff 21350.Jun 29 2021, 2:46 PM

Explicit the (ovs) journal client docstring (drop fixmes)

swh-public-ci added a comment.Jun 29 2021, 2:49 PM

Build has FAILED

Patch application report for D5919 (id=21350)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..4ed6283
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 57 ++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 182 insertions(+), 45 deletions(-)
Changes applied before test
commit 4ed62839099fd1da35456850f9d3e65a79670b8a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit bcbc732f3c8c3858df63a952933cf72cbe03e64b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 4e3bffd0e53643c999053733335b616faddccf1d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/389/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/389/console

Harbormaster failed remote builds in B22329: Diff 21350!Jun 29 2021, 2:49 PM
swh-public-ci added a comment.Jun 29 2021, 2:54 PM

Build has FAILED

Patch application report for D5919 (id=21350)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..4ed6283
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 57 ++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 182 insertions(+), 45 deletions(-)
Changes applied before test
commit 4ed62839099fd1da35456850f9d3e65a79670b8a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit bcbc732f3c8c3858df63a952933cf72cbe03e64b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit 4e3bffd0e53643c999053733335b616faddccf1d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/390/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/390/console

ardumont updated this revision to Diff 21351.Jun 29 2021, 3:16 PM

Make the doc build pass (hopefully)

swh-public-ci added a comment.Jun 29 2021, 3:19 PM

Build is green

Patch application report for D5919 (id=21351)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..4fa0b3a
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 58 +++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 183 insertions(+), 45 deletions(-)
Changes applied before test
commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/391/ for more details.

Harbormaster completed remote builds in B22331: Diff 21351.Jun 29 2021, 3:19 PM
ardumont updated this revision to Diff 21357.Jun 29 2021, 5:23 PM
  • journal_client: Compute next position for origin visit

(build will fail now)

swh-public-ci added a comment.Jun 29 2021, 5:25 PM

Build has FAILED

Patch application report for D5919 (id=21357)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..f3f5410
Fast-forward
 swh/scheduler/backend.py                   |  18 +++++
 swh/scheduler/interface.py                 |  19 +++++
 swh/scheduler/journal_client.py            |  94 ++++++++++++++++++++---
 swh/scheduler/model.py                     |   8 ++
 swh/scheduler/sql/30-schema.sql            |  19 +++++
 swh/scheduler/tests/test_api_client.py     |   2 +
 swh/scheduler/tests/test_journal_client.py | 115 ++++++++++++++++++++---------
 swh/scheduler/tests/test_scheduler.py      |  20 +++++
 8 files changed, 248 insertions(+), 47 deletions(-)
Changes applied before test
commit f3f54100ab547d75bc53e198c428351b85f5ff0c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 16:00:01 2021 +0200

    journal_client: Compute next position for origin visit

commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/392/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/392/console

Harbormaster failed remote builds in B22336: Diff 21357!Jun 29 2021, 5:25 PM
ardumont added inline comments.Jun 29 2021, 5:33 PM
swh/scheduler/journal_client.py
181–182

¯\_(ツ)_/¯

ardumont updated this revision to Diff 21358.Jun 29 2021, 5:39 PM

Fix typo (build will still fail)

swh-public-ci added a comment.Jun 29 2021, 5:43 PM

Build has FAILED

Patch application report for D5919 (id=21358)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..73c8c51
Fast-forward
 swh/scheduler/backend.py                   |  18 +++++
 swh/scheduler/interface.py                 |  19 +++++
 swh/scheduler/journal_client.py            |  94 ++++++++++++++++++++---
 swh/scheduler/model.py                     |   8 ++
 swh/scheduler/sql/30-schema.sql            |  19 +++++
 swh/scheduler/tests/test_api_client.py     |   2 +
 swh/scheduler/tests/test_journal_client.py | 115 ++++++++++++++++++++---------
 swh/scheduler/tests/test_scheduler.py      |  20 +++++
 8 files changed, 248 insertions(+), 47 deletions(-)
Changes applied before test
commit 73c8c511470bd8fa600b2e3c61e73268eee4eb73
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 16:00:01 2021 +0200

    journal_client: Compute next position for origin visit

commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/393/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/393/console

Harbormaster failed remote builds in B22337: Diff 21358!Jun 29 2021, 5:43 PM
ardumont updated this revision to Diff 21362.Jun 30 2021, 10:03 AM
ardumont added a child revision: D5950: journal_client: Compute next position for origin visit.

Drop the last commit [1] so the build stays green and the perimeter does not grow out of
hand.

[1] moved to D5950

ardumont updated this revision to Diff 21363.Jun 30 2021, 10:05 AM

Actually drop the commit

ardumont edited the summary of this revision. (Show Details)Jun 30 2021, 10:08 AM
swh-public-ci added a comment.Jun 30 2021, 10:09 AM

Build has FAILED

Patch application report for D5919 (id=21362)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..73c8c51
Fast-forward
 swh/scheduler/backend.py                   |  18 +++++
 swh/scheduler/interface.py                 |  19 +++++
 swh/scheduler/journal_client.py            |  94 ++++++++++++++++++++---
 swh/scheduler/model.py                     |   8 ++
 swh/scheduler/sql/30-schema.sql            |  19 +++++
 swh/scheduler/tests/test_api_client.py     |   2 +
 swh/scheduler/tests/test_journal_client.py | 115 ++++++++++++++++++++---------
 swh/scheduler/tests/test_scheduler.py      |  20 +++++
 8 files changed, 248 insertions(+), 47 deletions(-)
Changes applied before test
commit 73c8c511470bd8fa600b2e3c61e73268eee4eb73
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 16:00:01 2021 +0200

    journal_client: Compute next position for origin visit

commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/395/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/395/console

Harbormaster failed remote builds in B22341: Diff 21362!Jun 30 2021, 10:09 AM
swh-public-ci added a comment.Jun 30 2021, 10:11 AM

Build is green

Patch application report for D5919 (id=21363)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..4fa0b3a
Fast-forward
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 58 +++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 19 +++++++
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 8 files changed, 183 insertions(+), 45 deletions(-)
Changes applied before test
commit 4fa0b3a165658f767b2163d0ac8e6cca3eb6b184
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit b8148ed33d596dc3cc6bac5f8d266a16e4776d82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/396/ for more details.

Harbormaster completed remote builds in B22342: Diff 21363.Jun 30 2021, 10:11 AM
ardumont edited the test plan for this revision. (Show Details)Jul 1 2021, 10:25 AM
ardumont updated this revision to Diff 21379.Jul 1 2021, 10:39 AM

Add missing upgrade script

swh-public-ci added a comment.Jul 1 2021, 10:42 AM

Build is green

Patch application report for D5919 (id=21379)

Could not rebase; Attempt merge onto 1006f0aee4...

Updating 1006f0a..cb1edf1
Fast-forward
 sql/updates/29.sql                         | 27 ++++++++++
 swh/scheduler/backend.py                   | 18 +++++++
 swh/scheduler/interface.py                 | 19 +++++++
 swh/scheduler/journal_client.py            | 58 +++++++++++++++++----
 swh/scheduler/model.py                     |  8 +++
 swh/scheduler/sql/30-schema.sql            | 21 +++++++-
 swh/scheduler/tests/test_api_client.py     |  2 +
 swh/scheduler/tests/test_journal_client.py | 84 +++++++++++++++++-------------
 swh/scheduler/tests/test_scheduler.py      | 20 +++++++
 9 files changed, 211 insertions(+), 46 deletions(-)
 create mode 100644 sql/updates/29.sql
Changes applied before test
commit cb1edf1ab24d1c8db5821578a7fb2633fab50ff4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 18:07:59 2021 +0200

    Introduce storage for the recurrent visit scheduler queue position

commit ec6e69f6415a007611c46f25e7c48e909a793d53
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:42:26 2021 +0200

    Start handling of recurrent loading tasks in scheduler
    
    This deals first and foremost with the next_position_offset update done by the scheduler
    journal client.

commit c486b28ece7c0b127fea10bbb4d7f5d1ad5c50ba
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jun 29 14:41:07 2021 +0200

    journal_client: Explicit docstring

commit 98f99b9fd457820dc2d4b5dab7e89cb8261a34a4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 23 16:39:40 2021 +0200

    journal_client: Only check last_* fields for some permutation tests
    
    In a future commit, we will add new fields whose values will be permutation dependent.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/397/ for more details.

Harbormaster completed remote builds in B22358: Diff 21379.Jul 1 2021, 10:42 AM
ardumont published this revision for review.Jul 6 2021, 3:58 PM
ardumont retitled this revision from wip: Start handling of recurrent loading tasks in scheduler to Start handling of recurrent loading tasks in scheduler.
ardumont mentioned this in T2345: Improve handling of recurrent loading tasks in scheduler.Jul 8 2021, 12:30 PM
olasd accepted this revision.Jul 22 2021, 11:29 AM
This revision is now accepted and ready to land.Jul 22 2021, 11:29 AM
Closed by commit rDSCHc486b28ece7c: journal_client: Explicit docstring (authored by ardumont). · Explain WhyJul 22 2021, 2:22 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDSCHc486b28ece7c: journal_client: Explicit docstring.
ardumont added a commit: rDSCHec6e69f6415a: Start handling of recurrent loading tasks in scheduler.
ardumont added a commit: rDSCHcb1edf1ab24d: Introduce storage for the recurrent visit scheduler queue position.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
sql/
updates/
27 lines--
swh/
scheduler/
18 lines92%
Loading...
19 lines69%
Loading...
58 lines100%
Loading...
8 lines100%
Loading...
sql/
21 lines--
tests/
2 lines100%
Loading...
62 lines100%
Loading...
20 lines99%
Loading...

Diff 21379

View Options

sql/updates/29.sql

Loading...
View Options

swh/scheduler/backend.py

Loading...
View Options

swh/scheduler/interface.py

Loading...
View Options

swh/scheduler/journal_client.py

Loading...
View Options

swh/scheduler/model.py

Loading...
View Options

swh/scheduler/sql/30-schema.sql

Loading...
View Options

swh/scheduler/tests/test_api_client.py

Loading...
View Options

swh/scheduler/tests/test_journal_client.py

Loading...
View Options

swh/scheduler/tests/test_scheduler.py

Loading...
About · Developer information · Legal