Page MenuHomeSoftware Heritage

Add a (longer) specific cooldown for failed origin visits
ClosedPublic

Authored by olasd on Jun 21 2021, 5:37 PM.

Details

Summary

Depends on D5901

Test Plan

tests quasi-identical to the scheduled_cooldown

Event Timeline

Build is green

Patch application report for D5903 (id=21180)

Could not rebase; Attempt merge onto 9e1b4145fe...

Updating 9e1b414..96c7c51
Fast-forward
 swh/scheduler/backend.py              |  41 ++++++----
 swh/scheduler/interface.py            |   6 ++
 swh/scheduler/tests/test_scheduler.py | 143 +++++++++++++++++++++++++++++++---
 3 files changed, 167 insertions(+), 23 deletions(-)
Changes applied before test
commit 96c7c5197463974b5e705a5f3914b7f316f23e78
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jun 21 17:34:00 2021 +0200

    Add a (longer) specific cooldown for failed origin visits

commit 73121c76c21080e1ee2dba79532d9901860f4777
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jun 21 17:36:00 2021 +0200

    Make the origin visit scheduling cooldown configurable

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/371/ for more details.

olasd requested review of this revision.Jun 21 2021, 5:41 PM
ardumont added a subscriber: ardumont.

lgtm, one question inline.

swh/scheduler/tests/test_scheduler.py
976

Does it make sense to use both cooldown in one call?

If not, can we code the exclusion logic directly within the grab_next_visits backend (making one none if the other is specified)?

This revision is now accepted and ready to land.Jun 21 2021, 5:48 PM
swh/scheduler/tests/test_scheduler.py
976

Yes, we will use them all (that's why they all have a non-zero default value too). It's just (much, much) easier to test them separately.

Use parametrization to test the logic of multiple cooldowns

Build is green

Patch application report for D5903 (id=21181)

Could not rebase; Attempt merge onto 9e1b4145fe...

Updating 9e1b414..38b88c2
Fast-forward
 swh/scheduler/backend.py              |  41 ++++++++-----
 swh/scheduler/interface.py            |   6 ++
 swh/scheduler/tests/test_scheduler.py | 112 +++++++++++++++++++++++++++++++---
 3 files changed, 136 insertions(+), 23 deletions(-)
Changes applied before test
commit 38b88c2e2d4d01ae4f1dd2fbe023dfeaa76b3e27
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jun 21 17:34:00 2021 +0200

    Add a (longer) specific cooldown for failed origin visits

commit 73121c76c21080e1ee2dba79532d9901860f4777
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jun 21 17:36:00 2021 +0200

    Make the origin visit scheduling cooldown configurable

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/372/ for more details.

Neat refacto.

swh/scheduler/tests/test_scheduler.py
976

ack, thx ;)

Build is green

Patch application report for D5903 (id=21218)

Could not rebase; Attempt merge onto 7f51f274ed...

Updating 7f51f27..651ddcc
Fast-forward
 swh/scheduler/backend.py              |  41 ++++++++-----
 swh/scheduler/interface.py            |   6 ++
 swh/scheduler/tests/test_scheduler.py | 112 +++++++++++++++++++++++++++++++---
 3 files changed, 136 insertions(+), 23 deletions(-)
Changes applied before test
commit 651ddcc6cec829429f3e449e77f2250fa1ff2a24
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jun 21 17:34:00 2021 +0200

    Add a (longer) specific cooldown for failed origin visits

commit ce8608d1f8887993ae6ddf56169cb4c117243461
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jun 21 17:36:00 2021 +0200

    Make the origin visit scheduling cooldown configurable

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/376/ for more details.