Page MenuHomeSoftware Heritage

Add a specific cooldown for notfound origins
ClosedPublic

Authored by olasd on Jun 21 2021, 6:00 PM.

Details

Summary

This allows us to avoid repeating visits on them, until a next pass of
the lister can mark them as disabled.

Depends on D5903

Test Plan

new parametric test added for this cooldown

Event Timeline

Build is green

Patch application report for D5904 (id=21182)

Could not rebase; Attempt merge onto 9e1b4145fe...

Updating 9e1b414..1ca1ac9
Fast-forward
 swh/scheduler/backend.py              |  51 +++++++++++-----
 swh/scheduler/interface.py            |   9 +++
 swh/scheduler/tests/test_scheduler.py | 112 +++++++++++++++++++++++++++++++---
 3 files changed, 149 insertions(+), 23 deletions(-)
Changes applied before test
commit 1ca1ac9e65960428e11efbaa6091554f8be30118
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jun 1 15:47:19 2021 +0200

    Add a specific cooldown for notfound origins
    
    This allows us to avoid repeating visits on them, until a next pass of
    the lister can mark them as disabled.

commit 38b88c2e2d4d01ae4f1dd2fbe023dfeaa76b3e27
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jun 21 17:34:00 2021 +0200

    Add a (longer) specific cooldown for failed origin visits

commit 73121c76c21080e1ee2dba79532d9901860f4777
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jun 21 17:36:00 2021 +0200

    Make the origin visit scheduling cooldown configurable

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/373/ for more details.

olasd requested review of this revision.Jun 21 2021, 6:06 PM
This revision is now accepted and ready to land.Jun 21 2021, 6:17 PM

Build is green

Patch application report for D5904 (id=21219)

Could not rebase; Attempt merge onto 7f51f274ed...

Updating 7f51f27..ed81870
Fast-forward
 swh/scheduler/backend.py              |  51 +++++++++++-----
 swh/scheduler/interface.py            |   9 +++
 swh/scheduler/tests/test_scheduler.py | 112 +++++++++++++++++++++++++++++++---
 3 files changed, 149 insertions(+), 23 deletions(-)
Changes applied before test
commit ed818702c49c4c29ce8f648050a92e28873944d0
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jun 1 15:47:19 2021 +0200

    Add a specific cooldown for notfound origins
    
    This allows us to avoid repeating visits on them, until a next pass of
    the lister can mark them as disabled.

commit 651ddcc6cec829429f3e449e77f2250fa1ff2a24
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jun 21 17:34:00 2021 +0200

    Add a (longer) specific cooldown for failed origin visits

commit ce8608d1f8887993ae6ddf56169cb4c117243461
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jun 21 17:36:00 2021 +0200

    Make the origin visit scheduling cooldown configurable

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/377/ for more details.