Paths

Table of Contentst

recurrent_visits: Allow to set no origins scheduled backoff in config
ClosedPublic
Actions

Authored by anlambert on Sep 14 2022, 4:27 PM.

Details

Reviewers

vlorentz
ardumont

Group Reviewers

Reviewers

Commits

rDSCHb1afdab92003: recurrent_visits: Allow to set no origins scheduled backoff in config

Summary

The send_visits_for_visit_type function uses a default schedule backoff
of 20 minutes where there is no origins to schedule for a given visit
type.

It exists use cases when we would like that schedule backoff to be
shorter in order to schedule listed origins for loading into the
archive more rapidly, typically in the docker environment.

So allow to set that backoff value through configuration.

The purpose of that diff is to schedule loading tasks for listed
origins in the docker environment without having to restart the
swh-scheduler-schedule-recurrent service.

Test Plan

I did what I could to test this but things are hard to mock here
as it involves threading.

Diff Detail

Repository

rDSCH Scheduling utilities

Branch

config-no-origins-scheduled-backoff

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 31532
Build 49326: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 49325: arc lint + arc unit

Event Timeline

anlambert created this revision.Sep 14 2022, 4:27 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptSep 14 2022, 4:27 PM

Build is green

Patch application report for D8475 (id=30531)

Rebasing onto 7cfaa986c2...

Current branch diff-target is up to date.

Changes applied before test

commit e086db1855f00f2212b7f605086ccffcc200a9a2
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Sep 14 16:18:51 2022 +0200

    recurrent_visits: Allow to set no origins scheduled backoff in config
    
    The send_visits_for_visit_type function uses a default schedule backoff
    of 20 minutes where there is no origins to schedule for a given visit
    type.
    
    It exists use cases when we would like that schedule backoff to be
    shorter in order to schedule listed origins for loading into the
    archive more rapidly, typically in the docker environment.
    
    So allow to set that backoff value through configuration.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/559/ for more details.

Harbormaster completed remote builds in B31532: Diff 30531.Sep 14 2022, 4:32 PM

anlambert requested review of this revision.Sep 14 2022, 4:32 PM

vlorentz added a subscriber: vlorentz.Sep 15 2022, 9:02 AM

vlorentz added inline comments.

swh/scheduler/celery_backend/recurrent_visits.py
71	should be renamed to `DEFAULT_NO_ORIGINS_SCHEDULED_BACKOFF`

vlorentz requested changes to this revision.Sep 15 2022, 9:02 AM

This revision now requires changes to proceed.Sep 15 2022, 9:02 AM

Rename global variable

Build is green

Patch application report for D8475 (id=30543)

Rebasing onto 7cfaa986c2...

Current branch diff-target is up to date.

Changes applied before test

commit b1afdab9200327cdff89a2377525317ae736bcb1
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Sep 14 16:18:51 2022 +0200

    recurrent_visits: Allow to set no origins scheduled backoff in config
    
    The send_visits_for_visit_type function uses a default schedule backoff
    of 20 minutes where there is no origins to schedule for a given visit
    type.
    
    It exists use cases when we would like that schedule backoff to be
    shorter in order to schedule listed origins for loading into the
    archive more rapidly, typically in the docker environment.
    
    So allow to set that backoff value through configuration.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/560/ for more details.

Harbormaster completed remote builds in B31544: Diff 30543.Sep 15 2022, 10:46 AM

ardumont accepted this revision.Sep 15 2022, 1:48 PM

This revision was not accepted when it landed; it landed in state Needs Review.Sep 15 2022, 1:55 PM

Closed by commit rDSCHb1afdab92003: recurrent_visits: Allow to set no origins scheduled backoff in config (authored by anlambert). · Explain Why

This revision was automatically updated to reflect the committed changes.

anlambert added a commit: rDSCHb1afdab92003: recurrent_visits: Allow to set no origins scheduled backoff in config.

anlambert mentioned this in D8486: docker/conf/scheduler: Set no_origins_scheduled_backoff to 1 minute.Sep 15 2022, 2:38 PM