pattern: Make cgit/launchpad listers flush regularly origins to scheduler
AbandonedPublic
Actions

Authored by ardumont on Jan 29 2021, 4:44 PM.

Details

Reviewers

None

Group Reviewers

Reviewers

Summary

This reverts the pattern pushed in the main Lister class and defines it specifically in
the listers that were identified as needing it:

The launchpad lister has just one iterator of origins so this needs to be splitted to have incremental results flushed to the scheduler [1]

For cgit, it's mostly for instances which returns a high number of results per page (e.g git.eclipse.org, ...). [2]

[1] T2998#57546

[2] T2998#57500

[3] T376#57560

Diff Detail

Repository

rDLS Listers

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 18895
Build 29274: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 29273: arc lint + arc unit

Event Timeline

ardumont created this revision.Jan 29 2021, 4:44 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJan 29 2021, 4:44 PM

Build is green

Patch application report for D4972 (id=17754)

Rebasing onto 2e22073558...

Current branch diff-target is up to date.

Changes applied before test

commit df54cb57048316854fad1e69fca1873a07b2b51c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jan 29 16:33:05 2021 +0100

    pattern: Make cgit/launchpad listers flush regularly origins to scheduler
    
    This reverts the pattern pushed in the main Lister class and defines it specifically in
    the listers that were identified as needing it:
    
    - The launchpad lister has just one iterator of origins so this needs to be splitted to
      have incremental results flushed to the scheduler.
    
    - For cgit, it's mostly for instances which returns a high number of results per
      page (e.g git.eclispe.org).

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/231/ for more details.

Harbormaster completed remote builds in B18895: Diff 17754.Jan 29 2021, 4:47 PM

ardumont requested review of this revision.Jan 29 2021, 4:47 PM

ardumont edited the summary of this revision. (Show Details)Jan 29 2021, 4:49 PM

ardumont added inline comments.

swh/lister/cgit/lister.py
208	It may be worth pushing this in the `swh.scheduler.pattern.StatelessLister` class. For now, settled on only cgit and launchpad as they were the most flagrant ones needing it.