Differential D5785

sourceforge: retry for all retryable exceptions
ClosedPublic
Actions

Authored by Alphare on May 26 2021, 12:10 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T3310: Deploy sourceforge lister on staging

Commits

rDLS9ca5295a408b: sourceforge: retry for all retryable exceptions

Summary

Since this lister is doing a lot more requests than most other, it makes
sense that issues would arise more often. We want the lister to continue
even if the website is having issues and not break on the first 500 or
closed connection it encounters.

This change introduces a mechanism to retry all exceptions worth
retrying and uses it for the SourceForge lister. Other listers might
benefit from this, but this is out of scope here.

Tests had to be adjusted to stub the sleep function since retries happened
way more often.

Related to T3310

Diff Detail

Repository

rDLS Listers

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Alphare created this revision.May 26 2021, 12:10 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptMay 26 2021, 12:10 PM

Build is green

Patch application report for D5785 (id=20682)

Rebasing onto 8f3bbacd5e...

Current branch diff-target is up to date.

Changes applied before test

commit 9ca5295a408b7f1f5be3a39ca7c4e00524dab4ac
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed May 26 12:05:39 2021 +0200

    sourceforge: retry for all retryable exceptions
    
    Since this lister is doing a lot more requests than most other, it makes
    sense that issues would arise more often. We want the lister to continue
    even if the website is having issues and not break on the first 500 or
    closed connection it encounters.
    
    This change introduces a mechanism to retry all exceptions worth
    retrying and uses it for the SourceForge lister. Other listers might
    benefit from this, but this is out of scope here.
    
    Tests had to be adjusted to stub the sleep function since retries happened
    way more often.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/295/ for more details.

Harbormaster completed remote builds in B21646: Diff 20682.May 26 2021, 12:14 PM

Alphare requested review of this revision.May 26 2021, 12:14 PM

Alphare mentioned this in T3310: Deploy sourceforge lister on staging.May 26 2021, 12:15 PM

ardumont edited the summary of this revision. (Show Details)May 26 2021, 12:16 PM

ardumont added a task: T3310: Deploy sourceforge lister on staging.