lister_base: Split into chunks the tasks prior to creation
ClosedPublic
Actions

Authored by ardumont on Dec 19 2019, 10:55 AM.

Details

Reviewers

olasd
douardda

Group Reviewers

Reviewers

Commits

rDLS4b9f0e0553cd: lister_base: Split into chunks the tasks prior to creation

Summary

This should avoid the timeout-ing.

Trade off is that we create multiple transactions instead of one.

This will fix [1]

https://sentry.softwareheritage.org/share/issue/2858dec4b0794ee1a9029a080f6c7bb4/

Related to T2160

Test Plan

tox
prod: tested, the lister on the same task. It finished without any issue this time.

Diff Detail

Repository

rDLS Listers

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 9888
Build 14600: tox-on-jenkins	Jenkins
Build 14599: arc lint + arc unit

Event Timeline

ardumont created this revision.Dec 19 2019, 10:55 AM

ardumont edited the test plan for this revision. (Show Details)Dec 19 2019, 10:56 AM

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/516/ for more details.

Harbormaster completed remote builds in B9887: Diff 8794.Dec 19 2019, 10:58 AM

olasd accepted this revision.Dec 19 2019, 11:35 AM

This revision is now accepted and ready to land.Dec 19 2019, 11:35 AM

ardumont edited the test plan for this revision. (Show Details)Dec 19 2019, 11:48 AM

Plug to master branch

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/517/ for more details.

Harbormaster completed remote builds in B9888: Diff 8795.Dec 19 2019, 11:52 AM

Closed by commit rDLS4b9f0e0553cd: lister_base: Split into chunks the tasks prior to creation (authored by ardumont). · Explain WhyDec 19 2019, 11:53 AM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDLS4b9f0e0553cd: lister_base: Split into chunks the tasks prior to creation.

Revision Contents
Changeset List

Path

Size

swh/

lister/

core/

lister_base.py

12 lines

Diff 8795

View Options

swh/lister/core/lister_base.py

lister_base: Split into chunks the tasks prior to creationClosedPublicActions