lister_base: Split into chunks the tasks prior to creation
ClosedPublic
Actions

Authored by ardumont on Dec 19 2019, 10:55 AM.

Details

Reviewers

olasd
douardda

Group Reviewers

Reviewers

Commits

rDLS4b9f0e0553cd: lister_base: Split into chunks the tasks prior to creation

Summary

This should avoid the timeout-ing.

Trade off is that we create multiple transactions instead of one.

This will fix [1]

https://sentry.softwareheritage.org/share/issue/2858dec4b0794ee1a9029a080f6c7bb4/

Related to T2160

Test Plan

tox
prod: tested, the lister on the same task. It finished without any issue this time.

Diff Detail

Repository

rDLS Listers

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Dec 19 2019, 10:55 AM

ardumont edited the test plan for this revision. (Show Details)Dec 19 2019, 10:56 AM

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/516/ for more details.

Harbormaster completed remote builds in B9887: Diff 8794.Dec 19 2019, 10:58 AM

olasd accepted this revision.Dec 19 2019, 11:35 AM

This revision is now accepted and ready to land.Dec 19 2019, 11:35 AM

ardumont edited the test plan for this revision. (Show Details)Dec 19 2019, 11:48 AM

Plug to master branch

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/517/ for more details.

Harbormaster completed remote builds in B9888: Diff 8795.Dec 19 2019, 11:52 AM

Closed by commit rDLS4b9f0e0553cd: lister_base: Split into chunks the tasks prior to creation (authored by ardumont). · Explain WhyDec 19 2019, 11:53 AM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDLS4b9f0e0553cd: lister_base: Split into chunks the tasks prior to creation.

Revision Contents
Changeset List

Path

Size

swh/

lister/

core/

lister_base.py

12 lines

Diff 8796

View Options

swh/lister/core/lister_base.py

lister_base: Split into chunks the tasks prior to creationClosedPublicActions