Page MenuHomeSoftware Heritage

lister_base: Split into chunks the tasks prior to creation
ClosedPublic

Authored by ardumont on Dec 19 2019, 10:55 AM.

Details

Summary

This should avoid the timeout-ing.

Trade off is that we create multiple transactions instead of one.

This will fix [1]

https://sentry.softwareheritage.org/share/issue/2858dec4b0794ee1a9029a080f6c7bb4/

Related to T2160

Test Plan
  • tox
  • prod: tested, the lister on the same task. It finished without any issue this time.

Diff Detail

Repository
rDLS Listers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Dec 19 2019, 10:55 AM
ardumont edited the test plan for this revision. (Show Details)Dec 19 2019, 10:56 AM
olasd accepted this revision.Dec 19 2019, 11:35 AM
This revision is now accepted and ready to land.Dec 19 2019, 11:35 AM
ardumont edited the test plan for this revision. (Show Details)Dec 19 2019, 11:48 AM
ardumont updated this revision to Diff 8795.Dec 19 2019, 11:49 AM

Plug to master branch