Page MenuHomeSoftware Heritage

Split models into smaller chunks to avoid oversized db transactions
ClosedPublic

Authored by anlambert on Jun 28 2019, 3:49 PM.

Details

Summary

When testing the incoming cgit lister on that instance: http://hdiff.luite.com/cgit/,
the lister successfully retrieves all hosted repositories but fails to flush the
results into the local lister database (python3 process ended up hanging).

The issue came from the fact the underlying db transaction was too large.
The solution to resolve the issue is to split repo models into smaller
chunks to avoid that behavior. Processing like this also flushes results
to database more frequently.

Related T1659

Diff Detail

Repository
rDLS Listers
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 6557
Build 9139: tox-on-jenkinsJenkins
Build 9138: arc lint + arc unit

Event Timeline

ardumont edited the summary of this revision. (Show Details)
ardumont added a project: Lister.
This revision is now accepted and ready to land.Jun 28 2019, 3:50 PM
This revision was landed with ongoing or failed builds.Jun 28 2019, 3:51 PM
This revision was automatically updated to reflect the committed changes.