Page MenuHomeSoftware Heritage

Make the TenaciousProxyStorage retry when a single object add fails
ClosedPublic

Authored by douardda on May 5 2021, 12:45 PM.

Details

Summary

give a chance to one-object batches to be ingested, and reduce the
number of objects wrongly reported as non-ingested, e.g. during a
replayer session, where this situation can occur.

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D5680 (id=20290)

Rebasing onto 77ef651d95...

Current branch diff-target is up to date.
Changes applied before test
commit 6709efea71122f5ef39daa81b2b589572ae7f64d
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed May 5 11:43:09 2021 +0200

    Make the TenaciousProxyStorage retry when a single object add fails
    
    give a chance to one-object batches to be ingested, and reduce the
    number of objects wrongly reported as non-ingested, e.g. during a
    replayer session, where this situation can occur.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1307/ for more details.

vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/storage/tenacious.py
123

Could you add a comment here as well, about retries being the number of retries for single objects? (or maybe rename the variable)

148–150

shouldn't it be a constant (or even a configurable value) as you're using the "same" 3 in two places?

This revision is now accepted and ready to land.May 6 2021, 11:51 AM
swh/storage/tenacious.py
148–150

yes it should...

Make the number of (single-object insertion) retries configurable

Build has FAILED

Patch application report for D5680 (id=20353)

Rebasing onto 652e3d5c07...

First, rewinding head to replay your work on top of it...
Applying: Make the TenaciousProxyStorage retry when a single object add fails
Changes applied before test
commit fb7ab7a1bb2a8f8e003543741f335fb15e6332f4
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed May 5 11:43:09 2021 +0200

    Make the TenaciousProxyStorage retry when a single object add fails
    
    give a chance to one-object batches to be ingested, and reduce the
    number of objects wrongly reported as non-ingested, e.g. during a
    replayer session, where this situation can occur.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1311/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1311/console

Build has FAILED

Patch application report for D5680 (id=20354)

Rebasing onto 652e3d5c07...

Current branch diff-target is up to date.
Changes applied before test
commit bb914b4c155f8206e64e9061b99aab1787d829b2
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed May 5 11:43:09 2021 +0200

    Make the TenaciousProxyStorage retry when a single object add fails
    
    give a chance to one-object batches to be ingested, and reduce the
    number of objects wrongly reported as non-ingested, e.g. during a
    replayer session, where this situation can occur.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1312/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1312/console

Build is green

Patch application report for D5680 (id=20354)

Rebasing onto 35ae94a42d...

First, rewinding head to replay your work on top of it...
Applying: Make the TenaciousProxyStorage retry when a single object add fails
Changes applied before test
commit 23bbc3a93d71a41e4481b8ea2bdf532de56dfdcf
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed May 5 11:43:09 2021 +0200

    Make the TenaciousProxyStorage retry when a single object add fails
    
    give a chance to one-object batches to be ingested, and reduce the
    number of objects wrongly reported as non-ingested, e.g. during a
    replayer session, where this situation can occur.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1315/ for more details.

Build is green

Patch application report for D5680 (id=20392)

Rebasing onto 35ae94a42d...

Current branch diff-target is up to date.
Changes applied before test
commit 761709957a403eb80c4e2ff29e46977b5b9fdd72
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed May 5 11:43:09 2021 +0200

    Make the TenaciousProxyStorage retry when a single object add fails
    
    give a chance to one-object batches to be ingested, and reduce the
    number of objects wrongly reported as non-ingested, e.g. during a
    replayer session, where this situation can occur.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1316/ for more details.