Page MenuHomeSoftware Heritage
Differential D5680

Make the TenaciousProxyStorage retry when a single object add fails
ClosedPublic
Actions

Authored by douardda on May 5 2021, 12:45 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDSTO761709957a40: Make the TenaciousProxyStorage retry when a single object add fails
Summary

give a chance to one-object batches to be ingested, and reduce the
number of objects wrongly reported as non-ingested, e.g. during a
replayer session, where this situation can occur.

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.May 5 2021, 12:45 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptMay 5 2021, 12:45 PM
swh-public-ci added a comment.May 5 2021, 12:53 PM

Build is green

Patch application report for D5680 (id=20290)

Rebasing onto 77ef651d95...

Current branch diff-target is up to date.
Changes applied before test
commit 6709efea71122f5ef39daa81b2b589572ae7f64d
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed May 5 11:43:09 2021 +0200

    Make the TenaciousProxyStorage retry when a single object add fails
    
    give a chance to one-object batches to be ingested, and reduce the
    number of objects wrongly reported as non-ingested, e.g. during a
    replayer session, where this situation can occur.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1307/ for more details.

Harbormaster completed remote builds in B21276: Diff 20290.May 5 2021, 12:53 PM
douardda requested review of this revision.May 5 2021, 12:53 PM
vlorentz accepted this revision.May 6 2021, 11:51 AM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/storage/tenacious.py
123

Could you add a comment here as well, about retries being the number of retries for single objects? (or maybe rename the variable)

148–150

shouldn't it be a constant (or even a configurable value) as you're using the "same" 3 in two places?

This revision is now accepted and ready to land.May 6 2021, 11:51 AM
douardda added inline comments.May 6 2021, 2:31 PM
swh/storage/tenacious.py
148–150

yes it should...

douardda updated this revision to Diff 20353.May 6 2021, 2:47 PM

Make the number of (single-object insertion) retries configurable

swh-public-ci added a comment.May 6 2021, 2:48 PM

Build has FAILED

Patch application report for D5680 (id=20353)

Rebasing onto 652e3d5c07...

First, rewinding head to replay your work on top of it...
Applying: Make the TenaciousProxyStorage retry when a single object add fails
Changes applied before test
commit fb7ab7a1bb2a8f8e003543741f335fb15e6332f4
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed May 5 11:43:09 2021 +0200

    Make the TenaciousProxyStorage retry when a single object add fails
    
    give a chance to one-object batches to be ingested, and reduce the
    number of objects wrongly reported as non-ingested, e.g. during a
    replayer session, where this situation can occur.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1311/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1311/console

Harbormaster failed remote builds in B21335: Diff 20353!May 6 2021, 2:48 PM
douardda updated this revision to Diff 20354.May 6 2021, 2:48 PM

rebase

swh-public-ci added a comment.May 6 2021, 2:49 PM

Build has FAILED

Patch application report for D5680 (id=20354)

Rebasing onto 652e3d5c07...

Current branch diff-target is up to date.
Changes applied before test
commit bb914b4c155f8206e64e9061b99aab1787d829b2
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed May 5 11:43:09 2021 +0200

    Make the TenaciousProxyStorage retry when a single object add fails
    
    give a chance to one-object batches to be ingested, and reduce the
    number of objects wrongly reported as non-ingested, e.g. during a
    replayer session, where this situation can occur.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1312/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1312/console

Harbormaster failed remote builds in B21336: Diff 20354!May 6 2021, 2:49 PM
swh-public-ci added a comment.May 6 2021, 4:36 PM

Build is green

Patch application report for D5680 (id=20354)

Rebasing onto 35ae94a42d...

First, rewinding head to replay your work on top of it...
Applying: Make the TenaciousProxyStorage retry when a single object add fails
Changes applied before test
commit 23bbc3a93d71a41e4481b8ea2bdf532de56dfdcf
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed May 5 11:43:09 2021 +0200

    Make the TenaciousProxyStorage retry when a single object add fails
    
    give a chance to one-object batches to be ingested, and reduce the
    number of objects wrongly reported as non-ingested, e.g. during a
    replayer session, where this situation can occur.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1315/ for more details.

Harbormaster completed remote builds in B21336: Diff 20354.May 6 2021, 4:36 PM
douardda updated this revision to Diff 20392.May 7 2021, 1:47 PM

rebase

swh-public-ci added a comment.May 7 2021, 1:54 PM

Build is green

Patch application report for D5680 (id=20392)

Rebasing onto 35ae94a42d...

Current branch diff-target is up to date.
Changes applied before test
commit 761709957a403eb80c4e2ff29e46977b5b9fdd72
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed May 5 11:43:09 2021 +0200

    Make the TenaciousProxyStorage retry when a single object add fails
    
    give a chance to one-object batches to be ingested, and reduce the
    number of objects wrongly reported as non-ingested, e.g. during a
    replayer session, where this situation can occur.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1316/ for more details.

Harbormaster completed remote builds in B21371: Diff 20392.May 7 2021, 1:54 PM
Closed by commit rDSTO761709957a40: Make the TenaciousProxyStorage retry when a single object add fails (authored by douardda). · Explain WhyMay 7 2021, 1:55 PM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDSTO761709957a40: Make the TenaciousProxyStorage retry when a single object add fails.

Revision Contents
Changeset List

PathSize
swh/
storage/
44 lines
tests/
30 lines

Diff 20393

View Options

swh/storage/tenacious.py

Loading...
View Options

swh/storage/tests/test_tenacious.py

Loading...
About · Developer information · Legal