Page MenuHomeSoftware Heritage
Differential D5330

loaders: Move retry proxy to the bottom of the stack
ClosedPublic
Actions

Authored by ardumont on Mar 25 2021, 11:54 AM.

Details

Reviewers
olasd
Group Reviewers
System administrators
Commits
rSPSITE37f30ad798ab: loaders: Move retry proxy to the bottom of the stack
Summary

There is no advantage to retry at the outer level. As the operations performed should
mostly succeed, retrying those ops would be more work for noop result. Especially, the
filter part which would make the backend compute some more for noop.

So moving the retry part at the bottom would avoid those inconveniences.

Test Plan

octo:

bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker01
Found host worker01.softwareheritage.org
WARN     -> Environment "wip-pg-hba-rules-in-yaml" contained non-word characters, correcting name to wip_pg_hba_rules_in_yaml
Cloning into '/tmp/swh-ocd.pGp6iO4k/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.pGp6iO4k/environments/staging/data/private'...
done.
*** Running octocatalog-diff on host worker01.softwareheritage.org
I, [2021-03-25T11:53:23.419622 #18542]  INFO -- : Catalogs compiled for worker01.softwareheritage.org
I, [2021-03-25T11:53:24.476583 #18542]  INFO -- : Diffs computed for worker01.softwareheritage.org
diff origin/production/worker01.softwareheritage.org current/worker01.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/loader_archive.yml] =>
   parameters =>
     content =>
      @@ -3,5 +3,4 @@
         cls: pipeline
         steps:
      -  - cls: retry
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             release: 1000
         - cls: filter
      +  - cls: retry
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_cran.yml] =>
   parameters =>
     content =>
      @@ -3,5 +3,4 @@
         cls: pipeline
         steps:
      -  - cls: retry
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             release: 1000
         - cls: filter
      +  - cls: retry
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_debian.yml] =>
   parameters =>
     content =>
      @@ -3,5 +3,4 @@
         cls: pipeline
         steps:
      -  - cls: retry
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             release: 1000
         - cls: filter
      +  - cls: retry
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_deposit.yml] =>
   parameters =>
     content =>
      @@ -3,5 +3,4 @@
         cls: pipeline
         steps:
      -  - cls: retry
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             release: 1000
         - cls: filter
      +  - cls: retry
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_git.yml] =>
   parameters =>
     content =>
      @@ -3,5 +3,4 @@
         cls: pipeline
         steps:
      -  - cls: retry
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             release: 1000
         - cls: filter
      +  - cls: retry
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_mercurial.yml] =>
   parameters =>
     content =>
      @@ -3,5 +3,4 @@
         cls: pipeline
         steps:
      -  - cls: retry
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             release: 1000
         - cls: filter
      +  - cls: retry
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_nixguix.yml] =>
   parameters =>
     content =>
      @@ -3,5 +3,4 @@
         cls: pipeline
         steps:
      -  - cls: retry
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             release: 1000
         - cls: filter
      +  - cls: retry
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_npm.yml] =>
   parameters =>
     content =>
      @@ -3,5 +3,4 @@
         cls: pipeline
         steps:
      -  - cls: retry
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             release: 1000
         - cls: filter
      +  - cls: retry
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_pypi.yml] =>
   parameters =>
     content =>
      @@ -3,5 +3,4 @@
         cls: pipeline
         steps:
      -  - cls: retry
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             release: 1000
         - cls: filter
      +  - cls: retry
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_svn.yml] =>
   parameters =>
     content =>
      @@ -3,5 +3,4 @@
         cls: pipeline
         steps:
      -  - cls: retry
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             release: 1000
         - cls: filter
      +  - cls: retry
         - cls: remote
           args:
*******************************************
*** End octocatalog-diff on worker01.softwareheritage.org

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont requested review of this revision.Mar 25 2021, 11:54 AM
ardumont created this revision.
Harbormaster completed remote builds in B20198: Diff 19111.Mar 25 2021, 11:54 AM
ardumont edited the summary of this revision. (Show Details)Mar 25 2021, 12:01 PM
ardumont added a reviewer: System administrators.
ardumont edited the summary of this revision. (Show Details)Mar 25 2021, 12:18 PM
ardumont updated this revision to Diff 19115.Mar 25 2021, 12:19 PM
ardumont edited the summary of this revision. (Show Details)

Rework commit message according to diff description

Harbormaster completed remote builds in B20202: Diff 19115.Mar 25 2021, 12:19 PM
ardumont edited the summary of this revision. (Show Details)Mar 25 2021, 12:20 PM
ardumont updated this revision to Diff 19116.Mar 25 2021, 12:23 PM

Rework some more

Harbormaster completed remote builds in B20203: Diff 19116.Mar 25 2021, 12:23 PM
ardumont edited the summary of this revision. (Show Details)Mar 25 2021, 12:23 PM
olasd accepted this revision.Mar 25 2021, 12:57 PM
This revision is now accepted and ready to land.Mar 25 2021, 12:57 PM
ardumont updated this revision to Diff 19125.Mar 25 2021, 1:49 PM
ardumont edited the summary of this revision. (Show Details)

Rebase

Harbormaster completed remote builds in B20211: Diff 19125.Mar 25 2021, 1:49 PM
Closed by commit rSPSITE37f30ad798ab: loaders: Move retry proxy to the bottom of the stack (authored by ardumont). · Explain WhyMar 25 2021, 1:50 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rSPSITE37f30ad798ab: loaders: Move retry proxy to the bottom of the stack.

Revision Contents
Changeset List

PathSize
data/
common/
2 lines

Diff 19126

View Options

data/common/common.yaml

Loading...
About · Developer information · Legal