Differential D3986

loaders: Move the proxy storage filter after the buffer proxy
ClosedPublic
Actions

Authored by ardumont on Sep 18 2020, 2:05 PM.

Details

Reviewers

olasd

Group Reviewers

Reviewers

Maniphest Tasks

T2373: git loader OOM when loading huge repository

Commits

rSPSITE76db1fae5c5b: loaders: Move the proxy storage filter after the buffer proxy

Summary

in their pipeline configuration

context: D3976 for some DVCS loaders now send one object at a time to the
storage.

So this will allow batching calls to the *_missing endpoints (for dvcs loaders
e.g. git loader).

This slighly impacts the package loaders but this should tend towards null.

Prior to this we filtered unknown objects and kept a buffer of those unknown
objects to flush to the storage given a threshold hit.

Now, we will buffer all objects and then filter on said buffer of objects. So
we may increase calls to the *_missing endpoints.

Related to D3976
Related to T2373

Test Plan

run on staging node with this and everything runs fine (git, npm, pypi... see T2373#49135)

octocatalog

bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker01
Found host worker01.softwareheritage.org
Cloning into '/tmp/swh-ocd.lTqt9H4A/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.lTqt9H4A/environments/staging/data/private'...
done.
*** Running octocatalog-diff on host worker01.softwareheritage.org
I, [2020-09-18T14:06:13.917723 #6126]  INFO -- : Catalogs compiled for worker01.softwareheritage.org
I, [2020-09-18T14:06:14.899756 #6126]  INFO -- : Diffs computed for worker01.softwareheritage.org
diff origin/production/worker01.softwareheritage.org current/worker01.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/loader_archive.yml] =>
   parameters =>
     content =>
      @@ -4,5 +4,4 @@
         steps:
         - cls: retry
      -  - cls: filter
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             revision: 1000
             release: 1000
      +  - cls: filter
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_cran.yml] =>
   parameters =>
     content =>
      @@ -4,5 +4,4 @@
         steps:
         - cls: retry
      -  - cls: filter
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             revision: 1000
             release: 1000
      +  - cls: filter
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_debian.yml] =>
   parameters =>
     content =>
      @@ -4,5 +4,4 @@
         steps:
         - cls: retry
      -  - cls: filter
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             revision: 1000
             release: 1000
      +  - cls: filter
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_deposit.yml] =>
   parameters =>
     content =>
      @@ -4,5 +4,4 @@
         steps:
         - cls: retry
      -  - cls: filter
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             revision: 1000
             release: 1000
      +  - cls: filter
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_git.yml] =>
   parameters =>
     content =>
      @@ -4,5 +4,4 @@
         steps:
         - cls: retry
      -  - cls: filter
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             revision: 1000
             release: 1000
      +  - cls: filter
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_mercurial.yml] =>
   parameters =>
     content =>
      @@ -4,5 +4,4 @@
         steps:
         - cls: retry
      -  - cls: filter
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             revision: 1000
             release: 1000
      +  - cls: filter
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_nixguix.yml] =>
   parameters =>
     content =>
      @@ -4,5 +4,4 @@
         steps:
         - cls: retry
      -  - cls: filter
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             revision: 1000
             release: 1000
      +  - cls: filter
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_npm.yml] =>
   parameters =>
     content =>
      @@ -4,5 +4,4 @@
         steps:
         - cls: retry
      -  - cls: filter
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             revision: 1000
             release: 1000
      +  - cls: filter
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_pypi.yml] =>
   parameters =>
     content =>
      @@ -4,5 +4,4 @@
         steps:
         - cls: retry
      -  - cls: filter
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             revision: 1000
             release: 1000
      +  - cls: filter
         - cls: remote
           args:
*******************************************
  File[/etc/softwareheritage/loader_svn.yml] =>
   parameters =>
     content =>
      @@ -4,5 +4,4 @@
         steps:
         - cls: retry
      -  - cls: filter
         - cls: buffer
           min_batch_size:
      @@ -12,4 +11,5 @@
             revision: 1000
             release: 1000
      +  - cls: filter
         - cls: remote
           args:
*******************************************
*** End octocatalog-diff on worker01.softwareheritage.org

Diff Detail

Repository

rSPSITE puppet-swh-site

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Sep 18 2020, 2:05 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptSep 18 2020, 2:05 PM

Harbormaster completed remote builds in B15327: Diff 14057.Sep 18 2020, 2:05 PM

ardumont edited the summary of this revision. (Show Details)Sep 18 2020, 2:38 PM

ardumont mentioned this in D3988: loaders: Move the proxy storage filter after the buffer proxy.Sep 18 2020, 2:44 PM

ardumont edited the test plan for this revision. (Show Details)Sep 18 2020, 2:45 PM

Yeah, this definitely needs to happen before the new swh.loader.core is deployed.

This revision is now accepted and ready to land.Sep 18 2020, 2:50 PM

ardumont mentioned this in T2373: git loader OOM when loading huge repository.Sep 18 2020, 2:57 PM

ardumont edited the test plan for this revision. (Show Details)Sep 18 2020, 3:06 PM

ardumont edited the summary of this revision. (Show Details)Sep 18 2020, 3:10 PM

ardumont added a task: T2373: git loader OOM when loading huge repository.

Rework commit message according to diff description

Harbormaster completed remote builds in B15332: Diff 14062.Sep 18 2020, 3:12 PM

Closed by commit rSPSITE76db1fae5c5b: loaders: Move the proxy storage filter after the buffer proxy (authored by ardumont). · Explain WhySep 18 2020, 3:12 PM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rSPSITE76db1fae5c5b: loaders: Move the proxy storage filter after the buffer proxy.

Revision Contents
Changeset List

Path

Size

data/

defaults.yaml

2 lines

Diff 14063

View Options

loaders: Move the proxy storage filter after the buffer proxyClosedPublicActions