loaders: Add validation proxy setup
ClosedPublic
Actions

Authored by ardumont on Feb 24 2020, 2:38 PM.

Details

Reviewers

vlorentz
olasd

Group Reviewers

Reviewers

Commits

rSPSITE958beea6148e: loaders: Add validation proxy setup

Summary

Note that this also starts using the pipeline setup stanza.

Test Plan

octocatalog

on 1 production worker (sample snipped but it's the same for other loaders' setup):

$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker01
Found host worker01.softwareheritage.org
Cloning into '/tmp/swh-ocd.2dwzNJ7x/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.2dwzNJ7x/environments/staging/data/private'...
done.
*** Running octocatalog-diff on host worker01.softwareheritage.org
I, [2020-02-24T14:28:56.351019 #24865]  INFO -- : Catalogs compiled for worker01.softwareheritage.org
I, [2020-02-24T14:28:57.295964 #24865]  INFO -- : Diffs computed for worker01.softwareheritage.org
diff origin/production/worker01.softwareheritage.org current/worker01.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/loader_archive.yml] =>
   parameters =>
     content =>
      @@ -1,22 +1,19 @@
       ---
       storage:
      -  cls: retry
      -  args:
      -    storage:
      -      cls: filter
      -      args:
      -        storage:
      -          cls: buffer
      -          args:
      -            storage:
      -              cls: remote
      -              args:
      -                url: http://uffizi.internal.softwareheritage.org:5002/
      -            min_batch_size:
      -              content: 1000
      -              content_bytes: 52428800
      -              directory: 1000
      -              revision: 1000
      -              release: 1000
      +  cls: pipeline
      +  steps:
      +  - cls: validate
      +  - cls: retry
      +  - cls: filter
      +  - cls: buffer
      +    min_batch_size:
      +      content: 1000
      +      content_bytes: 52428800
      +      directory: 1000
      +      revision: 1000
      +      release: 1000
      +  - cls: remote
      +    args:
      +      url: http://uffizi.internal.softwareheritage.org:5002/
...

Same goes for staging workers:

  File[/etc/softwareheritage/loader_pypi.yml] =>
   parameters =>
     content =>
      @@ -1,22 +1,19 @@
       ---
       storage:
      -  cls: retry
      -  args:
      -    storage:
      -      cls: filter
      -      args:
      -        storage:
      -          cls: buffer
      -          args:
      -            storage:
      -              cls: remote
      -              args:
      -                url: http://storage0.internal.staging.swh.network:5002/
      -            min_batch_size:
      -              content: 1000
      -              content_bytes: 52428800
      -              directory: 1000
      -              revision: 1000
      -              release: 1000
      +  cls: pipeline
      +  steps:
      +  - cls: validate
      +  - cls: retry
      +  - cls: filter
      +  - cls: buffer
      +    min_batch_size:
      +      content: 1000
      +      content_bytes: 52428800
      +      directory: 1000
      +      revision: 1000
      +      release: 1000
      +  - cls: remote
      +    args:
      +      url: http://storage0.internal.staging.swh.network:5002/
       max_content_size: 104857600
       celery:
*******************************************
*** End octocatalog-diff on worker0.internal.staging.swh.network