octocatalog
on 1 production worker (sample snipped but it's the same for other loaders' setup):
```
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker01
Found host worker01.softwareheritage.org
Cloning into '/tmp/swh-ocd.2dwzNJ7x/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.2dwzNJ7x/environments/staging/data/private'...
done.
*** Running octocatalog-diff on host worker01.softwareheritage.org
I, [2020-02-24T14:28:56.351019 #24865] INFO -- : Catalogs compiled for worker01.softwareheritage.org
I, [2020-02-24T14:28:57.295964 #24865] INFO -- : Diffs computed for worker01.softwareheritage.org
diff origin/production/worker01.softwareheritage.org current/worker01.softwareheritage.org
*******************************************
File[/etc/softwareheritage/loader_archive.yml] =>
parameters =>
content =>
@@ -1,22 +1,19 @@
---
storage:
- cls: retry
- args:
- storage:
- cls: filter
- args:
- storage:
- cls: buffer
- args:
- storage:
- cls: remote
- args:
- url: http://uffizi.internal.softwareheritage.org:5002/
- min_batch_size:
- content: 1000
- content_bytes: 52428800
- directory: 1000
- revision: 1000
- release: 1000
+ cls: pipeline
+ steps:
+ - cls: validate
+ - cls: retry
+ - cls: filter
+ - cls: buffer
+ min_batch_size:
+ content: 1000
+ content_bytes: 52428800
+ directory: 1000
+ revision: 1000
+ release: 1000
+ - cls: remote
+ args:
+ url: http://uffizi.internal.softwareheritage.org:5002/
...
```
Same goes for staging workers:
```
File[/etc/softwareheritage/loader_pypi.yml] =>
parameters =>
content =>
@@ -1,22 +1,19 @@
---
storage:
- cls: retry
- args:
- storage:
- cls: filter
- args:
- storage:
- cls: buffer
- args:
- storage:
- cls: remote
- args:
- url: http://storage0.internal.staging.swh.network:5002/
- min_batch_size:
- content: 1000
- content_bytes: 52428800
- directory: 1000
- revision: 1000
- release: 1000
+ cls: pipeline
+ steps:
+ - cls: validate
+ - cls: retry
+ - cls: filter
+ - cls: buffer
+ min_batch_size:
+ content: 1000
+ content_bytes: 52428800
+ directory: 1000
+ revision: 1000
+ release: 1000
+ - cls: remote
+ args:
+ url: http://storage0.internal.staging.swh.network:5002/
max_content_size: 104857600
celery:
*******************************************
*** End octocatalog-diff on worker0.internal.staging.swh.network
```