Page MenuHomeSoftware Heritage

storages: Add model validation server side as well
AbandonedPublic

Authored by ardumont on Feb 27 2020, 10:54 AM.

Details

Reviewers
vlorentz
olasd
douardda
Group Reviewers
Reviewers
Summary

Belts and whistles!

Test Plan

octoctalog

$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging storage0.internal.staging.swh.network
Found host storage0.internal.staging.swh.network
Cloning into '/tmp/swh-ocd.7pMGQikp/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.7pMGQikp/environments/staging/data/private'...
done.
*** Running octocatalog-diff on host storage0.internal.staging.swh.network
I, [2020-02-27T10:42:12.080252 #30133]  INFO -- : Catalogs compiled for storage0.internal.staging.swh.network
I, [2020-02-27T10:42:13.221729 #30133]  INFO -- : Diffs computed for storage0.internal.staging.swh.network
diff origin/production/storage0.internal.staging.swh.network current/storage0.internal.staging.swh.network
*******************************************
  File[/etc/softwareheritage/storage/storage.yml] =>
   parameters =>
     content =>
      @@ -1,6 +1,8 @@
       ---
       storage:
      -  cls: local
      -  args:
      +  cls: pipeline
      +  steps:
      +  - cls: validate
      +  - cls: local
           db: host=db0.internal.staging.swh.network port=5432 user=swh dbname=swh password=swh::deploy::storage::db::password
           objstorage:
      @@ -9,12 +11,12 @@
               root: "/srv/softwareheritage/objects"
               slicing: 0:1/1:5
      -journal_writer:
      -  cls: kafka
      -  args:
      -    brokers:
      -    - journal0.internal.staging.swh.network
      -    prefix: swh.journal.objects
      -    client_id: swh.storage.journal_writer.storage0
      -    producer_config:
      -      message.max.bytes: 1000000000
      +    journal_writer:
      +      cls: kafka
      +      args:
      +        brokers:
      +        - journal0.internal.staging.swh.network
      +        prefix: swh.journal.objects
      +        client_id: swh.storage.journal_writer.storage0
      +        producer_config:
      +          message.max.bytes: 1000000000
      _
*******************************************
*** End octocatalog-diff on storage0.internal.staging.swh.network

$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging uffizi
Found host uffizi.softwareheritage.org
Cloning into '/tmp/swh-ocd.kVY7XoPT/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.kVY7XoPT/environments/staging/data/private'...
done.
*** Running octocatalog-diff on host uffizi.softwareheritage.org
I, [2020-02-27T10:46:18.673130 #2782]  INFO -- : Catalogs compiled for uffizi.softwareheritage.org
I, [2020-02-27T10:46:19.830727 #2782]  INFO -- : Diffs computed for uffizi.softwareheritage.org
diff origin/production/uffizi.softwareheritage.org current/uffizi.softwareheritage.org
*******************************************
- File[/etc/softwareheritage/journal]
*******************************************
  File[/etc/softwareheritage/storage/storage.yml] =>
   parameters =>
     content =>
      @@ -1,104 +1,107 @@
       ---
       storage:
      -  cls: local
      -  args:
      -    db: host=db.internal.softwareheritage.org user=swhstorage dbname=softwareheritage
      -      password=swh::deploy::storage::db::password
      -    objstorage:
      -      cls: multiplexer
      -      args:
      -        objstorages:
      -        - cls: pathslicing
      -          args:
      -            root: "/srv/softwareheritage/objects"
      -            slicing: 0:2/0:5
      -            compression: none
      -        - cls: filtered
      -          args:
      -            storage_conf:
      -              cls: pathslicing
      -              args:
      -                root: "/srv/softwareheritage/objects-xfs"
      -                slicing: 0:1/0:2/2:4/4:6
      -                compression: gzip
      -            filters_conf:
      -            - type: readonly
      -        - cls: azure-prefixed
      -          args:
      -            accounts:
      -              '0':
      -                account_name: 0euwestswh
      -                api_secret_key: swh::azure::credentials::0euwestswh
      -                container_name: contents
      -              '1':
      -                account_name: 1euwestswh
      -                api_secret_key: swh::azure::credentials::1euwestswh
      -                container_name: contents
      -              '2':
      -                account_name: 2euwestswh
      -                api_secret_key: swh::azure::credentials::2euwestswh
      -                container_name: contents
      -              '3':
      -                account_name: 3euwestswh
      -                api_secret_key: swh::azure::credentials::3euwestswh
      -                container_name: contents
      -              '4':
      -                account_name: 4euwestswh
      -                api_secret_key: swh::azure::credentials::4euwestswh
      -                container_name: contents
      -              '5':
      -                account_name: 5euwestswh
      -                api_secret_key: swh::azure::credentials::5euwestswh
      -                container_name: contents
      -              '6':
      -                account_name: 6euwestswh
      -                api_secret_key: swh::azure::credentials::6euwestswh
      -                container_name: contents
      -              '7':
      -                account_name: 7euwestswh
      -                api_secret_key: swh::azure::credentials::7euwestswh
      -                container_name: contents
      -              '8':
      -                account_name: 8euwestswh
      -                api_secret_key: swh::azure::credentials::8euwestswh
      -                container_name: contents
      -              '9':
      -                account_name: 9euwestswh
      -                api_secret_key: swh::azure::credentials::9euwestswh
      -                container_name: contents
      -              a:
      -                account_name: aeuwestswh
      -                api_secret_key: swh::azure::credentials::aeuwestswh
      -                container_name: contents
      -              b:
      -                account_name: beuwestswh
      -                api_secret_key: swh::azure::credentials::beuwestswh
      -                container_name: contents
      -              c:
      -                account_name: ceuwestswh
      -                api_secret_key: swh::azure::credentials::ceuwestswh
      -                container_name: contents
      -              d:
      -                account_name: deuwestswh
      -                api_secret_key: swh::azure::credentials::deuwestswh
      -                container_name: contents
      -              e:
      -                account_name: eeuwestswh
      -                api_secret_key: swh::azure::credentials::eeuwestswh
      -                container_name: contents
      -              f:
      -                account_name: feuwestswh
      -                api_secret_key: swh::azure::credentials::feuwestswh
      -                container_name: contents
      -    journal_writer:
      -      cls: kafka
      -      args:
      -        brokers:
      -        - esnode1.internal.softwareheritage.org
      -        - esnode2.internal.softwareheritage.org
      -        - esnode3.internal.softwareheritage.org
      -        prefix: swh.journal.objects
      -        client_id: swh.storage.journal_writer.uffizi
      -        producer_config:
      -          message.max.bytes: 1000000000
      +  cls: pipeline
      +  steps:
      +  - cls: validate
      +  - cls: local
      +    args:
      +      db: host=db.internal.softwareheritage.org user=swhstorage dbname=softwareheritage
      +        password=swh::deploy::storage::db::password
      +      objstorage:
      +        cls: multiplexer
      +        args:
      +          objstorages:
      +          - cls: pathslicing
      +            args:
      +              root: "/srv/softwareheritage/objects"
      +              slicing: 0:2/0:5
      +              compression: none
      +          - cls: filtered
      +            args:
      +              storage_conf:
      +                cls: pathslicing
      +                args:
      +                  root: "/srv/softwareheritage/objects-xfs"
      +                  slicing: 0:1/0:2/2:4/4:6
      +                  compression: gzip
      +              filters_conf:
      +              - type: readonly
      +          - cls: azure-prefixed
      +            args:
      +              accounts:
      +                '0':
      +                  account_name: 0euwestswh
      +                  api_secret_key: swh::azure::credentials::0euwestswh
      +                  container_name: contents
      +                '1':
      +                  account_name: 1euwestswh
      +                  api_secret_key: swh::azure::credentials::1euwestswh
      +                  container_name: contents
      +                '2':
      +                  account_name: 2euwestswh
      +                  api_secret_key: swh::azure::credentials::2euwestswh
      +                  container_name: contents
      +                '3':
      +                  account_name: 3euwestswh
      +                  api_secret_key: swh::azure::credentials::3euwestswh
      +                  container_name: contents
      +                '4':
      +                  account_name: 4euwestswh
      +                  api_secret_key: swh::azure::credentials::4euwestswh
      +                  container_name: contents
      +                '5':
      +                  account_name: 5euwestswh
      +                  api_secret_key: swh::azure::credentials::5euwestswh
      +                  container_name: contents
      +                '6':
      +                  account_name: 6euwestswh
      +                  api_secret_key: swh::azure::credentials::6euwestswh
      +                  container_name: contents
      +                '7':
      +                  account_name: 7euwestswh
      +                  api_secret_key: swh::azure::credentials::7euwestswh
      +                  container_name: contents
      +                '8':
      +                  account_name: 8euwestswh
      +                  api_secret_key: swh::azure::credentials::8euwestswh
      +                  container_name: contents
      +                '9':
      +                  account_name: 9euwestswh
      +                  api_secret_key: swh::azure::credentials::9euwestswh
      +                  container_name: contents
      +                a:
      +                  account_name: aeuwestswh
      +                  api_secret_key: swh::azure::credentials::aeuwestswh
      +                  container_name: contents
      +                b:
      +                  account_name: beuwestswh
      +                  api_secret_key: swh::azure::credentials::beuwestswh
      +                  container_name: contents
      +                c:
      +                  account_name: ceuwestswh
      +                  api_secret_key: swh::azure::credentials::ceuwestswh
      +                  container_name: contents
      +                d:
      +                  account_name: deuwestswh
      +                  api_secret_key: swh::azure::credentials::deuwestswh
      +                  container_name: contents
      +                e:
      +                  account_name: eeuwestswh
      +                  api_secret_key: swh::azure::credentials::eeuwestswh
      +                  container_name: contents
      +                f:
      +                  account_name: feuwestswh
      +                  api_secret_key: swh::azure::credentials::feuwestswh
      +                  container_name: contents
      +      journal_writer:
      +        cls: kafka
      +        args:
      +          brokers:
      +          - esnode1.internal.softwareheritage.org
      +          - esnode2.internal.softwareheritage.org
      +          - esnode3.internal.softwareheritage.org
      +          prefix: swh.journal.objects
      +          client_id: swh.storage.journal_writer.uffizi
      +          producer_config:
      +            message.max.bytes: 1000000000
      _
*******************************************
- Package[python3-swh.journal]
*******************************************
*** End octocatalog-diff on uffizi.softwareheritage.org

Diff Detail

Repository
rSPSITE puppet-swh-site
Branch
staging
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 10768
Build 16165: arc lint + arc unit

Event Timeline

Not yet.

To expand on the irl discussion:

  • the current validation proxy only converts raw dicts to model objects; if the clients already do that, then the RPC layer will pass these objects through
  • we probably want some integrity checking on the server side instead (i.e. checking that the pids of objects we're inserting are legit; that's sort of a last defense against T75)