Page MenuHomeSoftware Heritage

Adapt production vault server configuration to use graph and objstorage
ClosedPublic

Authored by ardumont on Jun 28 2021, 2:09 PM.

Details

Summary

This adds one extra commit with information about the graph (no longer related to this diff).

Related to T3412

Test Plan

octo-diff, only the production vault rpc server configuration is updated to reflect the changes.

Production vault got its configuration changed:

$ bin/octocatalog-diff --to staging vangogh.euwest.azure
...
diff origin/production/vangogh.euwest.azure.internal.softwareheritage.org current/vangogh.euwest.azure.internal.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/vault/server.yml] =>
   parameters =>
     content =>
      @@ -1,3 +1,95 @@
       ---
      +objstorage:
      +  cls: multiplexer
      +  args:
      +    objstorages:
      +    - cls: filtered
      +      args:
      +        storage_conf:
      +          cls: azure-prefixed
      +          args:
      +            accounts:
      +              '0':
      +                account_name: 0euwestswh
      +                api_secret_key: swh::azure::credentials::0euwestswh
      +                container_name: contents
      +              '1':
      +                account_name: 1euwestswh
      +                api_secret_key: swh::azure::credentials::1euwestswh
      +                container_name: contents
      +              '2':
      +                account_name: 2euwestswh
      +                api_secret_key: swh::azure::credentials::2euwestswh
      +                container_name: contents
      +              '3':
      +                account_name: 3euwestswh
      +                api_secret_key: swh::azure::credentials::3euwestswh
      +                container_name: contents
      +              '4':
      +                account_name: 4euwestswh
      +                api_secret_key: swh::azure::credentials::4euwestswh
      +                container_name: contents
      +              '5':
      +                account_name: 5euwestswh
      +                api_secret_key: swh::azure::credentials::5euwestswh
      +                container_name: contents
      +              '6':
      +                account_name: 6euwestswh
      +                api_secret_key: swh::azure::credentials::6euwestswh
      +                container_name: contents
      +              '7':
      +                account_name: 7euwestswh
      +                api_secret_key: swh::azure::credentials::7euwestswh
      +                container_name: contents
      +              '8':
      +                account_name: 8euwestswh
      +                api_secret_key: swh::azure::credentials::8euwestswh
      +                container_name: contents
      +              '9':
      +                account_name: 9euwestswh
      +                api_secret_key: swh::azure::credentials::9euwestswh
      +                container_name: contents
      +              a:
      +                account_name: aeuwestswh
      +                api_secret_key: swh::azure::credentials::aeuwestswh
      +                container_name: contents
      +              b:
      +                account_name: beuwestswh
      +                api_secret_key: swh::azure::credentials::beuwestswh
      +                container_name: contents
      +              c:
      +                account_name: ceuwestswh
      +                api_secret_key: swh::azure::credentials::ceuwestswh
      +                container_name: contents
      +              d:
      +                account_name: deuwestswh
      +                api_secret_key: swh::azure::credentials::deuwestswh
      +                container_name: contents
      +              e:
      +                account_name: eeuwestswh
      +                api_secret_key: swh::azure::credentials::eeuwestswh
      +                container_name: contents
      +              f:
      +                account_name: feuwestswh
      +                api_secret_key: swh::azure::credentials::feuwestswh
      +                container_name: contents
      +        filters_conf:
      +        - type: readonly
      +    - cls: filtered
      +      args:
      +        storage_conf:
      +          cls: remote
      +          args:
      +            url: http://banco.internal.softwareheritage.org:5003/
      +        filters_conf:
      +        - type: readonly
      +    - cls: filtered
      +      args:
      +        storage_conf:
      +          cls: remote
      +          args:
      +            url: http://saam.internal.softwareheritage.org:5003/
      +        filters_conf:
      +        - type: readonly
       storage:
         cls: retry
*******************************************
*** End octocatalog-diff on vangogh.euwest.azure.internal.softwareheritage.org

The staging vault uses also the objstorage from staging:

$ bin/octocatalog-diff --to staging vault.internal.staging.swh.network
...
diff origin/production/vault.internal.staging.swh.network current/vault.internal.staging.swh.network
*******************************************
  File[/etc/softwareheritage/vault/server.yml] =>
   parameters =>
     content =>
      @@ -1,3 +1,10 @@
       ---
      +objstorage:
      +  cls: filtered
      +  storage_conf:
      +    cls: remote
      +    url: http://storage1.internal.staging.swh.network:5003/
      +  filters_conf:
      +  - type: readonly
       storage:
         cls: retry
*******************************************
*** End octocatalog-diff on vault.internal.staging.swh.network

Vault cooker configuration does not change (worker11.euwest, worker1.staging, ...):

$ bin/octocatalog-diff --to staging worker11.euwest.azure
...
*** Running octocatalog-diff on host worker11.euwest.azure.internal.softwareheritage.org
I, [2021-06-28T14:18:54.629481 #29934]  INFO -- : Catalogs compiled for worker11.euwest.azure.internal.softwareheritage.org
I, [2021-06-28T14:18:55.503144 #29934]  INFO -- : Diffs computed for worker11.euwest.azure.internal.softwareheritage.org
I, [2021-06-28T14:18:55.503185 #29934]  INFO -- : No differences
*** End octocatalog-diff on worker11.euwest.azure.internal.softwareheritage.org

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.
This revision is now accepted and ready to land.Jun 28 2021, 2:16 PM
ardumont edited the test plan for this revision. (Show Details)
ardumont retitled this revision from Adapt vault configuration to use graph and objstorage to Adapt production vault server configuration to use graph and objstorage.

Rework commit message (use correct repo)

  • Adapt production vault server configuration to use the objstorage
  • Add information about swh.graph service
ardumont edited the test plan for this revision. (Show Details)
ardumont edited the test plan for this revision. (Show Details)