Adapt production vault server configuration to use graph and objstorage
ClosedPublic
Actions

Authored by ardumont on Jun 28 2021, 2:09 PM.

Details

Reviewers

vlorentz

Group Reviewers

System administrators

Maniphest Tasks

T3412: Deploy vault "git bare" tarball cooker (swh-vault v0.6)

Commits

rSPSITE57c17bb577e5: Add information about swh.graph service
rSPSITE9d0c85a87362: Adapt production vault server configuration to use the objstorage

Summary

This adds one extra commit with information about the graph (no longer related to this diff).

Related to T3412

Test Plan

octo-diff, only the production vault rpc server configuration is updated to reflect the changes.

Production vault got its configuration changed:

$ bin/octocatalog-diff --to staging vangogh.euwest.azure
...
diff origin/production/vangogh.euwest.azure.internal.softwareheritage.org current/vangogh.euwest.azure.internal.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/vault/server.yml] =>
   parameters =>
     content =>
      @@ -1,3 +1,95 @@
       ---
      +objstorage:
      +  cls: multiplexer
      +  args:
      +    objstorages:
      +    - cls: filtered
      +      args:
      +        storage_conf:
      +          cls: azure-prefixed
      +          args:
      +            accounts:
      +              '0':
      +                account_name: 0euwestswh
      +                api_secret_key: swh::azure::credentials::0euwestswh
      +                container_name: contents
      +              '1':
      +                account_name: 1euwestswh
      +                api_secret_key: swh::azure::credentials::1euwestswh
      +                container_name: contents
      +              '2':
      +                account_name: 2euwestswh
      +                api_secret_key: swh::azure::credentials::2euwestswh
      +                container_name: contents
      +              '3':
      +                account_name: 3euwestswh
      +                api_secret_key: swh::azure::credentials::3euwestswh
      +                container_name: contents
      +              '4':
      +                account_name: 4euwestswh
      +                api_secret_key: swh::azure::credentials::4euwestswh
      +                container_name: contents
      +              '5':
      +                account_name: 5euwestswh
      +                api_secret_key: swh::azure::credentials::5euwestswh
      +                container_name: contents
      +              '6':
      +                account_name: 6euwestswh
      +                api_secret_key: swh::azure::credentials::6euwestswh
      +                container_name: contents
      +              '7':
      +                account_name: 7euwestswh
      +                api_secret_key: swh::azure::credentials::7euwestswh
      +                container_name: contents
      +              '8':
      +                account_name: 8euwestswh
      +                api_secret_key: swh::azure::credentials::8euwestswh
      +                container_name: contents
      +              '9':
      +                account_name: 9euwestswh
      +                api_secret_key: swh::azure::credentials::9euwestswh
      +                container_name: contents
      +              a:
      +                account_name: aeuwestswh
      +                api_secret_key: swh::azure::credentials::aeuwestswh
      +                container_name: contents
      +              b:
      +                account_name: beuwestswh
      +                api_secret_key: swh::azure::credentials::beuwestswh
      +                container_name: contents
      +              c:
      +                account_name: ceuwestswh
      +                api_secret_key: swh::azure::credentials::ceuwestswh
      +                container_name: contents
      +              d:
      +                account_name: deuwestswh
      +                api_secret_key: swh::azure::credentials::deuwestswh
      +                container_name: contents
      +              e:
      +                account_name: eeuwestswh
      +                api_secret_key: swh::azure::credentials::eeuwestswh
      +                container_name: contents
      +              f:
      +                account_name: feuwestswh
      +                api_secret_key: swh::azure::credentials::feuwestswh
      +                container_name: contents
      +        filters_conf:
      +        - type: readonly
      +    - cls: filtered
      +      args:
      +        storage_conf:
      +          cls: remote
      +          args:
      +            url: http://banco.internal.softwareheritage.org:5003/
      +        filters_conf:
      +        - type: readonly
      +    - cls: filtered
      +      args:
      +        storage_conf:
      +          cls: remote
      +          args:
      +            url: http://saam.internal.softwareheritage.org:5003/
      +        filters_conf:
      +        - type: readonly
       storage:
         cls: retry
*******************************************
*** End octocatalog-diff on vangogh.euwest.azure.internal.softwareheritage.org

The staging vault uses also the objstorage from staging:

$ bin/octocatalog-diff --to staging vault.internal.staging.swh.network
...
diff origin/production/vault.internal.staging.swh.network current/vault.internal.staging.swh.network
*******************************************
  File[/etc/softwareheritage/vault/server.yml] =>
   parameters =>
     content =>
      @@ -1,3 +1,10 @@
       ---
      +objstorage:
      +  cls: filtered
      +  storage_conf:
      +    cls: remote
      +    url: http://storage1.internal.staging.swh.network:5003/
      +  filters_conf:
      +  - type: readonly
       storage:
         cls: retry
*******************************************
*** End octocatalog-diff on vault.internal.staging.swh.network

Vault cooker configuration does not change (worker11.euwest, worker1.staging, ...):

$ bin/octocatalog-diff --to staging worker11.euwest.azure
...
*** Running octocatalog-diff on host worker11.euwest.azure.internal.softwareheritage.org
I, [2021-06-28T14:18:54.629481 #29934]  INFO -- : Catalogs compiled for worker11.euwest.azure.internal.softwareheritage.org
I, [2021-06-28T14:18:55.503144 #29934]  INFO -- : Diffs computed for worker11.euwest.azure.internal.softwareheritage.org
I, [2021-06-28T14:18:55.503185 #29934]  INFO -- : No differences
*** End octocatalog-diff on worker11.euwest.azure.internal.softwareheritage.org

Diff Detail

Repository

rSENV Puppet Environment

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 22285
Build 34684: arc lint + arc unit

Event Timeline

ardumont requested review of this revision.Jun 28 2021, 2:09 PM

ardumont created this revision.

Harbormaster completed remote builds in B22284: Diff 21303.Jun 28 2021, 2:09 PM

ardumont edited the test plan for this revision. (Show Details)Jun 28 2021, 2:11 PM

vlorentz accepted this revision.Jun 28 2021, 2:16 PM

This revision is now accepted and ready to land.Jun 28 2021, 2:16 PM

ardumont edited the test plan for this revision. (Show Details)Jun 28 2021, 2:19 PM

ardumont edited the test plan for this revision. (Show Details)

ardumont retitled this revision from Adapt vault configuration to use graph and objstorage to Adapt production vault server configuration to use graph and objstorage.

Rework commit message

Harbormaster completed remote builds in B22285: Diff 21304.Jun 28 2021, 2:24 PM

Rework commit message (use correct repo)

Harbormaster completed remote builds in B22286: Diff 21305.Jun 28 2021, 2:24 PM

Rebase

Harbormaster completed remote builds in B22287: Diff 21306.Jun 28 2021, 2:38 PM