Page MenuHomeSoftware Heritage

swh-docs: Add storage sites documentation (v3)
AbandonedPublic

Authored by ftigeot on Nov 8 2019, 3:16 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Summary

Indicate where Softwareheritage stores its data, organize by
big storage locations:

  • Rocquencourt physical machines
  • Rocquencourt virtual machines
  • Azure-euwest
  • Amazon web services
  • Object storage is important and gets its own dedicated document
  • So do the hypervisors

Also link to the existing Elasticsearch document

Reviewed-by: ardumont, vlorentz

Diff Detail

Repository
rDDOC Development documentation
Branch
swh-docs-43da367ffff20ab97bfbcb4dff86ef5e9b9279da
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 8906
Build 13004: arc lint + arc unit

Event Timeline

ardumont added inline comments.
docs/infrastructure/hypervisors.rst
11

orsay?

docs/infrastructure/storage_site_azure_euwest.rst
9

journal nodes

10

I realize i don't know what those are.

11

storage/objstorage services used by the azure workers

12
  • vangogh: vault service + db (read/write) for the vault workers
13

webapp mirror of the moma one using the storage0 services to expose results.
It's not really used today.

14
  • worker01-10 + 13: indexer workers
  • worker11-12: vault worker (cooking)
25

stored*

38

I gave you some hints below those ;)

docs/infrastructure/storage_site_rocquencourt_virtual.rst
27

loader/lister workers

28

sysadm node (puppet master, grafana, primary dns, icinga2, etc...)

29

webapp and deposit services exposed publicly

Note that it'd be better if you update the diff when you update the documentation.

The command to do so:

arc diff --update <origin-diff-id> --head <commit-reference-start> <commit-reference-end>
  • origin-diff-id would be one of D2140 (v1), D2223 (v2) in the current context.

I know it's more work on the user opening the diff.
But, this:

  • avoids losing previous comments on the diff (@vlorentz and i did some on previous versions and i think i repeated some here)
  • allows us, as reviewers, to see the multiple changes incrementally (if we so wish, through the history tab in the page, after the comments).

Another approach would be to push the version one at a time, then create new diff on top of it.
Ultimately making the new diff smaller and smaller ;)
Easing the review process.

Cheers,