Page MenuHomeSoftware Heritage

Vault: Migrate vault infrastructure to azure
Closed, MigratedEdits Locked

Description

Except for cooking tasks (running on azure), the remaining services runs on the unibo infra (node: orangerie):

  • db (postgresql: swhvault)
  • api (gunicorn-swh-vault)
  • cache (gunicorn-swh-objstorage)

As the db is not that big, it can be moved to prado's secondary cluster.

Other services can move to a dedicated azure vm.

For the objstorage, the following should be enough:

  • Performance/Access tier: Standard/Cold
  • Replication: Locally-redundant storage (LRS)
  • Account kind: BlobStorage

Event Timeline

ardumont triaged this task as Normal priority.May 15 2019, 11:51 AM
ardumont created this task.
ardumont renamed this task from Vault: Move vault infrastructure to azure to Vault: Migrate vault infrastructure to azure.May 15 2019, 11:55 AM

objstorage/cache

ardumont@orangerie:~% sudo df -h /srv/softwareheritage/vault_cache/
Filesystem                          Size  Used Avail Use% Mounted on
10.100.0.1:/Q/shares/swh/orangerie   59T  5.5T   54T  10% /srv/softwareheritage

db

ncdu 1.12 ~ Use the arrow keys to navigate, press ? for help
--- /var/lib/postgresql -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    1.1 GiB [##########] /11
...

Considering the size of that database, and the fact that we don't have any provisions to automatically spin up a new database server, I think it would make more sense to repatriate it on our main postgres setup, rather than movig it to a new machine on azure.

To be clear, that would mean moving the database on the secondary postgres cluster on prado.

To be clear, that would mean moving the database on the secondary postgres cluster on prado.

Ok.

ardumont updated the task description. (Show Details)

Created the db on prado on secondary cluster (i normalized the user name from swhvault to swh-vault, same for the db name):

$ createuser -p 5434 -P swh-vault
$ createdb -p 5434 --lc-ctype=C.UTF-8 -T template1 swh-vault -O swh-vault
$ cat swh-vault.dump.sql | psql -p 5434 swh-vault

note:

  • swh-vault.dump.sql was created from orangerie's db (pg_dump swhvault > swh-vault.dump.sql)
  • then adapted to change the name to "swh-vault"

Updating the reference to orangerie's api to target the new db instance on prado
Related af9cd6965d7f01f2d20d735b5e812398bc130c38

Note: orangeriedev got its db updated to swh-vault as well.

ardumont updated the task description. (Show Details)
ardumont changed the task status from Open to Work in Progress.May 21 2019, 2:51 PM

So, as in D1495:

$ terraform apply
$ cd sysadm-provisioning
$ scp azure/provision-vm.sh ardumont@<ip>:/tmp/provision-vm.sh
$ ssh ardumont@<ip>
> chmod +x /tmp/provision-vm.sh
> sudo /tmp/provision-vm.sh private
$ ssh louvre
  > sudo clush -b -w ns0.euwest.azure -w pergamon puppet agent --test

Now still heading towards swh-site to make the vault configuration converge according to the new resources created.

Heads up, checking the new vault's objstorage speaking to azure blob storage is fine (credential and all):

  • using the docker environment with a vault plugged to the new azure's blobstorage
  • load data in the docker env
  • request a cooking in the docker webapp
  • check the cooking is ok (it is)
  • check the download of the cooking is ok (it is)
  • check the azure blob storage's new blob (it is and it's the same as the one from the local webapp).

So i'm now ready to actually migrate the webapp's setup regarding vault from orangerie to vangogh.

Webapp/cookers migrated to use the azure vault instance.

As per oral/irc discussion with @moranegg @douardda, the existing cooked data can be migrated next week [1]

[1] irc excerpt:

14:29 <douardda> moranegg[m]: do we know if we can start the (new) vault without cooked archives?
14:36 <moranegg[m]> douardda: no response. but as we said face to face- go go go!

Webapp/cookers migrated to use the azure vault instance.

Great!

As per oral/irc discussion with @moranegg @douardda, the existing cooked data can be migrated next week [1]

I'd say don't bother migrating old archives. Not migrating them will give us an opportunity to reassess the actual vault usage starting form a clean slate, and also reduce the storage cost.