Page MenuHomeSoftware Heritage
Create Task
Maniphest T2561

Sort out disk space on beaubourg / somerset
Closed, MigratedEdits Locked
Actions

Description

beaubourg has run out of disk space on the postgresql partition. This broke logical replication between belvedere and somerset, and probably the main indexer database too.

Ideally, now that the VM storage has been moved to ceph on dedicated SSDs, we should migrate this local postgres storage to ZFS.

One way we could do this:

  • file-level copy of the data to a dedicated ceph storage space
  • tear down the current lvm / raid pile
  • initialize a new zfs pool
  • copy the data back

We might want to also migrate the main indexer database to belvedere, as it still has some spare SSDs (T2440)

Related Objects
Search...

Event Timeline

olasd triaged this task as High priority.Sep 3 2020, 4:27 PM
olasd created this task.
vsellier added a subscriber: vsellier.Sep 3 2020, 5:28 PM
ardumont closed subtask T2562: Point web frontend to belvedere temporarily as Resolved.Sep 4 2020, 10:32 AM
olasd added a comment.Sep 4 2020, 3:19 PM

Current disk usage on somerset:

--- /srv/softwareheritage/postgres/11 -----------------------------------------
                         /..
    9.4 TiB [##########] /replica
    2.3 TiB [##        ] /indexer
  904.2 GiB [          ] /testdedup                                                                                                                                                                                                                                                                                          
  232.2 GiB [          ] /occurrences

belvedere has 5.7TB free before compression, and still has a bunch of unused disks. So we're definitely able to move the indexer database there.

olasd added a subscriber: seirl.Sep 8 2020, 2:54 PM

I've had agreement with Guillaume about making the occurrences db a static dump;

I need @seirl to confirm what to do with the testdedup cluster.

I've started looking at T2440 again because we're going to be quite tight on disk space on belvedere before being able to migrate the data of the indexer cluster (we have around 4TB of WALs piled up in the main cluster).

olasd created subtask T2576: Migrate dbreplica0.euwest.azure to zfs.Sep 8 2020, 3:16 PM
olasd added a comment.Sep 8 2020, 5:34 PM

I've started migrating the data of the indexer cluster now that disk space on belvedere has been sorted out.

On belvedere:

zfs create data/postgres-indexer-11 -o mountpoint=/srv/softwareheritage/postgres/11/indexer -o compression=lz4 -o atime=off -o primarycache=metadata -o recordsize=16k

On somerset:

root@somerset:/srv/softwareheritage/postgres/11# rsync -PaHAXx --info=progress2 --append indexer/ belvedere:/srv/softwareheritage/postgres/11/indexer/
ardumont mentioned this in T2547: pgstorage: Migrate db to storage 0.13.2 (db versions 160, 161).Sep 9 2020, 11:13 AM
olasd mentioned this in rSPSITEb8a2be661c55: Move indexer database to belvedere.Sep 9 2020, 11:53 AM
olasd changed the task status from Open to Work in Progress.Sep 9 2020, 11:54 AM
olasd claimed this task.

The indexer database data has been migrated to belvedere.

The replication setup to dbreplica1.euwest.azure has been updated:

  • On dbreplica1:
    • update /srv/softwareheritage/postgres/11/replica/recovery.conf to point at the new server
    • update /var/lib/postgresql/.pgpass to record credentials of the new server
    • update /var/lib/postgresql/.ssh/authorized_keys for keys present on belvedere
  • On belvedere:
    • ensure /var/lib/postgresql/.ssh/config uses a proper ssh key for the postgresql archive_command.
    • check that the archival works:
      • select pg_switch_wal(); to force WAL rotation
      • select * from pg_stat_archiver; to make sure the last wal archival was successful.

The indexer db configuration has also been updated in puppet.

gitlab-migration closed this task as Migrated.Oct 19 2022, 5:57 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T2562: Point web frontend to belvedere temporarily from Resolved to Migrated.Oct 19 2022, 5:57 PM
gitlab-migration closed subtask T2576: Migrate dbreplica0.euwest.azure to zfs as Migrated.
About · Developer information · Legal