The ClearlyDefined experiment is done.
The resources can be freed:
@zack could you confirm if we can perform the outboarding of tg1999 and what we need to backup from the experiment?
| rSPSITE puppet-swh-site | |||
| D5557 | rSPSITE239a1337af3d Remove clearly-defined resources | ||
| D5539 | rSPSITE73f28d682007 Outboarding of tg1999 | ||
@vsellier: ack on the outboarding, that is actionable as of now.
Regarding the resources, I'm getting @vlorentz in the loop.
Period crawling/integration of extrinsic metadata is something we have in the roadmap for this year, and ClearlyDefined data is one potential candidate for a first real-world experiment.
So, @vlorentz: what do you think we should keep of it, for now?
It seems to me that loading the dump didn't take much time. So a minimal solution would be to keep just that and drop everything else.
Or we can keep just the live postgres DB, but it seems risky, and also not necessarily worth if you don't think we're going to work on this for, say, months.
According to https://intranet.softwareheritage.org/wiki/Outboarding:
*TG1999 removed from interns members in the forge, no tasks were assigned to the user
root@louvre:/etc/openvpn/keys# ./easyrsa revoke tg1999
Note: using Easy-RSA configuration from: ./vars
Using SSL: openssl OpenSSL 1.1.1d 10 Sep 2019
Please confirm you wish to revoke the certificate with the following subject:
subject=
commonName = tg1999
Type the word 'yes' to continue, or any other input to abort.
Continue with revocation: yes
Using configuration from /etc/openvpn/keys/pki/safessl-easyrsa.cnf
Revoking Certificate 79E27B7CDB4E84BB17861A0FDE52912B.
Data Base Updated
IMPORTANT!!!
Revocation was successful. You must run gen-crl and upload a CRL to your
infrastructure in order to prevent the revoked cert from being accepted.root@louvre:/etc/openvpn/keys# ./easyrsa gen-crl; chmod a+r pki/crl.pem Note: using Easy-RSA configuration from: ./vars Using SSL: openssl OpenSSL 1.1.1d 10 Sep 2019 Using configuration from /etc/openvpn/keys/pki/safessl-easyrsa.cnf An updated CRL has been created. CRL file: /etc/openvpn/keys/pki/crl.pem
The compression and the transfer of the dump from clearly-defined server to saam is in progress.
ssh clearly-defined.internal.staging.swh.network "pbzip2 -c -9 /srv/softwareheritage/clearlydefined/clearcode-dump-2021-01-05/clearcode_backup-20210105-2111.dump" | pv | cat > clearcode_backup-20210105-2111.dump.bz2
When it will be done, and if the compression ratio is good, the compressed archive will be added to the public annex in a dataset/clearly-defined directory
The clearly-defined dump is backuped on /srv/storage/space/mirrors/clearly-defined/clearcode-dump-2021-01-05/clearcode_backup-20210105-2111.dump
root@db1:~# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data 27.3T 623G 26.7T - - 16% 2% 1.00x ONLINE - root@db1:~# df -h Filesystem Size Used Avail Use% Mounted on udev 48G 0 48G 0% /dev tmpfs 9.5G 2.0M 9.5G 1% /run /dev/mapper/ssd-root 20G 3.7G 16G 20% / tmpfs 48G 212K 48G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock /dev/sdm1 487M 5.2M 481M 2% /boot/efi data/postgres-misc 26T 128K 26T 1% /srv/softwareheritage/postgres data/postgres-main-12 27T 621G 26T 3% /srv/softwareheritage/postgres/12/main data/postgres-indexer-12 26T 128K 26T 1% /srv/softwareheritage/postgres/12/indexer data/postgres-secondary-12 26T 128K 26T 1% /srv/softwareheritage/postgres/12/secondary tmpfs 9.5G 0 9.5G 0% /run/user/1003 tmpfs 9.5G 0 9.5G 0% /run/user/1025
postgres=# drop database "clearly-defined"; DROP DATABASE postgres=# drop database "clearly-defined-full"; DROP DATABASE
root@db1:~# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data 27.3T 180G 27.1T - - 16% 0% 1.00x ONLINE - root@db1:~# df -h Filesystem Size Used Avail Use% Mounted on udev 48G 0 48G 0% /dev tmpfs 9.5G 1.9M 9.5G 1% /run /dev/mapper/ssd-root 20G 3.7G 16G 20% / tmpfs 48G 212K 48G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock /dev/sdm1 487M 5.2M 481M 2% /boot/efi data/postgres-misc 27T 128K 27T 1% /srv/softwareheritage/postgres data/postgres-main-12 27T 179G 27T 1% /srv/softwareheritage/postgres/12/main data/postgres-indexer-12 27T 128K 27T 1% /srv/softwareheritage/postgres/12/indexer data/postgres-secondary-12 27T 128K 27T 1% /srv/softwareheritage/postgres/12/secondary tmpfs 9.5G 0 9.5G 0% /run/user/1003 tmpfs 9.5G 0 9.5G 0% /run/user/1025 tmpfs 9.5G 0 9.5G 0% /run/user/113
puppet ressources cleaned:
root@pergamon:~# /usr/local/sbin/swh-puppet-master-decomission clearly-defined.internal.staging.swh.network + puppet node deactivate clearly-defined.internal.staging.swh.network Submitted 'deactivate node' for clearly-defined.internal.staging.swh.network with UUID 26eb9a73-add9-4745-b068-6106ab2b20b4 + puppet node clean clearly-defined.internal.staging.swh.network Notice: Revoked certificate with serial 256 Notice: Removing file Puppet::SSL::Certificate clearly-defined.internal.staging.swh.network at '/var/lib/puppet/ssl/ca/signed/clearly-defined.internal.staging.swh.network.pem' clearly-defined.internal.staging.swh.network + puppet cert clean clearly-defined.internal.staging.swh.network Warning: `puppet cert` is deprecated and will be removed in a future release. (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run') Notice: Revoked certificate with serial 256 + systemctl restart apache2