In T36#629, @olasd wrote:

sadly, 14k is only .1% ;)

In T36#626, @olasd wrote:

IPython notebook to play with the result times scatter plot :

result_times.ipynb2 KBDownload

In T36#628, @zack wrote:

Based on that data, here are the current average/stddev processing times per repository based on the first ~14k random repositories loaded (~1% of our total):

(Thanks for making me play for the first time with a IPython notebook, it's a pretty impressive environment to play with scientific data.)

IPython notebook to play with the result times scatter plot :

see also T62

see also T22

This makes me think that we are now i/o bound on writes on our storage.

This task made good progress today. I spent a small while perusing our logging to understand the margins for performance.

• /revision/<SHA1_GIT>: show commit information
• /directory/<SHA1_GIT>: show directory information (including ls)
• /directory/<SHA1_GIT>/path/to/file-or-dir: ditto, but for dir pointed by path
• /content/[<HASH_ALGO>:]<HASH>: show content information
• /release/<SHA1_GIT>: show release information
• /person/<PERSON_ID>: show person information
• /origin/<ORIGIN_ID>: show origin information
• /project/<PROJECT_ID>: show project information
• /organization/<ORGANIZATION_ID>: show organization information
• /directory/<TIMESTAMP>/<ORIGIN>|/<BRANCH>|/path/to/file-or-dir : show directory information at timestamp/origin/branch
• /revision/<TIMESTAMP>/<ORIGIN>|/<BRANCH> : show revision information at origin/branch/timestamp
• /revision/<TIMESTAMP>/<ORIGIN>| : Show all branches of origin at a given timestamp
• /revision/<TIMESTAMP>/<ORIGIN>|/<BRANCH>| : Show all revisions (~git log) of origin and branch at a given timestamp

update, backup conf is now:

daily pg_dump over the net is now setup on prado for the databases gitimport and snapshot.debian.org, see prado:/usr/local/bin/swh-postgres-backup-sesi and /srv/softwareheritage/postgres/backup.conf

Done as of rDLDG69a5070

Resolved as of rDLDGc8f7d27.

mv started on uffizi, in a screen session

this is now done (thanks Laurent!)

technology suggestion for how to deal with this nicely: https://pypi.python.org/pypi/retrying

service tag sent to the Dell commercial (thanks Laurent!)

Resolved as of rDSTO2b46e6941afe

As discussed on swh-private, this is no longer required now. We will reassess after having injected all the content we already have, selectively transfering only what we want/need.

It is now done for the workers, but not for the other hosts (louvre, tait, etc.)

Status update: I've established a first contact with the Dell commercial.
To proceed, he is asking the serial number of our *current* power vault.

new quotation iteration, after discussion with olasd:

• Done once with basic API
• Refactor to use an unified API call
• Keep up with latest change on swh-storage

as a start, I've created a couple of quotations on https://dell.quadrem.net/

• panier-dell-r430.html37 KBDownload
  : with the cheapest (~1K) dell server/configuration (R430) I've found, but AFAICT compatible with the controller required for the disk array
• panier-dell-r630.html38 KBDownload
  : slightly more expensive (2K) server (R630)

either way, the overall price is completely dominated by the disk cost…

done in rDSTObe3910ecff368967cbef7f803dbdf191c1510c3d (and subsequent fixups by olasd)

python3-swh.loader.git is installed and running on worker0{5..8}

gzip/checksumming restarted, after fixing the /etc/fstab mess on the machine

priority lowered as, for better or worse, we have already freed enough space on the machine for DB backups without having to transfer the data