Script: https://forge.softwareheritage.org/source/snippets/browse/master/vlorentz/migrate_extra_headers.py

Closing this as resolved now the search feature is using elasticsearch in production.

I just discussed the multiplexer-based migration process I described above with ardumont/olasd/vsellier.

Doesn't this deserve a state-of-the-art kind of thing?

Doesn't this deserve a state-of-the-art kind of thing? Are there documentation material on the subject? How does other (big) cassandra users handle this?

In T2602#63432, @vlorentz wrote:

For the harder cases, that involve changes to the PK, we could do something like this:

• create a new table with a new name (eg. revision_v[n+1]; like we do in swh-search except Cassandra does not support aliases)
• start an extra storage backend, that reads from that table instead of the old one (eg. revision_v[n]), and also reads from all the other tables as usual
• have a multiplexing storage proxy (like we have for the objstorage), that queries this new backend (which reads from v[n+1]), and falls back to the old backend (which reads from v[n])

What we can do, however:

Schema image is now properly displayed: https://docs.softwareheritage.org/devel/swh-storage/sql-storage.html#sql-storage

Thanks @faux @KShivendu @anlambert, team work ;)

if you remember the crash times (.zsh_history?), we could find a range of candidate SWHIDs...

The migration script has now run to completion (took around a week).

@KShivendu The linked script is a start. As it is, it requires direct access to the DB; so you need to create abstractions for it in swh-storage and swh-web

ok, thanks. It's actually tested in test_stat_counters in swh-storage/swh/storage/tests/storage_tests.py, which is used to test all four classes.

Hi guys. Any pointers on where to start?

I might be wrong but, I think it has been completed. Check out these :

No longer relevant

I've deployed the extid schema changes on all storages, and I've started the migration script on getty.

(and we should keep the origin topic; we already have an ExtSWHID for origins anyway)

The following objects remain:

After a lot of back and forth, and the release of swh.model v2.3.0 and swh.storage v0.26.0, this is now all done and deployed in staging and production.

After the release of swh.model v2, this is now done.

Let's organise a call next week to explore the options, including the new opportunities of testing that emerged recently.

@rdicosmo I have not, good idea. While they are probably too expansive to use as the main storage instead of SSDs (either via a regular FS or by using a Pmem-aware Cassandra fork), we could use them in addition to the above requirements.

Did you consider PMem (and other configurations for Intel Optane memory) in your discussion? It offers a very interesting price/performance ratio.
There are machines on Grid5000 available to test this technology if needed.