Build is green

Build was aborted

mention schema change in commit

Build is green

Rebase

thanks!

Starting with 10 nodes will allow to have some remaining space.

Can you try with this patch? P1118

The gros cluster at Nancy[1] has a lot of nodes(124) with small reservable SSD of 960Go. This can be a good candidate to create the second cluster. It will also allow to check the performance with data (and commit logs) on SSDs.
According to the main cluster, a minimum of 8 nodes are necessary to handle the volume of data (7.3 To and growing). Starting with 10 nodes will allow to have some remaining space.

it seems some more precise information can be logged by activating the full query logs without a big performance impact: https://cassandra.apache.org/doc/latest/cassandra/new/fqllogging.html

Next step: write tests for that updated view.

Build is green

Remove TEST_DIRS modification.

In D6117#158046, @vlorentz wrote:

I don't think that issue is specific to swh-search, I have it with other packages from time to time. rm build/ -rf should fix it.

I don't think that issue is specific to swh-search, I have it with other packages from time to time. rm build/ -rf should fix it.

you mean T3492

Should be fixed by T3482

Build is green

extra info description

Build is green

Update:

• Rename section titles according to @zack suggestions
• Remove vertical padding around counters to gain vertical space
• Add fallback when scheduler metrics or deposit lists are not available, widget with logos will stil be displayed but without counters info

As a minor suggestion I propose the following heading changes:

 listed origins -> regular crawling
 legacy origins -> discontinued hosting
deposited origins -> on demand archival

Build is green

In D6112#157975, @vlorentz wrote:

In D6112#157949, @anlambert wrote:

Looks good but the SQL schema migration file is missing in sql/upgrades so I cannot accept the diff yet.

Actually, we're going to drop the database + objstorage and recreate it, writing a migration for this looks like too much trouble for a cache.

Update: Provide RawExtrinsicMetadata targetting core SWHIDs as test inputs and reverse related changes.

We don't want to allow extended SWHID in the public API, you should restrict data generated by hypothesis instead

In D6115#157987, @vlorentz wrote:

Does it mean we need to do def test_inner on *every* test that uses hypothesis?

Build was aborted

I really can't find a better way to do this :(

Build is green

Does it mean we need to do def test_inner on *every* test that uses hypothesis?

We don't want to allow extended SWHID in the public API, you should restrict data generated by hypothesis instead

Build is green

update JS code

Build is green

In D6112#157949, @anlambert wrote:

Looks good but the SQL schema migration file is missing in sql/upgrades so I cannot accept the diff yet.

fix task arg serialization/deserialization

Rebase

Fix a comment

Oh I missed D6081, I guess we both encounter the same kind of issues but fixes are not exactly the same.

Looks good but the SQL schema migration file is missing in sql/upgrades so I cannot accept the diff yet.

@vlorentz, I have submitted D6115 and D6116 to ensure swh-web tests can be safely executed with multiple hypothesis examples.

@vsellier I am not sure about this.
The idea is to use this machine as the production server. (I guess this will host either postgres or mongodb after we decide on a preferred backend. But that is going to take some time)
@olasd or @aeviso will know better.

In ~40h, the backfill is done at ~5% for staging and less than 1% for the production