Remove TEST_DIRS modification.

In D6117#158046, @vlorentz wrote:

I don't think that issue is specific to swh-search, I have it with other packages from time to time. rm build/ -rf should fix it.

Update:

• Rename section titles according to @zack suggestions
• Remove vertical padding around counters to gain vertical space
• Add fallback when scheduler metrics or deposit lists are not available, widget with logos will stil be displayed but without counters info

As a minor suggestion I propose the following heading changes:

 listed origins -> regular crawling
 legacy origins -> discontinued hosting
deposited origins -> on demand archival

In D6112#157975, @vlorentz wrote:

In D6112#157949, @anlambert wrote:

Looks good but the SQL schema migration file is missing in sql/upgrades so I cannot accept the diff yet.

Actually, we're going to drop the database + objstorage and recreate it, writing a migration for this looks like too much trouble for a cache.

Update: Provide RawExtrinsicMetadata targetting core SWHIDs as test inputs and reverse related changes.

We don't want to allow extended SWHID in the public API, you should restrict data generated by hypothesis instead

In D6115#157987, @vlorentz wrote:

Does it mean we need to do def test_inner on *every* test that uses hypothesis?

Rebase

Fix a comment

Oh I missed D6081, I guess we both encounter the same kind of issues but fixes are not exactly the same.

Looks good but the SQL schema migration file is missing in sql/upgrades so I cannot accept the diff yet.

@vlorentz, I have submitted D6115 and D6116 to ensure swh-web tests can be safely executed with multiple hypothesis examples.

In D6083#157897, @vlorentz wrote:

Sounds good, thanks! Should I close this diff?

I need help to fix these failures...

Update:

• rebase
• implement search links for all origin types
• polish widget rendering: add scrollbar for origin type with lot of instances

Looks good to me, thanks !

Nice !

LGTM

+1, and code is more readable that way imho

LGTM, we should also rename that parameter in swh-web code related to the vault.

Could you add a test for this ?

LGTM

Great, thanks !

Update: Activate origins search links in collapsible elements.

Closing this as requested.

Fix typo.

I stumbled across GitLab GraphQL API while working on T3442, could be a great source of inspiration.

For the record, my lister is still running, 1320500 gitlab.com origins listed so far.

In T3127#67581, @anlambert wrote:

    I am a bit puzzled by the numbers shown: eeally we have only 200k origins for GitLab.com.?

Indeed there is something weird here as we have more than one million gitlab.com origins in database.

softwareheritage=> select count(*) from origin where url like 'https://gitlab.com/%';
  count  
---------
 1023499
(1 row)

Looks like something was missed when computing lister metrics from scheduler database, this needs further investigations.

Indeed, please do look into this, thanks.

• Improve origin category texts
• Fix code retrieving lister metrics as some of them were dicarded
• Add display of distribution of origins per origins type, hidden by default, can be displayed by clicking on any collapsible elements (total origin counts)

Thanks for these details: this count is missing the 800k git origins: @ardumont and @olasd should be able to tell you how to find them

Address @ardumont comment.

I am a bit puzzled by the numbers shown: eeally we have only 200k origins for GitLab.com.?

Instead, we could split the coverage widget into two tabs

• one giving a high level overview of the archived origins, similar to what we have now with logos and counters
• one giving the details of all forges we archived so far, displayed in a table as you suggested with relevant metrics and links to search origins for a given forge

Rebase

The diff seems to fix the previously observed flaky test but got bitten by another one which does not happen so often thanks to cypress retry features.

• Add paragraphs detailing each origins category
• Add missing Phabricator origins in listed ones
• Address @ardumont comment about doi.org netlocs for deposits
• Use only png logos to simplify their integration
• Resize some large logos
• Remove no longer used logos