Update: Activate origins search links in collapsible elements.

Closing this as requested.

Fix typo.

I stumbled across GitLab GraphQL API while working on T3442, could be a great source of inspiration.

For the record, my lister is still running, 1320500 gitlab.com origins listed so far.

In T3127#67581, @anlambert wrote:

    I am a bit puzzled by the numbers shown: eeally we have only 200k origins for GitLab.com.?

Indeed there is something weird here as we have more than one million gitlab.com origins in database.

softwareheritage=> select count(*) from origin where url like 'https://gitlab.com/%';
  count  
---------
 1023499
(1 row)

Looks like something was missed when computing lister metrics from scheduler database, this needs further investigations.

Indeed, please do look into this, thanks.

• Improve origin category texts
• Fix code retrieving lister metrics as some of them were dicarded
• Add display of distribution of origins per origins type, hidden by default, can be displayed by clicking on any collapsible elements (total origin counts)

Thanks for these details: this count is missing the 800k git origins: @ardumont and @olasd should be able to tell you how to find them

Address @ardumont comment.

I am a bit puzzled by the numbers shown: eeally we have only 200k origins for GitLab.com.?

Instead, we could split the coverage widget into two tabs

• one giving a high level overview of the archived origins, similar to what we have now with logos and counters
• one giving the details of all forges we archived so far, displayed in a table as you suggested with relevant metrics and links to search origins for a given forge

Rebase

The diff seems to fix the previously observed flaky test but got bitten by another one which does not happen so often thanks to cypress retry features.

• Add paragraphs detailing each origins category
• Add missing Phabricator origins in listed ones
• Address @ardumont comment about doi.org netlocs for deposits
• Use only png logos to simplify their integration
• Resize some large logos
• Remove no longer used logos

First version of guided tour has been deployed to production for ambassador and staff users.

I think we could also get an accurate count of deposit origins (HAL, IPOL) using swh-deposit API

Fix a guided tour step, element id was removed after a rebase.

Rebase

Remove added empty line

Update:

• install as root and in /opt
• update commit message

Only one nit about the display. Using modal windows/popover will mean that there will be no easy way to have, as a user, the full list: one will have to click on each logo one by one, which could be quite annoying. Would it be possible to have a page with a rendering of the table above? (not sure if we want all columns, but at least the last update time and the number of origins per forge instance looks relevant and interesting to me). It coule be either in addition of what you propose (e.g., as a "coverage details" link, leading to the full page), or as a replacement of it (e.g., by making each forge icon just a link to the relevant anchor within the table on the "coverage details" page).

Yes, that's a good point.
On the other hand, not including it means we install-depend on nodejs+yarn and a C compiler...

I added a couple of inline comments for some issues to fix and some improvements to add.

@anlambert We are going to need to precompile some assets in swh-search using JS dependencies. We define the grammar in JS, which can then be compile into a big JSON file, that will be read from Python; see swh/search/parser/package.json. Do you have some advice to make this work nicely?

Some reports of what have been done so far and some future directions regarding the display of those data in swh-web.

It seems the remaining lister instances to process are the phabricator ones that also need credentials.
This is what we currently have in the listers table in scheduler database.

In D5819#154171, @swh-public-ci wrote:

Build has FAILED

Patch application report for D5819 (id=21588)

Rebasing onto 3bfc6942fe...

Current branch diff-target is up to date.

Changes applied before test

commit c8e32e745aa3e76fc8d972cc89f6d01757e34e73
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Wed May 12 11:42:40 2021 +0200

    assets/webapp: Add UI guided tour to help users discover features
    
    Add a Web UI guided tour implemented using the intro.js library.
    
    The tour showcases the main browsing features offered by Software
    Heritage web application by highlighting the relevant DOM elements
    and displaying help texts in tooltips.
    
    The tour can span multiple web pages and steps are configured in
    a YAML file for help texts writing commodity.
    
    Currently, the homepage, the main origin browsing view and content
    browsing view are covered by the tour.
    
    Related to T3202

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/962/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/962/console

Rebase

Precise metrics about listed origins and their counts will be retrieved from the scheduler database, no need to backfill origins with swh-counters then, closing this.

Better indeed, thanks !

Looks good to me !