I stumbled across GitLab GraphQL API while working on T3442, could be a great source of inspiration.

In T3127#67581, @anlambert wrote:

    I am a bit puzzled by the numbers shown: eeally we have only 200k origins for GitLab.com.?

Indeed there is something weird here as we have more than one million gitlab.com origins in database.

softwareheritage=> select count(*) from origin where url like 'https://gitlab.com/%';
  count  
---------
 1023499
(1 row)

Looks like something was missed when computing lister metrics from scheduler database, this needs further investigations.

Indeed, please do look into this, thanks.

Thanks for these details: this count is missing the 800k git origins: @ardumont and @olasd should be able to tell you how to find them

In T3127#67581, @anlambert wrote:

I am a bit puzzled by the numbers shown: eeally we have only 200k origins for GitLab.com.?

Indeed there is something weird here as we have more than one million gitlab.com origins in database.

softwareheritage=> select count(*) from origin where url like 'https://gitlab.com/%';
  count  
---------
 1023499
(1 row)

Looks like something was missed when computing lister metrics from scheduler database, this needs further investigations.

I am a bit puzzled by the numbers shown: eeally we have only 200k origins for GitLab.com.?

I am a bit puzzled by the numbers shown: eeally we have only 200k origins for GitLab.com.?
And we know we had some 1.5m origins for Google code, why only 700k shown here?

Instead, we could split the coverage widget into two tabs

• one giving a high level overview of the archived origins, similar to what we have now with logos and counters
• one giving the details of all forges we archived so far, displayed in a table as you suggested with relevant metrics and links to search origins for a given forge

13. after the lines slides, you can imagine how it is a qualifier on a SWHID but you can't see it. Might be useful to open again the permalink box to see.

Thanks @anlambert for this very intuitive guided tour!

That's awesome! I did not spot any typos.

First version of guided tour has been deployed to production for ambassador and staff users.

I think we could also get an accurate count of deposit origins (HAL, IPOL) using swh-deposit API

Only one nit about the display. Using modal windows/popover will mean that there will be no easy way to have, as a user, the full list: one will have to click on each logo one by one, which could be quite annoying. Would it be possible to have a page with a rendering of the table above? (not sure if we want all columns, but at least the last update time and the number of origins per forge instance looks relevant and interesting to me). It coule be either in addition of what you propose (e.g., as a "coverage details" link, leading to the full page), or as a replacement of it (e.g., by making each forge icon just a link to the relevant anchor within the table on the "coverage details" page).

Thanks for this update, great work!

Some reports of what have been done so far and some future directions regarding the display of those data in swh-web.

highlight.js 11.1.0 has been released fixing numerous regressions since the 11.0 release, so it is time to perform the upgrade.

After some hacks on the current Jenkins pipeline for swh-web to test if it was feasible, I managed to setup the use of sorry-cypress but could not run multiple cypress in parallel as Jenkins pipelines forbid nested parallel stages.
So this would require setting up a dedicated pipeline build job for cypress tests as currently Python and cypress tests are executed in parallel in a single job.

An example of a metadata icon for the deposit-admin view:
https://thenounproject.com/term/metadata/77940/

The example link [3] does return information

The issue mentioned ^ has been fixed and deployed.
Everything is deployed.

In T3127#66684, @anlambert wrote:

As @olasd said in a previous comment, even if we compute the metrics, we will miss counters about origins not tight to a lister
(googlecode and gitorious for instance). So I am thinking again about an hybrid approach using the swh-counters metrics
implemented yersteday which gives a rough estimation on the number of origins by network location (as visit statuses are not
processed, only origins) and the scheduler metrics.

In T3127#66673, @ardumont wrote:

I guess the cli to update metrics is executed periodically in production ?

I don't think that they are yet but that just got a priority increase now ;)

I guess the cli to update metrics is executed periodically in production ?

The existing scheduler metrics are probably not complete enough for all we want to display (we should review them so they are), but the swh.scheduler journal client already gathers all the information needed, so we > should be able to compute all that we need from the scheduler tables.

After more thoughts about all those metrics, we could revamp the coverage widget into two tabs:

• one tab displaying metrics about loaded origins with detailed counts by forge and links to search interface to browse them
• one tab displaying metrics about listed origins from the data extracted from the scheduler database

In T3127#66665, @ardumont wrote:

@anlambert @rdicosmo

For information, discussing with @olasd, he reminded me that we had already a cli entrypoint [1]
to compute stats about what we want scheduler side.

What's missing implementation wise would be to expose an endpoint to actually display said information.

So, the question is, even though the implementation swh.counter started, do we really want that there
or this ^ scheduler side would be enough?

[1] https://forge.softwareheritage.org/source/swh-scheduler/browse/master/swh/scheduler/cli/origin.py$148-182

Sorry @anlambert, I was late at Monday's meeting and I completely missed this in your weekly plan, I would have pointed this out earlier.

For information, discussing with @olasd, he reminded me that we had already a cli entrypoint [1]
to compute stats about what we want scheduler side.

In T3127#66631, @rdicosmo wrote:

Nice to see this moving forward!

These entries in the counter log look suspicious, though, they are not origins:

b'atlassian@bitbucket.org' 2
b'taylorhakes@github.com' 2
b'bunnyhero@bitbucket.org' 1
b'dtrebbien@bitbucket.org' 1
b'eldargab@github.com' 1
b'git@github.com' 1
b'schierlm@git.code.sf.net' 1
b'tomakehurst@github.com' 1
b'wenshao@github.com' 1
b'zimbra-mirror@bitbucket.org' 1

Nice to see this moving forward!

Regarding this, to ease the mapping between a lister and an instance name, we may want to rework the instance names in the scheduler
model (listers table) so that the value is actually the netloc of the origin.

Great work! Awesome.

After some analysis, the data we need to properly implement this are:

• the set of lister names and their instance names in order to organize origins by forge types (gitlab, cgit, sourceforge, ...)
• a precise or estimated count for the origins listed by a given lister instance

Remain the runner to actually consume those regularly to be deployed (or maybe it's already the case, i need to check that part).