Wed, Jul 20
ongoing work through T1362
Jun 23 2022
Closing this, as these metrics are now visible on https://grafana.softwareheritage.org/d/FqGC4zu7z/vlorentz-loader-metrics
Feb 7 2022
the exporter is deployed.
The varnish stats are available on this dashboard: https://grafana.softwareheritage.org/d/pE2xMZank/varnish
Feb 3 2022
Jan 27 2022
we can always improve it, but now we have a decent dashboard, so let's consider this done.
Jan 20 2022
Is there a reason not to close this task?
Is there a reason not to close this task?
Dec 3 2021
Aug 26 2021
This would have caught T3502 earlier too.
Aug 3 2021
The computation of those metrics will be executed in production on a regular basis, probably each day, to keep them up to date.
Jul 29 2021
Jul 23 2021
In T3127#67581, @anlambert wrote: I am a bit puzzled by the numbers shown: eeally we have only 200k origins for GitLab.com.? Indeed there is something weird here as we have more than one million gitlab.com origins in database. softwareheritage=> select count(*) from origin where url like 'https://gitlab.com/%'; count --------- 1023499 (1 row) Looks like something was missed when computing lister metrics from scheduler database, this needs further investigations.Indeed, please do look into this, thanks.
Jul 22 2021
Jul 21 2021
I am a bit puzzled by the numbers shown: eeally we have only 200k origins for GitLab.com.?
I am a bit puzzled by the numbers shown: eeally we have only 200k origins for GitLab.com.?
And we know we had some 1.5m origins for Google code, why only 700k shown here?
Instead, we could split the coverage widget into two tabs
- one giving a high level overview of the archived origins, similar to what we have now with logos and counters
- one giving the details of all forges we archived so far, displayed in a table as you suggested with relevant metrics and links to search origins for a given forge
Jul 19 2021
I think we could also get an accurate count of deposit origins (HAL, IPOL) using swh-deposit API
Jul 16 2021
Only one nit about the display. Using modal windows/popover will mean that there will be no easy way to have, as a user, the full list: one will have to click on each logo one by one, which could be quite annoying. Would it be possible to have a page with a rendering of the table above? (not sure if we want all columns, but at least the last update time and the number of origins per forge instance looks relevant and interesting to me). It coule be either in addition of what you propose (e.g., as a "coverage details" link, leading to the full page), or as a replacement of it (e.g., by making each forge icon just a link to the relevant anchor within the table on the "coverage details" page).
Thanks for this update, great work!
Jul 13 2021
Some reports of what have been done so far and some future directions regarding the display of those data in swh-web.
Jul 9 2021
Precise metrics about listed origins and their counts will be retrieved from the scheduler database, no need to backfill origins with swh-counters then, closing this.
Jun 23 2021
I guess the cli to update metrics is executed periodically in production ?
The existing scheduler metrics are probably not complete enough for all we want to display (we should review them so they are), but the swh.scheduler journal client already gathers all the information needed, so we > should be able to compute all that we need from the scheduler tables.
After more thoughts about all those metrics, we could revamp the coverage widget into two tabs:
- one tab displaying metrics about loaded origins with detailed counts by forge and links to search interface to browse them
- one tab displaying metrics about listed origins from the data extracted from the scheduler database
Sorry @anlambert, I was late at Monday's meeting and I completely missed this in your weekly plan, I would have pointed this out earlier.
For information, discussing with @olasd, he reminded me that we had already a cli entrypoint [1]
to compute stats about what we want scheduler side.
Jun 22 2021
Nice to see this moving forward!