the decision is to create a GitLab repository dedicated to writing the FAQ (Ambassadors can do PR or add issues dedicated to the FAQ)

• the repository will contain a faq-website.md file that can be easily exported into html with the correct tags per question
• concerning the devs/users/sys-admins FAQ, they will be directly edited in the docs
• if issues on the FAQ repository will concern devs/users/sys-admins
  • these issues will be tagged and resolved with addition on the docs instance

In T3127#66684, @anlambert wrote:

As @olasd said in a previous comment, even if we compute the metrics, we will miss counters about origins not tight to a lister
(googlecode and gitorious for instance). So I am thinking again about an hybrid approach using the swh-counters metrics
implemented yersteday which gives a rough estimation on the number of origins by network location (as visit statuses are not
processed, only origins) and the scheduler metrics.

In T3127#66673, @ardumont wrote:

I guess the cli to update metrics is executed periodically in production ?

I don't think that they are yet but that just got a priority increase now ;)

I guess the cli to update metrics is executed periodically in production ?

The existing scheduler metrics are probably not complete enough for all we want to display (we should review them so they are), but the swh.scheduler journal client already gathers all the information needed, so we > should be able to compute all that we need from the scheduler tables.

After more thoughts about all those metrics, we could revamp the coverage widget into two tabs:

• one tab displaying metrics about loaded origins with detailed counts by forge and links to search interface to browse them
• one tab displaying metrics about listed origins from the data extracted from the scheduler database

In T3127#66665, @ardumont wrote:

@anlambert @rdicosmo

For information, discussing with @olasd, he reminded me that we had already a cli entrypoint [1]
to compute stats about what we want scheduler side.

What's missing implementation wise would be to expose an endpoint to actually display said information.

So, the question is, even though the implementation swh.counter started, do we really want that there
or this ^ scheduler side would be enough?

[1] https://forge.softwareheritage.org/source/swh-scheduler/browse/master/swh/scheduler/cli/origin.py$148-182

Sorry @anlambert, I was late at Monday's meeting and I completely missed this in your weekly plan, I would have pointed this out earlier.

For information, discussing with @olasd, he reminded me that we had already a cli entrypoint [1]
to compute stats about what we want scheduler side.

In T3127#66631, @rdicosmo wrote:

Nice to see this moving forward!

These entries in the counter log look suspicious, though, they are not origins:

b'atlassian@bitbucket.org' 2
b'taylorhakes@github.com' 2
b'bunnyhero@bitbucket.org' 1
b'dtrebbien@bitbucket.org' 1
b'eldargab@github.com' 1
b'git@github.com' 1
b'schierlm@git.code.sf.net' 1
b'tomakehurst@github.com' 1
b'wenshao@github.com' 1
b'zimbra-mirror@bitbucket.org' 1

Nice to see this moving forward!

Regarding this, to ease the mapping between a lister and an instance name, we may want to rework the instance names in the scheduler
model (listers table) so that the value is actually the netloc of the origin.

Great work! Awesome.

After some analysis, the data we need to properly implement this are:

• the set of lister names and their instance names in order to organize origins by forge types (gitlab, cgit, sourceforge, ...)
• a precise or estimated count for the origins listed by a given lister instance

great ;)

The save code now queue statistics are now displayed on the status.io page[1] as an example. The data are refreshed each 5 minutes.

Metrics can easily be pushed to the status page.
The simple poc for the save code now request is available here : https://forge.softwareheritage.org/source/snippets/browse/master/sysadmin/status.io/update_metrics.py

for the status.swh.org point of view, status.io is providing some api endpoint to push metrics. It should be possible to add some metrics (up to 10 with our plan) to expose the behavior of the platform (daily/weekly and monthly statistics).
As a first step, we could expose the number of pending save code now requests and the number of origin visits to have some live data. An example of a status page with metrics : https://status.docker.com/
I'm working on a code snippet to test the integration feasibility/complexity.

In T3087#63887, @rdicosmo wrote:

In T3087#63791, @douardda wrote:

So what about exports of the archive available on git-annex?

Deployed.

Trying to bisect the issue and failed (multiple dimensions got the best of me, master ok, debian build in stable ko... i need to improve my tooling there).

So, the debian/unstable broke which is fixed now [1] (it wasd missing one new deps to
test the migration part).

Landed. Remains to deploy. I'll attend to that tomorrow.

In T3087#63791, @douardda wrote:

So what about exports of the archive available on git-annex?