In T1414#55819, @vsellier wrote:

Can this task be closed since the subject was addressed in T2620 ?

Can this task be closed since the subject was addressed in T2620 ?

We've definitely improved on this (notably using proper hostnames for the instance label on prom metrics). I think we should make this task more actionable if we want to keep it open.

Wikimedia is using netbox as the source of trust in their infrastructure and puppet is configuring the facts from it. It's not exactly the same use case we want as we would like to have netbox automatically provisioned.

and their documentation : https://wikitech.wikimedia.org/wiki/Netbox

A docker-compose is available to easily test netbox : https://github.com/netbox-community/netbox-docker
This is the puppet configuration used at wikimedia : https://gerrit.wikimedia.org/r/c/operations/puppet/+/387880/

Netbox looks pretty nice as a full hardware/device inventory tool: https://netbox.readthedocs.io/en/stable/

Puppet changes added in 17b2b3041212aca9e0a9a35c510885de7bb78230.
Ideally the Debian package should now be added to the Software Heritage private repository.

Instructions to create Debian packages have been added in D2352.

AFAIU from last week work, munin is now gone

No relevant problem has been reported with our dataset/usage of Prometheus. Closing.

I do not see any missing piece in the Grafana dashboard, the Munin graph service/VM can be shut down.

Any chance we can close this now?

The most recent update of the state of this task has shown a regression in the journal test coverage, which, per se, is not a big deal (just a few points). But it does raise the question of how, once we have attained whatever "minimum" coverage we are OK with, we monitor overtime that there is no regression. For instance, I think that code reviews should show to the reviewers how the submitted diff affects code coverage. Ideally, reviewers should be able to so if it has a net positive or negative effect on coverage, and take that into account in their review decisions. (Which is not to say we should never accept diffs that decrease code coverage—there might be reasons to do so. But it is a data point that would be useful for reviewers to see.)

only 3% to go in -lister and -core \o/

these catch-all meta-tasks that will grow forever are not terribly useful, the individual tasks + their tasks should be enough

The grafana dashboards are stored in the postgresql database on pergamon, which is backed up through the full system backups.

There was a config/deployment bug on both the hg and svn loaders. Both bugs have been fixed and the tasks are running fine now.

Reopening this as the first submitted save code now tasks for hg and svn did not get executed so far (see [1]).
Nevertheless, they have been scheduled so this looks like some extra workers configuration is needed in production.

Thanks... looks like the tasks have been properly scheduled, but they have
not been executed... some more polishing may be needed ...

@rdicosmo , the possibility to submit hg and svn origin types through the "Save code now" form has been deployed to production [1].
I have submitted one origin of each type to save. Let's see if the underlying scheduler tasks get correctly executed before spreading
the news to the wild.

we (@anlambert and I) will try to have this task closed ASAP (like today, if no big bad stopper arise in front of us)

Grafanalib dashboards added to https://grafana.softwareheritage.org/ via the new provisioning mechanism of Grafana 5.x.
Fully automated provisioning is still a work-in-progress.

Prometheus does not provide storage device statistics for Proxmox container-based hosts.
The data can be read from their parent machine dashboards though.

Are there any blockers left? It would be really nice to roll this out in the very near future.

If we remove Munin before implementing missing graph replacements, we will lack a comparison base and possibly fail to discover bogus data.
Right now, Prometheus disk throughput and iops values are suspiciously low for example.

It is these graphs we are still missing.

Even though most/all of the Munin metrics are provided by Prometheus, Munin also provides graphs.
It is these graphs we are still missing.

Indeed it is the object of T1428. That's why I am a bit puzzled the work you have in progress does not simply target T1356. I was expecting some response to this very task in your grafanalib based code, which I did not find. So I was wondering if I missed something, that some data where still in munin only.

Wasn't that what T1428 was about ?
Apart from the list of pending packages, all commonly used Munin metrics should already have Prometheus equivalents.

In T1442#30575, @ftigeot wrote:

When I asked where to put such work-in-progress, you suggested the snippets repository.

When I asked where to put such work-in-progress, you suggested the snippets repository.

In T1442#30564, @ftigeot wrote:

Work-in-progress Grafanalib dashboards have been added to the https://forge.softwareheritage.org/source/snippets/ repository.

Work-in-progress Grafanalib dashboards have been added to the https://forge.softwareheritage.org/source/snippets/ repository.

(typo) nothing interesting, moving along.

Related D1411
Related rCDFD77f4b2e0617be57282e5ab4a972f7a643768e668
Related rCDFDf68bb33b1493d7ef46471e3162abd03e6d6b0021
Related rCDFD2f87f477046fc82b7ce4b6fede712c051e943c14

Let's call it done, event if the small dataset part has not been addressed.

Let's call it done, some minor parts may still need a bit of attention thou

Consider this done. Even if remains a background task.

Already marked as done on 2018-12-19.

This gargantuan query is now used on a grafana dashboard : https://grafana.softwareheritage.org/d/-lJ73Ujiz/scheduler-task-status

Full prometheus histogram-compatible query:

Current status of the SQL query:

@douardda I wish to work on this patch. Can you explain to me please what do I have to do in this?

\o/

All storage mocks are now removed in every swh modules, so closing this.