SQL schema version 32 (from D6812) with the updated update_metrics function has been deployed in staging and prod.

I have started test conversions of the OpenBSD CVS repository.

So the update scheduler metrics routine the graph (mentioned in the description)
relies upon is taking some time to finish (investigation ongoing [3]) hence the apparent
stale.

Scheduling has been done [1]. The listing process happened [2]. Now let the debian
loaders do its job (ongoing) [3].

Adaptations in the loader core for the ubuntu history release has been deployed within the v2.0 scope.
I'm looking into triggering the necessary task now.

On second thoughts: in order to run the docker-dev setup, I also had to run a virtual machine alongside the swh setup to host the text index file, and make sure the swh vm could access it.
I suppose that any vm/docker/baremetal machine with an apache/nginx server could do for that, as long as the lister can http-fetch the .fld file.

In T1724#75125, @ardumont wrote:

I'm asking you for a diff with the exact changes you had to make in the
swh-environment/docker/docker-compose.yml (and other folders) to actually make it run.
That will definitely help for the deployment on staging.

I'm not sure what you mean by the docker diff.

I'm not sure what you mean by the docker diff. Is that the update of the maven-index-exporter repository at D6740?
The above-mentioned repository has documentation to build, test and run the text index generation. As mentioned there I've also created a bunch of compressed text index exports, that can be used to test the lister/loader without running the docker image immediately. They are all real-world extracts obtained by running the docker image on the list of Maven repositories I could get as of last week. They together represent a few million artefacts.

I also tested the listing of Ubuntu historical releases and it went fine.

After discussing with @olasd, the plan is mostly ok for the "current" release.

Heads up, now that the lister/loader are unstuck on production (and debian origins are
actually scheduled). This is a technical go for this.

I also tested the listing of Linux Mint source packages. Apart a small fix to add to the lister (D6741), the listing went fine:

(swh) ✘-1 ~/swh/swh-environment/docker [master|✚ 1⚑ 41] 
14:22 $ docker-compose exec swh-scheduler swh scheduler task add list-debian-distribution -p oneshot distribution=LinuxMint mirror_url=https://mirror.crexio.com/linuxmint/packages/ suites="[betsy, cindy, debbie, debian, elyssa, felicia, gloria, helena, helena-fluxbox, helena-kde, helena-lxde, helena-xfce, isadora, isadora-fluxbox, isadora-kde, isadora-lxde, isadora-xfce, julia, julia-fluxbox, julia-kde, julia-lxde, julia-xfce, katya, katya-fluxbox, katya-kde, katya-lxde, lisa, lisa-kde, lisa-lxde, maya, nadia, olivia, petra, qiana, rafaela, rebecca, romeo, rosa, sarah, serena, sonya, sylvia, tara, tessa, tina, tricia, ulyana, ulyssa, uma, una]" components="[backport, debian, import, incoming, main, romeo, upstream]"

After trying to list and load ubuntu packages in the docker environment, I found a couple of issues in the debian lister and loader that were preventing succcessful listing and loading.
Those issues have been fixed in D6735, D6737 and D6738.

D6684 addresses another keyword expansion issue found while testing conversion of CVS's own history.

The above problem with the Header keyword can be worked around (at least for the GNU savannah site) with the patch in D6678.

Hi there,

Another problem with keyword expansion found during testing:

@ardumont For production, drop the / at the end of the url, in staging, it's duplicated...

This got added to the staging infrastructure which lead to some surprises.
This mostly got fixed in the commits ^.

In T3691#73518, @stsp wrote:

There is another problem related to keywords: Some CVS-based projects use custom keywords, instead of the standard $Id$ keyword. This prevents wrong expansion of $Id$ when code is imported from one project to another. Usually the project's name will be used as the custom keyword name, such as $OpenBSD$ or $NetBSD$, instead of $Id$. At present, to expand keywords correctly in this case, we need to use the pserver access method to benefit from server-side keyword expansion. But we will end up with different hashes if rsync is used to import the same origin again. We might be able to auto-detect use of custom keywords if the rsync server allows access to the CVSROOT folder, but this is not always the case. If CVSROOT is hidden from rsync, the only reliable way to detect custom keywords would be a parameter that gets passed into the loader. We could, for example, allow passing the name of a custom keyword as a parameter embedded in the origin URL.

As of D6623 the CVS loader is able to convert GNU dino correctly over both rsync and pserver access.

Status update:

Deployed in production.

Deployed in staging.