Makes sense to me to add them in sysadm.
Also, jsyk, currently, the support-services stub files are already present in the sysadm instance, remains to to fill in the blank at some point.

The new kafka is deployed in storage.
The partition reassignment is in progress

We have followed the plan on the hedgedoc (in the last comment).
Just realized we haven't decided if the support-tools should be in sys-admin dimension, so I'm adding this to this task to decide.

Build is green

In T3648#72310, @ardumont wrote:

/me *sigh* it's still red...

That really should be a team effort, to strive to keep our builds green.
So for this build, i concur with @anlambert, we should make it build during diff.

15:14 <+anlambert> ardumont: in order to catch sphinx issues in swh-docs, I was thinking of running the swh-docs/dev Jenkins job when submitting a diff to swh-docs. I think we only need to setup a new harbomaster build plan for that (https://forge.softwareheritage.org/harbormaster/plan/).
15:14 <+ardumont> anlambert: yes, that'd be good because i'm still on T3648 and that's really frustrating

@anlambert thx for the suggestion ^

Remove not needed sorting operation on swhids list.

Deployed: https://docs.softwareheritage.org/user/
and with a redirect from https://docs.softwareheritage.org/users/ to https://docs.softwareheritage.org/user/

Rebase

/me *sigh* it's still red...

Lister got adapted to actually activate the sourceforge origins.
That lister got deployed.

Rebase

Build is green

yes, those static dicts should be moved out of swh-web but I do not have a proper solution in mind so far.

I've run alter system commands to bump these configuration variables in $DATADIR/postgresql.auto.conf, then ran a pg_reload_config():

Rebase

The log is flooded with

2021-10-14 15:24:54.422 UTC [3951720] LOG:  checkpoints are occurring too frequently (28 seconds apart)
2021-10-14 15:24:54.422 UTC [3951720] HINT:  Consider increasing the configuration parameter "max_wal_size".

Build is green

Build is green

Looks good to me, and yes those static dicts should be moved out of swh-web but I do not have a proper solution in mind so far.

Rebase

Rebase

• use journal1.i.s.s.n as zookeeper name (only this because the other kafka configuration is mainly using the fqdn of the server)
• declare a new le certificate for storage1.i.s.s.n
• exclude the kafka log directory from the backup

Ah, now that I read through this again; would it make sense for the zookeeper server to be called using the CNAME instead of the host FQDN ?

Yes sure it should be better to hide the storage role of the server, let's try to use the CNAME first.
I ask myself which one to use and forgot about it when it started to work locally

zfs dataset created on storage1:

root@storage1:~# zfs create -o mountpoint=/srv/kafka -o atime=off -o relatime=on data/kafka
root@storage1:~# zfs list
NAME           USED  AVAIL     REFER  MOUNTPOINT
data          5.39T  21.0T       96K  /data
data/kafka      96K  21.0T       96K  /srv/kafka
data/objects  5.38T  21.0T     5.38T  /srv/softwareheritage/objects

When D6477 will be validated and applied, the move action will be:

Ah, now that I read through this again; would it make sense for the zookeeper server to be called using the CNAME instead of the host FQDN ?

Looks good, except for a missing new TLS certificate, I think.

That's been done for years now.

Actual count on bitbucket origins:

14:23:32 softwareheritage@belvedere:5432=> select now(), count(distinct url) from origin o inner join origin_visit ov on o.id=ov.origin where o.url like 'https://bitbucket.org/%' and ov.type='hg';
+-------------------------------+--------+
|              now              | count  |
+-------------------------------+--------+
| 2021-10-14 12:23:33.901117+00 | 336795 |
+-------------------------------+--------+
(1 row)

In T3658#72284, @olasd wrote:

We could argue that adding a separate, "virtual" lister instance for these bulk archived origins would make sense, but I don't know if it's worth the bother.

I was thinking of something ad-hoc such as:

Proper rebase

Rebase

Build is green