Migrate jenkins nodes to bullseye
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	ardumont
	Dec 6 2021, 2:30 PM

Description

This means following the plan [3] for:

jenkins-debian1: the debian package build node [1]
thyssen: jenkins server node [2]

It should be fine to just upgrade the nodes as their respective major tooling are part of debian packages.

[1] The main tools here are sbuild and chroot.

[2] The main tooling are within docker constraint (so it's "pinned")

[3]

task=T3770
puppet agent --disable "$task: dist-upgrade to bullseye"
cd /etc
sed -i -e 's/buster/bullseye/' /etc/apt/sources.list.d/*
sed -i -e 's,bullseye/updates,bullseye-security,' /etc/apt/sources.list.d/debian-security.list
git status
grep bullseye-security /etc/apt/sources.list.d/debian-security.list
git add .
git commit -m "$task: Migrate sources.list to bullseye"
apt update
apt upgrade -y
apt dist-upgrade -y
reboot
puppet agent --enable && puppet agent --test

Revisions and Commits

rCJSWH Jenkins jobs
	D6779	rCJSWH5485cc047f22 jenkins: Change node name to "built-in"
rSPSITE puppet-swh-site
	D6757	rSPSITEf7b31905c9ea jenkins: Install required jre in jenkins base profile
	D6754	rSPSITE006aefad8601 public_keys: Update debian repository key for jenkins

Related Objects
Search...

		Status	Assigned	Task
		Migrated	gitlab-migration	T3579 Meta-task: upgrade infrastructure to Debian Bullseye
		Migrated	gitlab-migration	T3770 Migrate jenkins nodes to bullseye

Event Timeline

ardumont triaged this task as Normal priority.Dec 6 2021, 2:30 PM

ardumont created this task.

ardumont updated the task description. (Show Details)Dec 6 2021, 2:38 PM

ardumont added a revision: D6754: public_keys: Update jenkins keys according to latest changes.Dec 6 2021, 5:23 PM

ardumont mentioned this in rSENV7e985f5685d3: Vagrantfile: Add missing jenkins nodes.Dec 6 2021, 5:52 PM

ardumont changed the task status from Open to Work in Progress.Dec 7 2021, 10:05 AM

ardumont moved this task from Backlog to In progress on the System administration (Component upgrades) board.

Vagrant tests in progress btw.
Manifest are incomplete so fixing them along the way.

ardumont added a revision: D6757: jenkins: Install required jre in jenkins base profile.Dec 7 2021, 10:17 AM

Vagrant tests in progress btw.
Manifest are incomplete so fixing them along the way.

Tests went fine (after fixing the puppet limitations).
We can proceed with the actual migration.

I'll attend to it in the afternoon.

ardumont mentioned this in rSPSITE68c4aae28a4e: subnets/vagrant: Update host mapping to integrate jenkins nodes.Dec 7 2021, 2:55 PM

ardumont added a commit: rSPSITE006aefad8601: public_keys: Update debian repository key for jenkins.

ardumont added a commit: rSPSITEf7b31905c9ea: jenkins: Install required jre in jenkins base profile.

ardumont updated the task description. (Show Details)Dec 7 2021, 2:59 PM

As amended in the description, jenkins-debian1 is done.
I'm waiting for the current swh-web release to come through to ensure there is no actual side-effect.

Then, i'll attend to thyssen.

In T3770#75051, @ardumont wrote:

As amended in the description, jenkins-debian1 is done.
I'm waiting for the current swh-web release to come through to ensure there is no actual side-effect.

That build failed somewhat unrelatedly to the migration. Nothing is broken.
It failed because there was a debian/ in the master branch's .gitignore file.
And the now new git version (from bullseye) is be more strict. It refused justly to comply.
The fix is landed (in master, drop that wrong instruction, then release).
I'm pretty sure it's fine now (build still ongoing).

As swh-web's build si too slow, i triggered another (swh.loader.core) and it went fine [1]

[1] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/

Then, i'll attend to thyssen.

So, no more blocker besides people using the ci ;)

Upgrade of thyssen applied (which includes an upgrade of jenkins).
Thyssen is still running buster.

ardumont added a revision: D6779: jenkins: Change node name to built-in.Dec 7 2021, 6:21 PM

ardumont added a commit: rCJSWH5485cc047f22: jenkins: Change node name to "built-in".Dec 7 2021, 6:51 PM

ardumont mentioned this in T3774: Deploy swh.loader.core v2.0.0.Dec 7 2021, 6:54 PM

ardumont updated the task description. (Show Details)Dec 7 2021, 6:59 PM

Heads up, this ui should be pretty cool to prevent build from happening prior to the actual dist-upgrade [1].

[1] https://jenkins.softwareheritage.org/prepareShutdown/

Thyssen:

Prepare jenkins in "prepare shutdown" given this task as notice (using the previous link)
Upgrade according to the plan
Rebooted

So thyssen is now in bullseye.

ardumont@thyssen:~% lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye
ardumont@thyssen:~% uptime
 07:38:13 up 0 min,  2 users,  load average: 2.98, 0.74, 0.25

Remains to check that everything is fine.

Jenkins is responding fine.

ardumont updated the task description. (Show Details)Dec 8 2021, 8:47 AM

There may remains plugin upgrade to do.
I need to check with the team how to properly do that (well how we do it).

ardumont mentioned this in P1239 (bullseye) unbound 1.13 not too happy about listen-docker.conf instruction ([from buster] unbound 1.9).Dec 8 2021, 9:36 AM

unbound was unhappy though [1]

The configuration file (apparently installed manually) had to change:

root@thyssen:~# cat /etc/unbound/unbound.conf.d/listen-docker.conf
interface: 0.0.0.0
interface: ::0
access-control: 127.0.0.0/8 allow
access-control: 172.16.0.0/12 allowa
access-control: 192.168.0.0/16 allow
root@thyssen:~# unbound-checkconf
/etc/unbound/unbound.conf.d/listen-docker.conf:1: error: syntax error
read /etc/unbound/unbound.conf failed: 1 errors in configuration file
root@thyssen:~# unbound -dd
/etc/unbound/unbound.conf.d/listen-docker.conf:1: error: syntax error
read /etc/unbound/unbound.conf failed: 1 errors in configuration file
[1638952385] unbound[6720:0] fatal error: Could not read config file: /etc/unbound/unbound.conf. Maybe try unbound -dd, it stays on the commandline to see more errors, or unbound-checkconf

The correct syntax becomes:

root@thyssen:~# cat /etc/unbound/unbound.conf.d/listen-docker.conf
server:
  interface: 0.0.0.0
  interface: ::0
  access-control: 127.0.0.0/8 allow
  access-control: 172.16.0.0/12 allow
  access-control: 192.168.0.0/16 allow
root@thyssen:~# unbound-checkconf
unbound-checkconf: no errors in /etc/unbound/unbound.conf
root@thyssen:~# unbound -dd
[1638952639] unbound[8445:0] error: cannot open pidfile /run/unbound.pid: Permission denied
[1638952639] unbound[8445:0] notice: init module 0: subnet
[1638952639] unbound[8445:0] notice: init module 1: validator
[1638952639] unbound[8445:0] notice: init module 2: iterator
[1638952639] unbound[8445:0] info: start of service (unbound 1.13.1).
[1638952646] unbound[8445:0] info: generate keytag query _ta-4f66. NULL IN
^C[1638952658] unbound[8445:0] info: service stopped (unbound 1.13.1).
[1638952658] unbound[8445:0] info: server stats for thread 0: 12 queries, 0 answers from cache, 12 recursions, 0 prefetch, 0 rejected by ip ratelimiting
[1638952658] unbound[8445:0] info: server stats for thread 0: requestlist max 1 avg 0.5 exceeded 0 jostled 0
[1638952658] unbound[8445:0] info: average recursion processing time 0.169166 sec
[1638952658] unbound[8445:0] info: histogram of recursion processing times
[1638952658] unbound[8445:0] info: [25%]=0.0229376 median[50%]=0.032768 [75%]=0.218453
[1638952658] unbound[8445:0] info: lower(secs) upper(secs) recursions
[1638952658] unbound[8445:0] info:    0.000512    0.001024 1
[1638952658] unbound[8445:0] info:    0.016384    0.032768 5
[1638952658] unbound[8445:0] info:    0.065536    0.131072 1
[1638952658] unbound[8445:0] info:    0.131072    0.262144 3
[1638952658] unbound[8445:0] info:    0.524288    1.000000 2
CTRL-C
root@thyssen:~# systemctl restart unbound
root@thyssen:~# systemctl status unbound | head -3
● unbound.service - Unbound DNS server
     Loaded: loaded (/lib/systemd/system/unbound.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-12-08 08:38:35 UTC; 4min 59s ago

[1] P1239