⚓ T3915 Deploy Bzr loader to the staging environment

swh-doco exec swh-loader swh loader run | grep bzr
+ cd /home/tony/work/inria/repo/swh/swh-environment/docker
+ docker-compose -f docker-compose.yml -f docker-compose.override.yml exec swh-loader swh loader run
Usage: swh loader run [OPTIONS] [archive|bzr|cran|
Error: Missing argument '[archive|bzr|cran|debian|deposit|git|git_disk|maven|mercurial|nixguix|npm|opam|pypi|svn]'.  Choose from:
        bzr,

Which made me realize we need to create the debian package. I'll attend to it after
having making the docker part ok.

ardumont updated the task description. (Show Details)Feb 8 2022, 2:55 PM

ardumont changed the task status from Open to Work in Progress.Feb 8 2022, 5:08 PM

ardumont moved this task from Weekly backlog to in-progress on the System administration board.

ardumont added a revision: D7124: Allow loader-bzr to run within docker.Feb 8 2022, 5:10 PM

loader-bzr run in docker \o/:

$ swh-doco exec swh-loader swh loader run bzr https://launchpad.net/wayland-protocols
+ cd /home/tony/work/inria/repo/swh/swh-environment/docker
+ docker-compose -f docker-compose.yml -f docker-compose.override.yml exec swh-loader swh loader run bzr https://launchpad.net/wayland-protocols
INFO:swh.loader.bzr.Loader:Load origin 'https://launchpad.net/wayland-protocols' with type 'bzr'
INFO:brz:Created new control directory.
{'status': 'eventful'}

Repository configured correctly regarding the ci.
But the hook was missing on the repository.
Fixed [1].
Let's do the first release.

Hook post-receive-swh-modules successfully installed on /srv/phabricator/repos/240/:
lrwxrwxrwx 1 phabricator phabricator 39 Feb  9 09:19 post-receive -> ../../../hooks/post-receive-swh-modules

[1] https://docs.softwareheritage.org/sysadm/deployment/howto-debian-packaging.html#setting-up-the-repository-on-phabricator

Here we go, first release is up. [1]

Now on to the debian part.

[1] https://pypi.org/project/swh.loader.bzr/

ardumont mentioned this in rDENV79b21876c6a7: .mrconfig: Reference the swh-loader-bzr repository.Feb 9 2022, 11:10 AM

ardumont mentioned this in rDENV7727114e507b: bin/debpkg-bootstrap-branches: Fix idempotency call.

ardumont updated the task description. (Show Details)Feb 9 2022, 11:14 AM

Prepare the necessary debian metadata files to allow CI package build [1]

Build is fine on unstable [2] (python3-breezy ~3.2).
Less so in stable [3] (current loader wants to use some function not present yet in python3-breezy 3.0.0)

[1]

$ ../bin/debpkg-bootstrap-branches v0.1.0 bzr python3-swh.core.db.pytestplugin
$ swh-debian-build-unstable  # local wrapper to build locally the package a-la ci
...
$ swh-debian-build-sign  # sign stuff to trigger the build on ci
$ git push origin --follow-tags  # push and let the ci build stuff

[2] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DLDBZR/job/gbp-buildpackage/1/console

[3] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DLDBZR/job/gbp-buildpackage/2/console

ardumont updated the task description. (Show Details)Feb 9 2022, 11:28 AM

ardumont added a revision: D7132: bzr: Allow cloning with breezy version without cmd_clone function.Feb 9 2022, 2:31 PM

ardumont updated the task description. (Show Details)Feb 9 2022, 2:49 PM

ardumont updated the task description. (Show Details)Feb 9 2022, 3:07 PM

ardumont added a revision: D7135: tests: Reference tasks module for tests to be ok with debian build.Feb 9 2022, 4:00 PM

ardumont added a commit: rDLDBZR907efa0ae1f3: MANIFEST.in: Reference the tests data folder.Feb 9 2022, 4:14 PM

ardumont added a commit: rDLDBZR5077af52cb1d: tests: Reference tasks module for tests to be ok with debian build.

ardumont added a commit: rDENVdd5e80612fd1: Allow loader-bzr to run within docker.Feb 9 2022, 7:02 PM

D7130 has been committed, so a new version of the loader for faster ingestion will need to be released.

Yes, thanks for the heads up.

For information, I still need to debug the other debian package failure first anyway (buster) [1].

[1] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DLDBZR/job/gbp-buildpackage/5/console

ardumont added a revision: D7150: test: Simplify test_needs_upgrade.Feb 10 2022, 4:09 PM

ardumont added a commit: rDLDBZR551e32945f71: test: Simplify test_needs_upgrade.Feb 10 2022, 4:51 PM

Previous one fixed.
pypi-upload job failed for some reason.
Fixed.

And now, another error [1]

[1] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DLDBZR/job/gbp-buildpackage/7/console

ardumont mentioned this in rPPDWbd538eb3ac9e: Drop no longer required patch which is integrated upstream.Feb 11 2022, 10:47 AM

ardumont mentioned this in rPPDW761a8712bc9d: d/changelog: Release backport upgrade.

ardumont mentioned this in rPPDW83b56a1cff05: d/changelog: Release backport upgrade.

ardumont mentioned this in rDLDBZR5d5077df513b: d/rules: Drop swh/loader/__init__.py.Feb 11 2022, 5:46 PM

After some more fight and backporting dependencies [1] [2]. The build is now happy on
stable as well [3].

[1]

$ swhdebianrepo@pergamon:/srv/softwareheritage/repository$ for p in python3-fastbencode python3-patiencediff python3-breezy; do reprepro ls $p; done
python3-fastbencode | 0.0.5-1~swh1 | buster-swh | amd64
python3-patiencediff | 0.2.1-1~swh1 | buster-swh | amd64
python3-breezy | 3.2.1+bzr7585-1~swh1 | buster-swh | amd64

[1] I spent some time trying to figure out stuff (multiple paths taken). Only 1 path
worked, rebuilding locally the package and lessen the python3 deps to the stable one
(3.7). Then uploaded the deps to our reprepro instance. That finally make the
swh-loader-bzr build ok on stable. took the easy way and just rebuild manually and
uploaded the deb package

[3] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DLDBZR/job/gbp-buildpackage/10/

Of course, the build was ok, but not the actual install [1]
Another build release ongoing to fix that... [2]

And it's now ok.
Landing D7112 now.

[1] P1287

[2] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DLDBZR/job/gbp-buildpackage/

[3]

 vagrant node:
root@worker0:~# apt install python3-swh.loader.bzr
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  python3-swh.loader.bzr
0 upgraded, 1 newly installed, 0 to remove and 10 not upgraded.
Need to get 52.9 kB of archives.
After this operation, 123 kB of additional disk space will be used.
Get:1 https://debian.softwareheritage.org buster-swh/main amd64 python3-swh.loader.bzr all 1.1.0-1~swh3~bpo10+1 [52.9 kB]
Fetched 52.9 kB in 0s (881 kB/s)
(Reading database ... 85784 files and directories currently installed.)
Preparing to unpack .../python3-swh.loader.bzr_1.1.0-1~swh3~bpo10+1_all.deb ...
Unpacking python3-swh.loader.bzr (1.1.0-1~swh3~bpo10+1) ...
Setting up python3-swh.loader.bzr (1.1.0-1~swh3~bpo10+1) ...
root@worker0:~# dpkg -l python3-swh.loader.bzr | grep ii
ii  python3-swh.loader.bzr 1.1.0-1~swh3~bpo10+1 all          Software Heritage Bazaar/Breezy intent

ardumont added a commit: rSPSITE7bee2b18f918: Deploy swh-worker@loader_bzr service to staging workers.Feb 11 2022, 5:52 PM

And deployed.
Next step: Actually feeding them some bzr origins.
We'll need to discuss some more with @Alphare.

root@pergamon:/srv/softwareheritage/repository# clush -b -w @staging-loader-workers "systemctl status swh-worker@loader_bzr" | grep Active
     Active: active (running) since Fri 2022-02-11 16:55:54 UTC; 13s ago
     Active: active (running) since Fri 2022-02-11 16:55:54 UTC; 14s ago
     Active: active (running) since Fri 2022-02-11 16:55:54 UTC; 14s ago

ardumont updated the task description. (Show Details)Feb 11 2022, 5:57 PM

ardumont updated the task description. (Show Details)

ardumont moved this task from in-progress to deployed/landed/monitoring on the System administration board.Feb 11 2022, 6:11 PM

@Alphare fixed the sourceforge lister to list correctly bzr origins.
Tagged a v2.6.4 with that fix in the intent to deploy this on staging.

ardumont updated the task description. (Show Details)Feb 14 2022, 5:35 PM

ardumont updated the task description. (Show Details)Feb 15 2022, 10:11 AM

New lister did its work, 74 new origins got seen [1] [2].

Scheduler did not schedule them though.
Ah, I missed the new task tyep registering in the scheduler db.
It's done now.

And now the bzr origins are scheduled properly [3].
The old and invalid ones will fail.
Hopefully some of the new ones will work.

[1] P1289

[2]

10:04:28 swh-scheduler@db1:5432=> select now(), count(*) from listed_origins where visit_type='bzr';
select now(), count(*) from listed_origins where visit_type='bzr';
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-02-15 09:05:26.474038+00 |   364 |
+-------------------------------+-------+
(1 row)

Time: 705.564 ms

[3]

Feb 15 09:11:07 scheduler0 swh[1373001]: INFO:swh.scheduler.celery_backend.recurrent_visits:bzr: 364 visits scheduled in queue swh.loader.bzr.tasks.LoadBazaar

ardumont closed this task as Resolved.Feb 16 2022, 3:14 PM

ardumont claimed this task.

ardumont updated the task description. (Show Details)

ardumont moved this task from deployed/landed/monitoring to done on the System administration board.

This task has been migrated to GitLab.

rDLDBZR BZR loader
	Abandoned		D7132 bzr: Allow cloning with breezy version without cmd_clone function
		D7150	rDLDBZR551e32945f71 test: Simplify test_needs_upgrade
		D7135	rDLDBZR5077af52cb1d tests: Reference tasks module for tests to be ok with debian build
		D7135	rDLDBZR907efa0ae1f3 MANIFEST.in: Reference the tests data folder
rDENV Development environment
		D7124	rDENVdd5e80612fd1 Allow loader-bzr to run within docker
rDLS Listers
	Closed		D7172 sourceforge: fix support for listing bzr origins
rSPSITE puppet-swh-site
		D7112	rSPSITE7bee2b18f918 Deploy swh-worker@loader_bzr service to staging workers

Deploy Bzr loader to the staging environment
Closed, MigratedEdits Locked
Actions

Description

Revisions and Commits

Related Objects
Search...

Event Timeline

Deploy Bzr loader to the staging environmentClosed, MigratedEdits LockedActions

Description

Revisions and Commits

Related ObjectsSearch...

Event Timeline

Deploy Bzr loader to the staging environment
Closed, MigratedEdits Locked
Actions

Related Objects
Search...