Deploy next-gen listers on production
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	ardumont
	Feb 3 2021, 2:01 PM

Description

"Just" a matter upgrading workers with latest python3-swh.lister v0.8.0 and restart swh-worker@lister service

And, first, for cgit listers, prepare a migration script (so the listing is more efficient for those instances) [1]

Second, schedule the eclipse cgit instance to be listed [2]

[1] meaning adding the base_git_url to the arguments#>>'{kwargs}' dict of the task, T2999#57621

[2] T376

Related Objects

Mentioned In: T376: ingest git.eclipse.org repositories
Mentioned Here: T376: ingest git.eclipse.org repositories
T2999: Optimize the number of HTTP requests sent by the cgit lister

Event Timeline

ardumont triaged this task as Normal priority.Feb 3 2021, 2:01 PM

ardumont created this task.

ardumont moved this task from Backlog to Weekly backlog on the System administration board.

And, first, for cgit listers, prepare a migration script for the existing scheduled
cgit task instances to add the base_git_url to the arguments/kwargs dict [1]

Retrieve production information on cgit current instances:

softwareheritage-scheduler=> select arguments, id from task where type='list-cgit' order by id;
                                                 arguments                                                  |    id
------------------------------------------------------------------------------------------------------------+-----------
 {"args": [], "kwargs": {"url": "https://git.savannah.gnu.org/cgit/", "instance": "gnu-savannah"}}          | 168392627
 {"args": [], "kwargs": {"url": "https://gitweb.torproject.org/", "instance": "tor"}}                       | 168393646
 {"args": [], "kwargs": {"url": "https://git.kernel.org/", "instance": "git-kernel"}}                       | 168394139
 {"args": [], "kwargs": {"url": "https://fedorapeople.org/cgit/", "instance": "fedora"}}                    | 168395142
 {"args": [], "kwargs": {"url": "https://cgit.kde.org/", "instance": "kde"}}                                | 168395984
 {"args": [], "kwargs": {"url": "https://www.happyassassin.net/cgit/", "instance": "happyassassin"}}        | 168395985
 {"args": [], "kwargs": {"url": "https://git.openembedded.org/", "instance": "openembedded"}}               | 168395986
 {"args": [], "kwargs": {"url": "https://git.zx2c4.com/", "instance": "zx2c4"}}                             | 168395987
 {"args": [], "kwargs": {"url": "http://git.gnu.org.ua/cgit/", "instance": "git.gnu.org.ua"}}               | 168395988
 {"args": [], "kwargs": {"url": "https://git.alpinelinux.org/", "instance": "alpinelinux"}}                 | 168395989
 {"args": [], "kwargs": {"url": "http://hdiff.luite.com/cgit/", "instance": "hdiff.luite"}}                 | 168395990
 {"args": [], "kwargs": {"url": "https://git.baserock.org/cgit/", "instance": "baserock"}}                  | 168395991
 {"args": [], "kwargs": {"url": "https://code.qt.io/cgit/", "instance": "qt.io"}}                           | 168395992
 {"args": [], "kwargs": {"url": "http://git.yoctoproject.org/clean/cgit.cgi/", "instance": "yoctoproject"}} | 168395993
 {"args": [], "kwargs": {"url": "https://git.joeyh.name/"}}                                                 | 219341472

Upgrade script:

update task set arguments='{"args": [], "kwargs": {"url": "https://git.savannah.gnu.org/cgit/", "instance": "gnu-savannah", "base_git_url": "https://git.savannah.gnu.org/git/"}}' where id=168392627;
update task set arguments='{"args": [], "kwargs": {"url": "http://hdiff.luite.com/cgit/", "instance": "hdiff.luite", "base_git_url": "http://hdiff.luite.com/cgit/"}}' where id=168395990;
update task set arguments='{"args": [], "kwargs": {"url": "https://git.baserock.org/cgit/", "instance": "baserock", "base_git_url": "https://git.baserock.org/git/"}}' where id=168395991;
update task set arguments='{"args": [], "kwargs": {"url": "https://git.kernel.org/", "instance": "git-kernel", "base_git_url": "https://git.kernel.org/"}}' where id=168394139;
update task set arguments='{"args": [], "kwargs": {"url": "https://fedorapeople.org/cgit/", "instance": "fedora", "base_git_url": "https://fedorapeople.org/cgit/"}}' where id=168395142;
update task set arguments='{"args": [], "kwargs": {"url": "https://gitweb.torproject.org/", "instance": "tor", "base_git_url": "https://gitweb.torproject.org/"}}' where id=168393646;
update task set arguments='{"args": [], "kwargs": {"url": "https://code.qt.io/cgit/", "instance": "qt.io", "base_git_url": "http://code.qt.io/"}}' where id=168395992;
update task set arguments='{"args": [], "kwargs": {"url": "https://git.zx2c4.com/", "instance": "zx2c4", "base_git_url": "https://git.zx2c4.com/"}}' where id=168395987;
update task set arguments='{"args": [], "kwargs": {"url": "http://git.gnu.org.ua/cgit/", "instance": "git.gnu.org.ua", "base_git_url": "https://git.gnu.org.ua/repo/"}}' where id=168395988;
update task set arguments='{"args": [], "kwargs": {"url": "https://git.openembedded.org/", "instance": "openembedded", "base_git_url": "https://git.openembedded.org/"}}' where id=168395986;
update task set arguments='{"args": [], "kwargs": {"url": "https://git.joeyh.name/", "base_git_url": "https://git.joeyh.name/git/"}}' where id=219341472;
update task set arguments='{"args": [], "kwargs": {"url": "http://git.yoctoproject.org/clean/cgit.cgi/", "instance": "yoctoproject", "base_git_url": "https://git.yoctoproject.org/git/"}}' where id=168395993;

-- not touched because some urls cannot be computed
-- https://git.alpinelinux.org/

-- not touched because site down
-- https://www.happyassassin.net/cgit/
-- https://cgit.kde.org/

After tryout:

begin;
-- run previous script above
...
softwareheritage-scheduler=> select arguments, id from task where type='list-cgit' order by id;
                                                                            arguments                                                                            |    id
-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------
 {"args": [], "kwargs": {"url": "https://git.savannah.gnu.org/cgit/", "instance": "gnu-savannah", "base_git_url": "https://git.savannah.gnu.org/git/"}}          | 168392627
 {"args": [], "kwargs": {"url": "https://gitweb.torproject.org/", "instance": "tor", "base_git_url": "https://gitweb.torproject.org/"}}                          | 168393646
 {"args": [], "kwargs": {"url": "https://git.kernel.org/", "instance": "git-kernel", "base_git_url": "https://git.kernel.org/"}}                                 | 168394139
 {"args": [], "kwargs": {"url": "https://fedorapeople.org/cgit/", "instance": "fedora", "base_git_url": "https://fedorapeople.org/cgit/"}}                       | 168395142
 {"args": [], "kwargs": {"url": "https://cgit.kde.org/", "instance": "kde"}}                                                                                     | 168395984
 {"args": [], "kwargs": {"url": "https://www.happyassassin.net/cgit/", "instance": "happyassassin"}}                                                             | 168395985
 {"args": [], "kwargs": {"url": "https://git.openembedded.org/", "instance": "openembedded", "base_git_url": "https://git.openembedded.org/"}}                   | 168395986
 {"args": [], "kwargs": {"url": "https://git.zx2c4.com/", "instance": "zx2c4", "base_git_url": "https://git.zx2c4.com/"}}                                        | 168395987
 {"args": [], "kwargs": {"url": "http://git.gnu.org.ua/cgit/", "instance": "git.gnu.org.ua", "base_git_url": "https://git.gnu.org.ua/repo/"}}                    | 168395988
 {"args": [], "kwargs": {"url": "https://git.alpinelinux.org/", "instance": "alpinelinux"}}                                                                      | 168395989
 {"args": [], "kwargs": {"url": "http://hdiff.luite.com/cgit/", "instance": "hdiff.luite", "base_git_url": "http://hdiff.luite.com/cgit/"}}                      | 168395990
 {"args": [], "kwargs": {"url": "https://git.baserock.org/cgit/", "instance": "baserock", "base_git_url": "https://git.baserock.org/git/"}}                      | 168395991
 {"args": [], "kwargs": {"url": "https://code.qt.io/cgit/", "instance": "qt.io", "base_git_url": "http://code.qt.io/"}}                                          | 168395992
 {"args": [], "kwargs": {"url": "http://git.yoctoproject.org/clean/cgit.cgi/", "instance": "yoctoproject", "base_git_url": "https://git.yoctoproject.org/git/"}} | 168395993
 {"args": [], "kwargs": {"url": "https://git.joeyh.name/", "base_git_url": "https://git.joeyh.name/git/"}}                                                       | 219341472
(15 rows)

rollback;

This ^ sounds good enough.

ardumont updated the task description. (Show Details)Feb 3 2021, 3:20 PM

ardumont updated the task description. (Show Details)Feb 3 2021, 3:23 PM

ardumont updated the task description. (Show Details)Feb 3 2021, 3:26 PM

ardumont changed the task status from Open to Work in Progress.Feb 3 2021, 6:05 PM

ardumont moved this task from Weekly backlog to in-progress on the System administration board.

migration script ran
restart swh-worker@lister
disable down cgit instance (kde, happyassassin)

-- site down
update task set status='disabled' where id=168395984;  -- deactivate cgit.kde.org
update task set status='disabled' where id=168395985;  -- deactivate happyassassin

Everything is fine:

softwareheritage-scheduler=> select status, arguments#>>'{kwargs,url}' as url from task where type='list-cgit' order by status;
         status         |                     url
------------------------+---------------------------------------------
 next_run_not_scheduled | https://git.alpinelinux.org/
 next_run_not_scheduled | https://git.baserock.org/cgit/
 next_run_not_scheduled | http://hdiff.luite.com/cgit/
 next_run_not_scheduled | https://git.savannah.gnu.org/cgit/
 next_run_not_scheduled | https://git.joeyh.name/
 next_run_not_scheduled | https://git.openembedded.org/
 next_run_not_scheduled | http://git.gnu.org.ua/cgit/
 next_run_not_scheduled | https://git.zx2c4.com/
 next_run_not_scheduled | https://code.qt.io/cgit/
 next_run_not_scheduled | https://gitweb.torproject.org/
 next_run_not_scheduled | https://fedorapeople.org/cgit/
 next_run_not_scheduled | https://git.kernel.org/
 next_run_scheduled     | http://git.yoctoproject.org/clean/cgit.cgi/
 disabled               | https://cgit.kde.org/
 disabled               | https://www.happyassassin.net/cgit/

I took the opportunity to unstuck yoctoproject which was in 03/2020 as the next-run-scheduled listing.
It ran with the new lister version and now urls are referenced in the scheduler table listed_origins.

Scheduler cgit lister for instance eclipse

 swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task add list-cgit url=https://git.eclipse.org/c/ instance=eclipse base_git_url=https://git.eclipse.org/r/
Created 1 tasks

Task 358860307
  Next run: today (2021-02-04T08:26:28.104180+00:00)
  Interval: 64 days, 0:00:00
  Type: list-cgit
  Policy: recurring
  Args:
  Keyword args:
    base_git_url: 'https://git.eclipse.org/r/'
    instance: 'eclipse'
    url: 'https://git.eclipse.org/c/'

ardumont moved this task from in-progress to deployed/landed/monitoring on the System administration board.Feb 4 2021, 9:28 AM

ardumont mentioned this in T376: ingest git.eclipse.org repositories.

ardumont closed this task as Resolved.Feb 4 2021, 12:42 PM

ardumont moved this task from deployed/landed/monitoring to done on the System administration board.

This task has been migrated to GitLab.

Deploy next-gen listers on productionClosed, MigratedEdits LockedActions

Description

Related Objects

Event Timeline

Deploy next-gen listers on production
Closed, MigratedEdits Locked
Actions