Launchpad lister landed in master (and packaged).
Deploy and run instance in staging.
Goals:
- ensure it's running fine (unit/integration tests: ok, on docker: ok)
- fix the missed spots if any
- manifest created for staging will be reused for production
Description
Description
Revisions and Commits
Revisions and Commits
| rDLS Listers | |||
| D3887 | rDLS74ca3d0b8732 Launchpad: rename task name to match conventions | ||
| rSPSITE puppet-swh-site | |||
| D3884 | rSPSITEe966d29e4d3a lister configuration: Add launchpad lister tasks | ||
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Migrated | gitlab-migration | T1734 Create a Lister for launchpad.net | ||
| Migrated | gitlab-migration | T2358 Deploy launchpad lister on staging |
Event Timeline
Comment Actions
FTR, I've run the launchpad lister in a docker and it executed fine, with fine being "it created 19340 load-git tasks"
Comment Actions
The launchpad lister (v0.1.2) is deployed and running on staging
- puppet configuration applied
- task type declared
swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register -p lister.launchpad WARNING:swh.core.cli:Could not load subcommand storage: No module named 'swh.journal' INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.launchpad INFO:swh.scheduler.cli.task_type:Create task type launchpad-lister-incremental in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-full in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-new in scheduler
- launchpad-lister-incremental task type renamed to match the conventions (fixed in https://forge.softwareheritage.org/D3884 , will be released in swh-lister@v0.1.3). On `db0`:
swh-scheduler=# begin; update set type='list-launchpad-incremental' where type='launchpad-lister-incremental';commit;
- database schema updated to create the new lister table :
swhscheduler@scheduler0:~$ swh lister --db-url postgresql://swh-lister:******@db0.*****:5432/swh-lister db-init WARNING:swh.core.cli:Could not load subcommand storage: No module named 'swh.journal' INFO:swh.lister.cli:Loading lister bitbucket INFO:swh.lister.cli:Loading lister cgit INFO:swh.lister.cli:Loading lister cran INFO:swh.lister.cli:Loading lister debian INFO:swh.lister.cli:Loading lister gitea INFO:swh.lister.cli:Loading lister github INFO:swh.lister.cli:Loading lister gitlab INFO:swh.lister.cli:Loading lister gnu INFO:swh.lister.cli:Loading lister launchpad INFO:swh.lister.cli:Loading lister npm INFO:swh.lister.cli:Loading lister packagist INFO:swh.lister.cli:Loading lister phabricator INFO:swh.lister.cli:Loading lister pypi INFO:swh.lister.cli:Initializing database INFO:swh.lister.core.models:Creating tables INFO:swh.lister.cli:Calling init hook for debian
- launchpad task created :
swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task add list-launchpad-full WARNING:swh.core.cli:Could not load subcommand storage: No module named 'swh.journal' INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml Created 1 tasks Task 1240540 Next run: just now (2020-09-08 13:08:07+00:00) Interval: 90 days, 0:00:00 Type: list-launchpad-full Policy: recurring Args: Keyword args:
- workers restarted :
# clush -b -w @staging-workers systemctl restart swh-worker@lister
Comment Actions
The task ran in 30mn (1887s):
Sep 08 13:45:34 worker1 python3[237586]: [2020-09-08 13:45:34,851: INFO/ForkPoolWorker-4] Task swh.lister.launchpad.tasks.FullLaunchpadLister[73e298be-aeda-4882-b52d-dfe5a2ec316c] succeeded in 1887.75128286588s: {'status': 'eventful'}Actually there is 19490 repository (only git type) :
swh-lister=# select count(*) from launchpad_repo; count ------- 19490 (1 row)
swh-lister=# select distinct origin_type from launchpad_repo; -[ RECORD 1 ]---- origin_type | git
Comment Actions
An email was sent on the swh-devel mailing list to ask for reviews.
The deployment in production will be performed in the middle of week 38 is no problems are raised.