Page MenuHomeSoftware Heritage

Deploy new listers (gnu, phabricator)
Closed, MigratedEdits Locked

Description

Deploy and run those:

Steps:

  • D1572: Update defaults.yaml to reference the new listers
  • Tag swh-lister (v0.0.24)
  • Check the debian build package is fine on jenkins [1]
  • Deploy latest listers
  • Initialize the db with phabricator/gnu lister models [2]
  • Give read-only access to new db to guest user
  • Insert new task types (list-gnu-full, load-gnu, list-phab, list-phab-incremental) [3] [4]
  • Deploy latest version changes
  • D1588: Fix phabricator credential setup
  • Deploy latest changes
  • Insert new tasks:
    • 1 recurring list-gnu-full task [5]
    • 1 recurring list-phab (instance swh)
    • 1 recurring list-phab-incremental (instance swh)

[1] https://jenkins.softwareheritage.org/view/Debian%20packages/job/debian/job/packages/job/DLS/job/gbp-buildpackage/

[2] Initialize db stanza:

$ workon swh
$ swh lister db-init --db-url postgres://swh-lister:<redacted>@belvedere.internal.softwareheritage.org:5432/swh-lister gnu
INFO:swh.lister.cli:Initializing lister gnu
INFO:swh.lister.cli:Creating tables for gnu
$ swh lister db-init --db-url postgres://swh-lister:<redacted>@belvedere.internal.softwareheritage.org:5432/swh-lister phabricator
INFO:swh.lister.cli:Initializing lister phabricator
INFO:swh.lister.cli:Creating tables for phabricator

[3] Initialize task-type stanza:

$ SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/
$ swh scheduler --url $SCHEDULER_API_URL task-type add \
    list-gnu-full "swh.lister.gnu.tasks.GNUListerTask" "Full GNU Lister" \
    --default-interval '90 days' \
    --backoff-factor 1
$ swh scheduler --url $SCHEDULER_API_URL task-type add \
    list-phabricator-full "swh.lister.phabricator.tasks.FullPhabricatorLister" "Full Phabricator instance Lister" \
    --default-interval '90 days' \
    --backoff-factor 1
$ swh scheduler --url $SCHEDULER_API_URL task-type add \
    list-phabricator-incremental "swh.lister.phabricator.tasks.IncrementalPhabricatorLister" "Incremental Phabricator instance Lister" \
    --default-interval '1 week' \
    --backoff-factor 1

[4] load-gnu does not exist but we need something in the db for the listing to not fail. That task-type will need to be completed when we actually have something for it.

[5]

swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ task add list-gnu-full --policy recurring
Created 1 tasks

Task 167876594
  Next run: just now (2019-06-14 14:40:31+00:00)
  Interval: 90 days, 0:00:00
  Type: list-gnu-full
  Policy: recurring
  Args:
  Keyword args:

Event Timeline

ardumont triaged this task as Normal priority.Jun 12 2019, 11:12 AM
ardumont created this task.
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)
ardumont renamed this task from Deploy latest listers to Deploy new listers (gnu, phabricator).Jun 13 2019, 5:48 PM
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)

I've splitted the task in 2.

The gnu lister was straight forward.
It took a bit of more time than anticipated because there was a migration in the middle. Still it's done now.

The phabricator one may not be completely ready though.
The authentication token is passed as a parameter of the lister (and not read from some configuration file as the other listers do, that'd be great to have that as a fallback).
Nothing too bad but still, i cannot deploy immediately.

ardumont updated the task description. (Show Details)
ardumont changed the task status from Open to Work in Progress.Jun 17 2019, 6:37 PM
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)
gitlab-migration changed the status of subtask T1809: Deploy phabricator lister on swh forge from Resolved to Migrated.