Page MenuHomeSoftware Heritage

Deploy new listers (gnu, phabricator)
Closed, ResolvedPublic

Description

Deploy and run those:

Steps:

  • D1572: Update defaults.yaml to reference the new listers
  • Tag swh-lister (v0.0.24)
  • Check the debian build package is fine on jenkins [1]
  • Deploy latest listers
  • Initialize the db with phabricator/gnu lister models [2]
  • Give read-only access to new db to guest user
  • Insert new task types (list-gnu-full, load-gnu, list-phab, list-phab-incremental) [3] [4]
  • Deploy latest version changes
  • D1588: Fix phabricator credential setup
  • Deploy latest changes
  • Insert new tasks:
    • 1 recurring list-gnu-full task [5]
    • 1 recurring list-phab (instance swh)
    • 1 recurring list-phab-incremental (instance swh)

[1] https://jenkins.softwareheritage.org/view/Debian%20packages/job/debian/job/packages/job/DLS/job/gbp-buildpackage/

[2] Initialize db stanza:

$ workon swh
$ swh lister db-init --db-url postgres://swh-lister:<redacted>@belvedere.internal.softwareheritage.org:5432/swh-lister gnu
INFO:swh.lister.cli:Initializing lister gnu
INFO:swh.lister.cli:Creating tables for gnu
$ swh lister db-init --db-url postgres://swh-lister:<redacted>@belvedere.internal.softwareheritage.org:5432/swh-lister phabricator
INFO:swh.lister.cli:Initializing lister phabricator
INFO:swh.lister.cli:Creating tables for phabricator

[3] Initialize task-type stanza:

$ SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/
$ swh scheduler --url $SCHEDULER_API_URL task-type add \
    list-gnu-full "swh.lister.gnu.tasks.GNUListerTask" "Full GNU Lister" \
    --default-interval '90 days' \
    --backoff-factor 1
$ swh scheduler --url $SCHEDULER_API_URL task-type add \
    list-phabricator-full "swh.lister.phabricator.tasks.FullPhabricatorLister" "Full Phabricator instance Lister" \
    --default-interval '90 days' \
    --backoff-factor 1
$ swh scheduler --url $SCHEDULER_API_URL task-type add \
    list-phabricator-incremental "swh.lister.phabricator.tasks.IncrementalPhabricatorLister" "Incremental Phabricator instance Lister" \
    --default-interval '1 week' \
    --backoff-factor 1

[4] load-gnu does not exist but we need something in the db for the listing to not fail. That task-type will need to be completed when we actually have something for it.

[5]

swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ task add list-gnu-full --policy recurring
Created 1 tasks

Task 167876594
  Next run: just now (2019-06-14 14:40:31+00:00)
  Interval: 90 days, 0:00:00
  Type: list-gnu-full
  Policy: recurring
  Args:
  Keyword args:

Event Timeline

ardumont triaged this task as Normal priority.
ardumont updated the task description. (Show Details)Wed, Jun 12, 12:09 PM
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)Wed, Jun 12, 12:17 PM
ardumont updated the task description. (Show Details)Wed, Jun 12, 12:27 PM
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)Wed, Jun 12, 3:22 PM
ardumont updated the task description. (Show Details)Wed, Jun 12, 6:13 PM
ardumont updated the task description. (Show Details)Wed, Jun 12, 6:15 PM
ardumont updated the task description. (Show Details)Thu, Jun 13, 4:33 PM
ardumont renamed this task from Deploy latest listers to Deploy new listers (gnu, phabricator).Thu, Jun 13, 5:48 PM
ardumont updated the task description. (Show Details)Fri, Jun 14, 2:43 PM
ardumont updated the task description. (Show Details)Fri, Jun 14, 3:50 PM
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)Fri, Jun 14, 3:52 PM
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)Fri, Jun 14, 4:46 PM
ardumont added a comment.EditedFri, Jun 14, 5:13 PM

I've splitted the task in 2.

The gnu lister was straight forward.
It took a bit of more time than anticipated because there was a migration in the middle. Still it's done now.

The phabricator one may not be completely ready though.
The authentication token is passed as a parameter of the lister (and not read from some configuration file as the other listers do, that'd be great to have that as a fallback).
Nothing too bad but still, i cannot deploy immediately.

ardumont updated the task description. (Show Details)Mon, Jun 17, 5:48 PM
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)Mon, Jun 17, 6:37 PM
ardumont changed the task status from Open to Work in Progress.
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)Mon, Jun 17, 6:43 PM
ardumont closed this task as Resolved.