Page MenuHomeSoftware Heritage

pypi: Schedule ingestion
Closed, MigratedEdits Locked

Description

Once everything is developed, packaged, puppetized, trigger the ingestion:

  • scheduler updates (manifest, scheduler task type)
  • pypi lister (manifest, lister-pypi db)
  • pypi loader (manifest)
  • swh-workers: Deploy swh-loader-pypi service
  • saatchi: Deploy scheduler update
  • scheduler: Add task_type origin-update-pypi
  • Add scheduler task for listing pypi (and let the magic happen)

Event Timeline

ardumont triaged this task as Normal priority.Sep 6 2018, 5:31 PM
ardumont created this task.
ardumont renamed this task from pypi: Trigger listing task to pypi: Schedule ingestion.Sep 6 2018, 5:37 PM
ardumont updated the task description. (Show Details)
ardumont changed the task status from Open to Work in Progress.Sep 19 2018, 7:52 PM
ardumont updated the task description. (Show Details)

Schedule the lister-pypi:

swhscheduler@saatchi:~$ echo swh-lister-pypi | python3 -m swh.scheduler.cli task schedule -c type -
Created 1 tasks

Task 101693919
  Next run: just now (2018-09-20 07:46:32+00:00)
  Interval: 1 day, 0:00:00
  Type: swh-lister-pypi
  Policy: recurring
  Args:
  Keyword args:

Status:

swhscheduler@saatchi:~$ python3 -m swh.scheduler.cli task list-pending -t swh-lister-pypi
Found 1 tasks

Task 101693919
  Next run: 2 minutes ago (2018-09-20 07:46:32+00:00)
  Interval: 1 day, 0:00:00
  Type: swh-lister-pypi
  Policy: recurring
  Args:
  Keyword args:

Now, it's scheduled. Just need to wait for the swh-scheduler-runner.service to finish its loop on task_types.

ardumont claimed this task.