Change Details

The re-engineering of the GitHub lister implies using celery tasks for the "incremental" and "full" operations, so they can be integrated in the scheduler. Specifically for the GitHub lister, three kinds of tasks are implied, divided in two queues (the incremental operation can be parallelized with the full operation): - `swh_lister_github_incremental` queue - incremental task maps to the current `ghlister catchup` operation - `swh_lister_github_full` queue - full "meta"-task that schedules range updates - update-range task that maps to the current `ghlister list <start>-<stop>` operation which takes a range of ids, and can be parallelized.

The re-engineering of the GitHub lister implies using celery tasks for the "incremental" and "full" operations, so they can be integrated in the scheduler. Specifically for the GitHub lister, three kinds of tasks are implied, divided in two queues (the incremental operation needs to be able to bypass the queue for the full operation): - `swh_lister_github_incremental` queue - incremental task maps to the current `ghlister catchup` operation - `swh_lister_github_full` queue - full "meta"-task that schedules range updates - update-range task that maps to the current `ghlister list <start>-<stop>` operation which takes a range of ids, and can be parallelized.

The re-engineering of the GitHub lister implies using celery tasks for the "incremental" and "full" operations, so they can be integrated in the scheduler. Specifically for the GitHub lister, three kinds of tasks are implied, divided in two queues (the incremental operation can be parallelized withneeds to be able to bypass the queue for the full operation): - `swh_lister_github_incremental` queue - incremental task maps to the current `ghlister catchup` operation - `swh_lister_github_full` queue - full "meta"-task that schedules range updates - update-range task that maps to the current `ghlister list <start>-<stop>` operation which takes a range of ids, and can be parallelized.