Page MenuHomeSoftware Heritage

Sprint 2021 01Goal
ActivePublic

Details

Description

Scheduler/Lister refactoring

Recent Activity

Oct 28 2021

ardumont closed T3667: Orchestrate origins scheduling according to scheduler metrics feedback, a subtask of T2345: Improve handling of recurrent loading tasks in scheduler, as Resolved.
Oct 28 2021, 5:13 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont closed T3667: Orchestrate origins scheduling according to scheduler metrics feedback as Resolved.
Oct 28 2021, 5:13 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont moved T3667: Orchestrate origins scheduling according to scheduler metrics feedback from in-progress to deployed/landed on the System administration board.
Oct 28 2021, 4:43 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont moved T3667: Orchestrate origins scheduling according to scheduler metrics feedback from Backlog to in-progress on the System administration board.
Oct 28 2021, 4:43 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a project to T3667: Orchestrate origins scheduling according to scheduler metrics feedback: System administration.
Oct 28 2021, 4:42 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a comment to T3667: Orchestrate origins scheduling according to scheduler metrics feedback.

Deployed in production.

Oct 28 2021, 4:42 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont changed the status of T3667: Orchestrate origins scheduling according to scheduler metrics feedback, a subtask of T2345: Improve handling of recurrent loading tasks in scheduler, from Open to Work in Progress.
Oct 28 2021, 4:34 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont changed the status of T3667: Orchestrate origins scheduling according to scheduler metrics feedback from Open to Work in Progress.

Deployed in staging.

Oct 28 2021, 4:34 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
olasd added a revision to T3667: Orchestrate origins scheduling according to scheduler metrics feedback: D6583: scheduler: Add service to schedule recurrent visits.
Oct 28 2021, 4:16 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a revision to T3667: Orchestrate origins scheduling according to scheduler metrics feedback: D6574: scheduler: Add schedule recurrent tasks service.
Oct 28 2021, 11:35 AM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

Oct 22 2021

ardumont added a revision to T3667: Orchestrate origins scheduling according to scheduler metrics feedback: D6543: docker: Install scheduler runner which deals with recurrent tasks.
Oct 22 2021, 5:19 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont updated the task description for T3667: Orchestrate origins scheduling according to scheduler metrics feedback.
Oct 22 2021, 1:51 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont updated the task description for T3667: Orchestrate origins scheduling according to scheduler metrics feedback.
Oct 22 2021, 12:21 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

Oct 20 2021

ardumont added a revision to T3667: Orchestrate origins scheduling according to scheduler metrics feedback: D6520: Add a new cli endpoint to schedule recurrent visits in Celery.
Oct 20 2021, 4:50 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a subtask for T2345: Improve handling of recurrent loading tasks in scheduler: T3674: Determine how to reference manually listed/ingested origins in the scheduler metrics.
Oct 20 2021, 12:19 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a revision to T3667: Orchestrate origins scheduling according to scheduler metrics feedback: D6515: Drop deprecated listener module.
Oct 20 2021, 10:45 AM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a revision to T3667: Orchestrate origins scheduling according to scheduler metrics feedback: D6514: Add docstring to runner and listener modules.
Oct 20 2021, 10:44 AM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

Oct 18 2021

ardumont updated the task description for T3667: Orchestrate origins scheduling according to scheduler metrics feedback.
Oct 18 2021, 3:09 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont updated the task description for T3667: Orchestrate origins scheduling according to scheduler metrics feedback.
Oct 18 2021, 2:37 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont updated the task description for T3667: Orchestrate origins scheduling according to scheduler metrics feedback.
Oct 18 2021, 12:25 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont triaged T3667: Orchestrate origins scheduling according to scheduler metrics feedback as High priority.
Oct 18 2021, 12:24 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

Sep 14 2021

ardumont closed T3538: Send scheduler metrics to prometheus, a subtask of T2345: Improve handling of recurrent loading tasks in scheduler, as Resolved.
Sep 14 2021, 11:00 AM · Sprint 2021 01, Archive coverage, Scheduling utilities

Sep 3 2021

ardumont added a subtask for T2345: Improve handling of recurrent loading tasks in scheduler: T3538: Send scheduler metrics to prometheus.
Sep 3 2021, 5:17 PM · Sprint 2021 01, Archive coverage, Scheduling utilities

Aug 26 2021

ardumont added a comment to T2345: Improve handling of recurrent loading tasks in scheduler.

What's next, as a summary, subsequent subtasks should be created later:

Aug 26 2021, 5:45 PM · Sprint 2021 01, Archive coverage, Scheduling utilities

Aug 13 2021

ardumont added a comment to T2345: Improve handling of recurrent loading tasks in scheduler.
  • Refactor a bit the journal client to update a docstring and inline one function (done, that'd be the 2 previous commits mentioned here just below that comment ^).
  • Deactivate failing visits (delegating to listers the act of activating back those origins which gets live again). I have diffs which deal with this that needs some rebase and work according to latest change (I need to get back to it) [1].
  • Deploy the current scheduler implementation (master) when that previous point is done. (That's gonna be my goal to reach prior to some vacation break).
Aug 13 2021, 4:44 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont closed T3471: production: Deploy swh.scheduler v0.17, a subtask of T2345: Improve handling of recurrent loading tasks in scheduler, as Resolved.
Aug 13 2021, 3:48 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont changed the status of T3471: production: Deploy swh.scheduler v0.17, a subtask of T2345: Improve handling of recurrent loading tasks in scheduler, from Open to Work in Progress.
Aug 13 2021, 10:33 AM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a comment to T3471: production: Deploy swh.scheduler v0.17.

including the next-gen scheduler runner not yet puppetized [4]

Aug 13 2021, 10:32 AM · System administration, Archive coverage, Scheduling utilities

Aug 12 2021

ardumont moved T3471: production: Deploy swh.scheduler v0.17 from Backlog to in-progress on the Sprint 2021 01 board.
Aug 12 2021, 8:41 AM · System administration, Archive coverage, Scheduling utilities
ardumont added a comment to T3471: production: Deploy swh.scheduler v0.17.

Following actions in order:

Aug 12 2021, 8:41 AM · System administration, Archive coverage, Scheduling utilities

Aug 9 2021

ardumont closed T3456: staging: Deploy scheduler v0.17 as Resolved.
Aug 9 2021, 11:07 AM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont closed T3456: staging: Deploy scheduler v0.17, a subtask of T2345: Improve handling of recurrent loading tasks in scheduler, as Resolved.
Aug 9 2021, 11:07 AM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont moved T3456: staging: Deploy scheduler v0.17 from code-review/monitoring to deployed/landed on the System administration board.
Aug 9 2021, 11:07 AM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont triaged T3471: production: Deploy swh.scheduler v0.17 as High priority.
Aug 9 2021, 11:06 AM · System administration, Archive coverage, Scheduling utilities

Aug 6 2021

ardumont moved T3456: staging: Deploy scheduler v0.17 from in-progress to code-review/monitoring on the System administration board.
Aug 6 2021, 3:11 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont updated the task description for T3456: staging: Deploy scheduler v0.17.
Aug 6 2021, 3:11 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a comment to T3456: staging: Deploy scheduler v0.17.

Ensure the journal client is doing its new job

Aug 6 2021, 3:10 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont updated the task description for T3456: staging: Deploy scheduler v0.17.
Aug 6 2021, 3:09 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont updated the task description for T3456: staging: Deploy scheduler v0.17.
Aug 6 2021, 12:12 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont changed the status of T3456: staging: Deploy scheduler v0.17, a subtask of T2345: Improve handling of recurrent loading tasks in scheduler, from Open to Work in Progress.
Aug 6 2021, 12:11 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont changed the status of T3456: staging: Deploy scheduler v0.17 from Open to Work in Progress.
Aug 6 2021, 12:11 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a project to T3456: staging: Deploy scheduler v0.17: System administration.
Aug 6 2021, 12:11 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont updated the task description for T3456: staging: Deploy scheduler v0.17.
Aug 6 2021, 11:10 AM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

Aug 5 2021

ardumont claimed T3456: staging: Deploy scheduler v0.17.
Aug 5 2021, 3:25 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

Aug 4 2021

ardumont triaged T3456: staging: Deploy scheduler v0.17 as High priority.
Aug 4 2021, 10:10 AM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

Aug 3 2021

ardumont added a comment to T2345: Improve handling of recurrent loading tasks in scheduler.

Deactivate failing visits (delegating to listers the act of activating back those
origins which gets live again). I have diffs which deal with this that needs some
rebase and work according to latest change (I need to get back to it) [1].

Aug 3 2021, 8:58 AM · Sprint 2021 01, Archive coverage, Scheduling utilities

Jul 30 2021

ardumont changed the status of T2345: Improve handling of recurrent loading tasks in scheduler from Open to Work in Progress.

(^ for a while ;)

Jul 30 2021, 3:55 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a comment to T2345: Improve handling of recurrent loading tasks in scheduler.

Status on this, after the recent refactoring we did with @olasd to simplify the actual
implementation (backend and journal client). There remains to:

Jul 30 2021, 3:54 PM · Sprint 2021 01, Archive coverage, Scheduling utilities

Jul 9 2021

ardumont added a comment to T2345: Improve handling of recurrent loading tasks in scheduler.

Updated stats in descending order on the no_last_update column:

Jul 9 2021, 3:11 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a comment to T2345: Improve handling of recurrent loading tasks in scheduler.

Relatedly to this task, some work has been started to make the pypi lister list its
origins with the last_update information in the diff D5977 / T3399 (review got done
and the implementation needs to be improved but still ;).

Jul 9 2021, 3:05 PM · Sprint 2021 01, Archive coverage, Scheduling utilities