Page MenuHomeSoftware Heritage

Scheduling utilitiesFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Recent Activity

Today

vsellier updated the task description for T2978: Deploy visit-stats journal client on staging.
Wed, Jan 20, 7:08 PM · Sprint 2021 01, Scheduling utilities
vsellier added a comment to T2978: Deploy visit-stats journal client on staging.

Backfill launched from storage1 with this script : P927
(10 ranges in //)

Wed, Jan 20, 6:46 PM · Sprint 2021 01, Scheduling utilities
vsellier updated the task description for T2978: Deploy visit-stats journal client on staging.
Wed, Jan 20, 6:44 PM · Sprint 2021 01, Scheduling utilities
vlorentz added a revision to T2444: Implement the scheduling policy for the recurrent visit scheduler: D4899: Add scheduling policy for already visited origins with known last update.
Wed, Jan 20, 5:46 PM · Sprint 2021 01, Scheduling utilities
vlorentz added a revision to T2444: Implement the scheduling policy for the recurrent visit scheduler: D4898: Add scheduling policy for never visited origins.
Wed, Jan 20, 5:46 PM · Sprint 2021 01, Scheduling utilities
vsellier updated the task description for T2978: Deploy visit-stats journal client on staging.
Wed, Jan 20, 5:34 PM · Sprint 2021 01, Scheduling utilities
vlorentz added a comment to T2974: Define (and implement) scheduler performance metrics.
  • "'outdatedest' origin": excluding disabled origins and origins visited after their last_activity (if any), the min(current_time - last_visit) (lower is better)
Wed, Jan 20, 5:33 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
vsellier added a comment to T2978: Deploy visit-stats journal client on staging.

All staging worker stopped:

root@pergamon:~# sudo clush -b -w @staging-workers 'puppet agent --disable "Deploy new storage version"; cd /etc/systemd/system/multi-user.target.wants; for unit in swh-worker@*; do systemctl disable $unit; done; systemctl stop swh-worker@*'
Wed, Jan 20, 5:32 PM · Sprint 2021 01, Scheduling utilities
vsellier updated the task description for T2978: Deploy visit-stats journal client on staging.
Wed, Jan 20, 5:23 PM · Sprint 2021 01, Scheduling utilities
vsellier updated the task description for T2978: Deploy visit-stats journal client on staging.
Wed, Jan 20, 5:06 PM · Sprint 2021 01, Scheduling utilities
vsellier updated the task description for T2978: Deploy visit-stats journal client on staging.
Wed, Jan 20, 4:37 PM · Sprint 2021 01, Scheduling utilities
vsellier updated the task description for T2978: Deploy visit-stats journal client on staging.
Wed, Jan 20, 3:49 PM · Sprint 2021 01, Scheduling utilities
vsellier updated the task description for T2978: Deploy visit-stats journal client on staging.
Wed, Jan 20, 3:49 PM · Sprint 2021 01, Scheduling utilities
tenma closed T2955: Port Bitbucket lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, as Resolved.
Wed, Jan 20, 3:47 PM · Sprint 2021 01, Scheduling utilities
ardumont moved T2978: Deploy visit-stats journal client on staging from in-progress to code review on the Sprint 2021 01 board.
Wed, Jan 20, 9:59 AM · Sprint 2021 01, Scheduling utilities

Yesterday

ardumont added a revision to T2967: Write journal client subcribed to origin_visit_status topics : D4890: journal_client: Read visit_stats entries by batch out of the loop.
Tue, Jan 19, 6:52 PM · Sprint 2021 01, Scheduling utilities
ardumont added a revision to T2967: Write journal client subcribed to origin_visit_status topics : D4888: scheduler: Make origin_visit_stats_get read multiple entries.
Tue, Jan 19, 6:32 PM · Sprint 2021 01, Scheduling utilities
vsellier added a revision to T2978: Deploy visit-stats journal client on staging: D4884: Deploy the scheduler's journal client.
Tue, Jan 19, 4:08 PM · Sprint 2021 01, Scheduling utilities
vsellier added a revision to T2978: Deploy visit-stats journal client on staging: D4882: scheduler: Reorganize scheduler configuration files.
Tue, Jan 19, 3:15 PM · Sprint 2021 01, Scheduling utilities
douardda added a revision to T2444: Implement the scheduling policy for the recurrent visit scheduler: D4881: Move the `last_scheduled` ts from ListedOrigin to OriginVisitStatus.
Tue, Jan 19, 2:49 PM · Sprint 2021 01, Scheduling utilities
anlambert changed the status of T2979: Port debian lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, from Open to Work in Progress.
Tue, Jan 19, 2:29 PM · Sprint 2021 01, Scheduling utilities
vsellier changed the status of T2978: Deploy visit-stats journal client on staging, a subtask of T2967: Write journal client subcribed to origin_visit_status topics , from Open to Work in Progress.
Tue, Jan 19, 1:48 PM · Sprint 2021 01, Scheduling utilities
vsellier changed the status of T2978: Deploy visit-stats journal client on staging from Open to Work in Progress.
Tue, Jan 19, 1:48 PM · Sprint 2021 01, Scheduling utilities
vsellier triaged T2978: Deploy visit-stats journal client on staging as Normal priority.
Tue, Jan 19, 1:48 PM · Sprint 2021 01, Scheduling utilities
vsellier updated the task description for T2967: Write journal client subcribed to origin_visit_status topics .
Tue, Jan 19, 1:45 PM · Sprint 2021 01, Scheduling utilities
vlorentz triaged T2977: Add type annotations to swh.scheduler.interface as Low priority.
Tue, Jan 19, 11:31 AM · Easy hack, Scheduling utilities

Mon, Jan 18

douardda added a comment to T2974: Define (and implement) scheduler performance metrics.

thanks, looks a good starting point.

Mon, Jan 18, 4:36 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a revision to T2967: Write journal client subcribed to origin_visit_status topics : D4876: scheduler.cli.journal: Add `swh scheduler journal visit-stats` cli.
Mon, Jan 18, 3:36 PM · Sprint 2021 01, Scheduling utilities
olasd added a comment to T2974: Define (and implement) scheduler performance metrics.
  • "origins with pending changes": Number of origins where last_visit < last_activity (lower is better)
Mon, Jan 18, 2:29 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
olasd added a comment to T2974: Define (and implement) scheduler performance metrics.

Some potentially interesting and "easy" metrics:

Mon, Jan 18, 2:27 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
olasd moved T2974: Define (and implement) scheduler performance metrics from Backlog to in-progress on the Sprint 2021 01 board.
Mon, Jan 18, 2:17 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
olasd changed the status of T2974: Define (and implement) scheduler performance metrics from Open to Work in Progress.
Mon, Jan 18, 2:17 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
olasd added revisions to T2444: Implement the scheduling policy for the recurrent visit scheduler: D4844: Introduce a `swh scheduler origin grab-next` cli, D4846: Introduce a `swh scheduler origin schedule-next` cli.
Mon, Jan 18, 2:14 PM · Sprint 2021 01, Scheduling utilities
olasd added a revision to T2973: Implement a scheduler simulator: D4856: Introduce scaffolding for a scheduler simulator.
Mon, Jan 18, 2:13 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
olasd changed the status of T2973: Implement a scheduler simulator, a subtask of T2345: Improve handling of recurrent loading tasks in scheduler, from Open to Work in Progress.
Mon, Jan 18, 2:12 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
olasd changed the status of T2973: Implement a scheduler simulator from Open to Work in Progress.
Mon, Jan 18, 2:12 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
olasd triaged T2973: Implement a scheduler simulator as High priority.
Mon, Jan 18, 2:12 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
vlorentz changed the status of T2444: Implement the scheduling policy for the recurrent visit scheduler, a subtask of T2345: Improve handling of recurrent loading tasks in scheduler, from Open to Work in Progress.
Mon, Jan 18, 2:08 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
vlorentz changed the status of T2444: Implement the scheduling policy for the recurrent visit scheduler from Open to Work in Progress.
Mon, Jan 18, 2:08 PM · Sprint 2021 01, Scheduling utilities
vlorentz moved T2444: Implement the scheduling policy for the recurrent visit scheduler from Backlog to todo on the Sprint 2021 01 board.
Mon, Jan 18, 2:08 PM · Sprint 2021 01, Scheduling utilities
vsellier closed T2966: Backfill origin_visit_status **with** the `visit_type` field properly given, a subtask of T2443: Implement a bulk-queryable cache of latest visits for use by the recurrent visit scheduler, as Resolved.
Mon, Jan 18, 12:02 PM · Sprint 2021 01, Scheduling utilities
vsellier closed T2966: Backfill origin_visit_status **with** the `visit_type` field properly given as Resolved.
Mon, Jan 18, 12:02 PM · Storage manager, Sprint 2021 01, Scheduling utilities

Fri, Jan 15

ardumont updated the task description for T2967: Write journal client subcribed to origin_visit_status topics .
Fri, Jan 15, 5:30 PM · Sprint 2021 01, Scheduling utilities
ardumont added a revision to T2967: Write journal client subcribed to origin_visit_status topics : D4873: journal_client: Improve stats detection.
Fri, Jan 15, 4:57 PM · Sprint 2021 01, Scheduling utilities
anlambert changed the status of T2972: Port npm lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, from Open to Work in Progress.
Fri, Jan 15, 4:23 PM · Sprint 2021 01, Scheduling utilities
ardumont moved T2966: Backfill origin_visit_status **with** the `visit_type` field properly given from in-progress to code review on the Sprint 2021 01 board.
Fri, Jan 15, 2:55 PM · Storage manager, Sprint 2021 01, Scheduling utilities
vsellier added a revision to T2966: Backfill origin_visit_status **with** the `visit_type` field properly given: D4871: Backfiller: Add type to the origin_visit_status topic.
Fri, Jan 15, 2:41 PM · Storage manager, Sprint 2021 01, Scheduling utilities
vsellier changed the status of T2966: Backfill origin_visit_status **with** the `visit_type` field properly given, a subtask of T2443: Implement a bulk-queryable cache of latest visits for use by the recurrent visit scheduler, from Open to Work in Progress.
Fri, Jan 15, 2:40 PM · Sprint 2021 01, Scheduling utilities
vsellier changed the status of T2966: Backfill origin_visit_status **with** the `visit_type` field properly given from Open to Work in Progress.
Fri, Jan 15, 2:40 PM · Storage manager, Sprint 2021 01, Scheduling utilities
vsellier closed T2964: Adapt origin_visit_status_(get|add) api to deal with the visit_type, a subtask of T2443: Implement a bulk-queryable cache of latest visits for use by the recurrent visit scheduler, as Resolved.
Fri, Jan 15, 2:01 PM · Sprint 2021 01, Scheduling utilities