Page MenuHomeSoftware Heritage
Feed Advanced Search

Jul 30 2021

ardumont added a comment to T2345: Improve handling of recurrent loading tasks in scheduler.

Status on this, after the recent refactoring we did with @olasd to simplify the actual
implementation (backend and journal client). There remains to:

Jul 30 2021, 3:54 PM · Sprint 2021 01, Archive coverage, Scheduling utilities

Jul 9 2021

ardumont added a comment to T2345: Improve handling of recurrent loading tasks in scheduler.

Updated stats in descending order on the no_last_update column:

Jul 9 2021, 3:11 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a comment to T2345: Improve handling of recurrent loading tasks in scheduler.

Relatedly to this task, some work has been started to make the pypi lister list its
origins with the last_update information in the diff D5977 / T3399 (review got done
and the implementation needs to be improved but still ;).

Jul 9 2021, 3:05 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont closed T3399: Improve PyPI lister to pull last update information when running incrementally, a subtask of T2345: Improve handling of recurrent loading tasks in scheduler, as Resolved.
Jul 9 2021, 2:52 PM · Sprint 2021 01, Archive coverage, Scheduling utilities

Jul 8 2021

ardumont added a comment to T2345: Improve handling of recurrent loading tasks in scheduler.

Status on the latest development for this task, "Baseline for the recurrence of origin
visits" chapter has been implemented in the following stacked diffs (in review):

Jul 8 2021, 12:30 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a revision to T2345: Improve handling of recurrent loading tasks in scheduler: D5980: journal_client: Disable origins when too many visited attempts failed.
Jul 8 2021, 11:26 AM · Sprint 2021 01, Archive coverage, Scheduling utilities

Jul 7 2021

ardumont added a revision to T2345: Improve handling of recurrent loading tasks in scheduler: D5978: Add a successive_visits counter to origin visit stats.
Jul 7 2021, 5:26 PM · Sprint 2021 01, Archive coverage, Scheduling utilities

Jul 1 2021

ardumont added a revision to T2345: Improve handling of recurrent loading tasks in scheduler: D5956: Introduce new scheduling policy to grab origins without last update.
Jul 1 2021, 12:34 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a revision to T2345: Improve handling of recurrent loading tasks in scheduler: D5950: journal_client: Compute next position for origin visit.
Jul 1 2021, 10:14 AM · Sprint 2021 01, Archive coverage, Scheduling utilities

Jun 29 2021

ardumont updated the task description for T2967: Write journal client subcribed to origin_visit_status topics .
Jun 29 2021, 2:31 PM · Sprint 2021 01, Scheduling utilities

Jun 23 2021

ardumont added a revision to T2345: Improve handling of recurrent loading tasks in scheduler: D5919: Start handling of recurrent loading tasks in scheduler.
Jun 23 2021, 6:11 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a revision to T2345: Improve handling of recurrent loading tasks in scheduler: D5914: backend: Auto-generate origin visit stats upsert query.
Jun 23 2021, 3:32 PM · Sprint 2021 01, Archive coverage, Scheduling utilities

Jun 21 2021

ardumont updated the task description for T2345: Improve handling of recurrent loading tasks in scheduler.
Jun 21 2021, 5:50 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
olasd added a revision to T2444: Implement the scheduling policy for the recurrent visit scheduler: D5809: Direct scheduling of origin visits in celery.
Jun 21 2021, 4:04 PM · Sprint 2021 01, Scheduling utilities
olasd added a comment to T2345: Improve handling of recurrent loading tasks in scheduler.

Summary of the data available in the listed_origins table, broken down by lister and "known state" of origins:

Jun 21 2021, 2:27 PM · Sprint 2021 01, Archive coverage, Scheduling utilities

Mar 4 2021

ardumont abandoned D4927: lister.docs: add a lister template for the new API.

This got landed through another diff [1] already.

Mar 4 2021, 10:51 AM · Sprint 2021 01, Lister
ardumont commandeered D4927: lister.docs: add a lister template for the new API.
Mar 4 2021, 10:50 AM · Sprint 2021 01, Lister

Feb 26 2021

ardumont closed T3073: Properly document the new unified API lister, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, as Resolved.
Feb 26 2021, 5:33 PM · Sprint 2021 01, Scheduling utilities
ardumont closed T3073: Properly document the new unified API lister as Resolved.
Feb 26 2021, 5:33 PM · Sprint 2021 01, Scheduling utilities
ardumont added a revision to T3073: Properly document the new unified API lister: D5069: docs: Add new "howto write a lister tutorial" with unified lister api.
Feb 26 2021, 3:57 PM · Sprint 2021 01, Scheduling utilities
ardumont triaged T3073: Properly document the new unified API lister as Normal priority.
Feb 26 2021, 3:56 PM · Sprint 2021 01, Scheduling utilities

Feb 9 2021

swh-public-ci added a comment to D4927: lister.docs: add a lister template for the new API.

Build is green

Feb 9 2021, 11:34 AM · Sprint 2021 01, Lister
tenma updated the diff for D4927: lister.docs: add a lister template for the new API.

Small docstring fix

Feb 9 2021, 11:32 AM · Sprint 2021 01, Lister

Feb 8 2021

olasd added a comment to T2345: Improve handling of recurrent loading tasks in scheduler.

Here's my understanding of the status of the migration to the next generation scheduler as of today:

Feb 8 2021, 12:01 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
vlorentz reassigned T2973: Implement a scheduler simulator from vlorentz to olasd.
Feb 8 2021, 12:00 PM · Sprint 2021 01, Archive coverage, Scheduling utilities

Feb 5 2021

tenma added inline comments to D4927: lister.docs: add a lister template for the new API.
Feb 5 2021, 10:32 AM · Sprint 2021 01, Lister

Feb 2 2021

anlambert closed T2442: Provide a unified API for listers to interact with the scheduler, a subtask of T2345: Improve handling of recurrent loading tasks in scheduler, as Resolved.
Feb 2 2021, 4:08 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
anlambert closed T2442: Provide a unified API for listers to interact with the scheduler, a subtask of T2444: Implement the scheduling policy for the recurrent visit scheduler, as Resolved.
Feb 2 2021, 4:08 PM · Sprint 2021 01, Scheduling utilities
anlambert closed T2442: Provide a unified API for listers to interact with the scheduler as Resolved by committing rDLS89335445210f: Remove no longer used legacy Lister API and update CLI options.
Feb 2 2021, 4:08 PM · Sprint 2021 01, Scheduling utilities
anlambert closed T2991: Port packagist lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, as Resolved.
Feb 2 2021, 4:08 PM · Sprint 2021 01, Scheduling utilities
anlambert added a revision to T2442: Provide a unified API for listers to interact with the scheduler: D4992: Remove no longer used legacy Lister API and update CLI options.
Feb 2 2021, 12:45 PM · Sprint 2021 01, Scheduling utilities

Feb 1 2021

anlambert changed the status of T2991: Port packagist lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, from Open to Work in Progress.
Feb 1 2021, 1:37 PM · Sprint 2021 01, Scheduling utilities
tenma updated the summary of D4927: lister.docs: add a lister template for the new API.
Feb 1 2021, 9:59 AM · Sprint 2021 01, Lister

Jan 29 2021

anlambert closed T2990: Port GNU lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, as Resolved.
Jan 29 2021, 3:10 PM · Sprint 2021 01, Scheduling utilities
ardumont closed T2993: Deploy visit-stats journal client on production, a subtask of T2444: Implement the scheduling policy for the recurrent visit scheduler, as Resolved.
Jan 29 2021, 12:05 PM · Sprint 2021 01, Scheduling utilities

Jan 28 2021

anlambert closed T2992: Port launchpad lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, as Resolved.
Jan 28 2021, 3:28 PM · Sprint 2021 01, Scheduling utilities

Jan 27 2021

ardumont moved T2979: Port debian lister to the new Lister API from code review to done on the Sprint 2021 01 board.
Jan 27 2021, 4:25 PM · Lister, Sprint 2021 01
anlambert changed the status of T2992: Port launchpad lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, from Open to Work in Progress.
Jan 27 2021, 4:05 PM · Sprint 2021 01, Scheduling utilities
ardumont accepted D4927: lister.docs: add a lister template for the new API.
Jan 27 2021, 10:06 AM · Sprint 2021 01, Lister
ardumont added a comment to D4927: lister.docs: add a lister template for the new API.

Looks good to me.

Jan 27 2021, 10:06 AM · Sprint 2021 01, Lister

Jan 26 2021

ardumont closed T2987: Port gitlab lister to the new `swh.lister.pattern.Lister` API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, as Resolved.
Jan 26 2021, 7:06 PM · Sprint 2021 01, Scheduling utilities
anlambert closed T2979: Port debian lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, as Resolved.
Jan 26 2021, 5:29 PM · Sprint 2021 01, Scheduling utilities
anlambert closed T2979: Port debian lister to the new Lister API as Resolved by committing rDLSbb0184c004d2: debian: Reimplement lister using new Lister API.
Jan 26 2021, 5:29 PM · Lister, Sprint 2021 01
anlambert closed T2989: Port CRAN lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, as Resolved.
Jan 26 2021, 3:24 PM · Sprint 2021 01, Scheduling utilities
swh-public-ci added a comment to D4927: lister.docs: add a lister template for the new API.

Build is green

Jan 26 2021, 3:12 PM · Sprint 2021 01, Lister
tenma updated the diff for D4927: lister.docs: add a lister template for the new API.

Add implementation instructions to the template

Jan 26 2021, 3:08 PM · Sprint 2021 01, Lister
ardumont changed the status of T2993: Deploy visit-stats journal client on production, a subtask of T2444: Implement the scheduling policy for the recurrent visit scheduler, from Open to Work in Progress.
Jan 26 2021, 12:55 PM · Sprint 2021 01, Scheduling utilities
ardumont closed T2984: Port cgit lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, as Resolved.
Jan 26 2021, 9:57 AM · Sprint 2021 01, Scheduling utilities
ardumont closed T2984: Port cgit lister to the new Lister API as Resolved.
Jan 26 2021, 9:57 AM · Lister, CGit lister, Sprint 2021 01

Jan 25 2021

vsellier added a revision to T2984: Port cgit lister to the new Lister API: D4943: cgit lister: Add missing types on the init method.
Jan 25 2021, 6:33 PM · Lister, CGit lister, Sprint 2021 01
tenma added inline comments to D4926: Port cgit lister to the new lister api.
Jan 25 2021, 5:02 PM · Sprint 2021 01, Lister
tenma added inline comments to D4926: Port cgit lister to the new lister api.
Jan 25 2021, 4:58 PM · Sprint 2021 01, Lister
tenma closed D4907: Reimplement Gitea lister using new Lister API.
Jan 25 2021, 4:09 PM · Sprint 2021 01, Lister
tenma closed T2971: Port Gitea lister to the new Lister API as Resolved.
Jan 25 2021, 4:09 PM · Sprint 2021 01, Lister
tenma updated the test plan for D4907: Reimplement Gitea lister using new Lister API.
Jan 25 2021, 3:46 PM · Sprint 2021 01, Lister
anlambert accepted D4907: Reimplement Gitea lister using new Lister API.

Looks good to me !

Jan 25 2021, 3:41 PM · Sprint 2021 01, Lister
swh-public-ci added a comment to D4907: Reimplement Gitea lister using new Lister API.

Build is green

Jan 25 2021, 3:20 PM · Sprint 2021 01, Lister
tenma updated the diff for D4907: Reimplement Gitea lister using new Lister API.

improve handling of credentials

Jan 25 2021, 3:17 PM · Sprint 2021 01, Lister
tenma added inline comments to D4907: Reimplement Gitea lister using new Lister API.
Jan 25 2021, 3:14 PM · Sprint 2021 01, Lister
vsellier closed D4926: Port cgit lister to the new lister api.
Jan 25 2021, 3:06 PM · Sprint 2021 01, Lister
anlambert accepted D4926: Port cgit lister to the new lister api.

Looks good to me !

Jan 25 2021, 3:06 PM · Sprint 2021 01, Lister
tenma retitled D4907: Reimplement Gitea lister using new Lister API from WIP Reimplement Gitea lister using new Lister API to Reimplement Gitea lister using new Lister API.
Jan 25 2021, 3:03 PM · Sprint 2021 01, Lister
swh-public-ci added a comment to D4926: Port cgit lister to the new lister api.

Build is green

Jan 25 2021, 3:01 PM · Sprint 2021 01, Lister
ardumont added inline comments to D4926: Port cgit lister to the new lister api.
Jan 25 2021, 3:00 PM · Sprint 2021 01, Lister
vsellier updated the diff for D4926: Port cgit lister to the new lister api.
  • rebase
  • update tests according to the review feedbacks
Jan 25 2021, 2:58 PM · Sprint 2021 01, Lister
anlambert changed the status of T2989: Port CRAN lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, from Open to Work in Progress.
Jan 25 2021, 2:57 PM · Sprint 2021 01, Scheduling utilities
ardumont changed the status of T2987: Port gitlab lister to the new `swh.lister.pattern.Lister` API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, from Open to Work in Progress.
Jan 25 2021, 2:19 PM · Sprint 2021 01, Scheduling utilities
ardumont added a parent task for T2984: Port cgit lister to the new Lister API: T2442: Provide a unified API for listers to interact with the scheduler.
Jan 25 2021, 2:19 PM · Lister, CGit lister, Sprint 2021 01
ardumont added a subtask for T2442: Provide a unified API for listers to interact with the scheduler: T2984: Port cgit lister to the new Lister API.
Jan 25 2021, 2:19 PM · Sprint 2021 01, Scheduling utilities
anlambert added a subtask for T2442: Provide a unified API for listers to interact with the scheduler: T2987: Port gitlab lister to the new `swh.lister.pattern.Lister` API.
Jan 25 2021, 2:18 PM · Sprint 2021 01, Scheduling utilities
anlambert added inline comments to D4907: Reimplement Gitea lister using new Lister API.
Jan 25 2021, 2:08 PM · Sprint 2021 01, Lister
anlambert requested changes to D4907: Reimplement Gitea lister using new Lister API.
Jan 25 2021, 2:00 PM · Sprint 2021 01, Lister
vsellier added inline comments to D4926: Port cgit lister to the new lister api.
Jan 25 2021, 12:17 PM · Sprint 2021 01, Lister
tenma requested review of D4907: Reimplement Gitea lister using new Lister API.
Jan 25 2021, 12:04 PM · Sprint 2021 01, Lister
anlambert added inline comments to D4926: Port cgit lister to the new lister api.
Jan 25 2021, 11:57 AM · Sprint 2021 01, Lister
anlambert added inline comments to D4926: Port cgit lister to the new lister api.
Jan 25 2021, 11:54 AM · Sprint 2021 01, Lister
anlambert requested changes to D4926: Port cgit lister to the new lister api.
Jan 25 2021, 11:51 AM · Sprint 2021 01, Lister
vsellier updated the task description for T2967: Write journal client subcribed to origin_visit_status topics .
Jan 25 2021, 10:56 AM · Sprint 2021 01, Scheduling utilities
ardumont added a comment to T2443: Implement a bulk-queryable cache of latest visits for use by the recurrent visit scheduler.

Done (see subtasks for details)

Jan 25 2021, 8:43 AM · Sprint 2021 01, Scheduling utilities
ardumont closed T2443: Implement a bulk-queryable cache of latest visits for use by the recurrent visit scheduler, a subtask of T2444: Implement the scheduling policy for the recurrent visit scheduler, as Resolved.
Jan 25 2021, 8:42 AM · Sprint 2021 01, Scheduling utilities
ardumont closed T2443: Implement a bulk-queryable cache of latest visits for use by the recurrent visit scheduler, a subtask of T2345: Improve handling of recurrent loading tasks in scheduler, as Resolved.
Jan 25 2021, 8:42 AM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont closed T2443: Implement a bulk-queryable cache of latest visits for use by the recurrent visit scheduler as Resolved.
Jan 25 2021, 8:42 AM · Sprint 2021 01, Scheduling utilities
ardumont moved T2443: Implement a bulk-queryable cache of latest visits for use by the recurrent visit scheduler from in-progress to code review on the Sprint 2021 01 board.
Jan 25 2021, 8:42 AM · Sprint 2021 01, Scheduling utilities

Jan 22 2021

ardumont moved T2972: Port npm lister to the new Lister API from code review to done on the Sprint 2021 01 board.
Jan 22 2021, 5:13 PM · Npm Lister, Lister, Sprint 2021 01
tenma planned changes to D4927: lister.docs: add a lister template for the new API.
Jan 22 2021, 4:27 PM · Sprint 2021 01, Lister
vsellier requested review of D4926: Port cgit lister to the new lister api.
Jan 22 2021, 4:24 PM · Sprint 2021 01, Lister
douardda added a comment to D4927: lister.docs: add a lister template for the new API.

thanks. I think however, given its purpose, this example code should be heavily commented: each constant (eg. MyPageType) and each method should be commented (not docstrings but comments exaplaining what the method/variable is used for).

Jan 22 2021, 4:15 PM · Sprint 2021 01, Lister
tenma moved T2979: Port debian lister to the new Lister API from in-progress to code review on the Sprint 2021 01 board.
Jan 22 2021, 3:59 PM · Lister, Sprint 2021 01
tenma moved T2971: Port Gitea lister to the new Lister API from in-progress to code review on the Sprint 2021 01 board.
Jan 22 2021, 3:59 PM · Sprint 2021 01, Lister
tenma closed T2956: Port PyPI lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, as Resolved.
Jan 22 2021, 3:59 PM · Sprint 2021 01, Scheduling utilities
tenma closed T2956: Port PyPI lister to the new Lister API as Resolved.
Jan 22 2021, 3:59 PM · Sprint 2021 01, Lister
tenma requested review of D4927: lister.docs: add a lister template for the new API.
Jan 22 2021, 3:59 PM · Sprint 2021 01, Lister
ardumont moved T2984: Port cgit lister to the new Lister API from in-progress to code review on the Sprint 2021 01 board.
Jan 22 2021, 3:53 PM · Lister, CGit lister, Sprint 2021 01
vsellier added a revision to T2984: Port cgit lister to the new Lister API: D4926: Port cgit lister to the new lister api.
Jan 22 2021, 3:53 PM · Lister, CGit lister, Sprint 2021 01
anlambert added a revision to T2979: Port debian lister to the new Lister API: D4925: debian: Reimplement lister using new Lister API.
Jan 22 2021, 12:18 PM · Lister, Sprint 2021 01
anlambert added a revision to T2979: Port debian lister to the new Lister API: D4924: model/ListedOrigin: Set extra_loader_arguments type to Dict[str, Any].
Jan 22 2021, 11:30 AM · Lister, Sprint 2021 01
vsellier closed T2978: Deploy visit-stats journal client on staging as Resolved.
Jan 22 2021, 11:12 AM · Sprint 2021 01, Scheduling utilities
vsellier closed T2978: Deploy visit-stats journal client on staging, a subtask of T2967: Write journal client subcribed to origin_visit_status topics , as Resolved.
Jan 22 2021, 11:12 AM · Sprint 2021 01, Scheduling utilities
vsellier updated the task description for T2978: Deploy visit-stats journal client on staging.
Jan 22 2021, 11:12 AM · Sprint 2021 01, Scheduling utilities
anlambert closed T2972: Port npm lister to the new Lister API as Resolved by committing rDLSff232f0d9180: npm: Reimplement lister using new Lister API.
Jan 22 2021, 11:11 AM · Npm Lister, Lister, Sprint 2021 01