Page MenuHomeSoftware Heritage
Create Task
Maniphest T2443

Implement a bulk-queryable cache of latest visits for use by the recurrent visit scheduler
Closed, MigratedEdits Locked
Actions

Description

To implement its scheduling policy, the recurrent visit scheduler needs to be able, fairly quickly, to get a list of origins and their latest visits present in the archive.

We also need to store more information, e.g. how long ago the last eventful visit was, whether the last run failed (and if so, for how long tasks have been failing), etc.

This cache needs two components: a database and associated API, and a journal client we can use to update it.

pad: https://hedgedoc.softwareheritage.org/DURdfOGyQomXEwbyFaS4gg?edit

Revisions and Commits

rDSCH Scheduling utilities
D4838rDSCHa62003397d6e Add an new origin visit info model object and related backend api
rDSTO Storage manager
ClosedD4849 Adapt cassandra storage to ignore the new OriginVisitStatus.type field
rDMOD Data model
D4848rDMOD1ca92a5ce003 Add an optional type field on OriginVisitStatus object

Related Objects
Search...

Event Timeline

olasd triaged this task as High priority.Jun 9 2020, 4:51 PM
olasd created this task.
olasd added a parent task: T2444: Implement the scheduling policy for the recurrent visit scheduler.Jun 9 2020, 5:10 PM
ardumont added a project: Sprint 2021 01.Jan 11 2021, 12:37 PM
ardumont changed the task status from Open to Work in Progress.Jan 11 2021, 2:03 PM
ardumont moved this task from Backlog to in-progress on the Sprint 2021 01 board.
vlorentz reassigned this task from olasd to douardda.Jan 11 2021, 2:16 PM
douardda mentioned this in T2961: Add a "not_found" value in OriginVisitStatus.status.Jan 12 2021, 11:24 AM
vsellier added a revision: D4838: Add an new origin visit info model object and related backend api.Jan 12 2021, 12:16 PM
vsellier added a revision: D4848: Add an optional type field on OriginVisitStatus object.Jan 12 2021, 6:11 PM
vsellier added a revision: D4849: Adapt cassandra storage to ignore the new OriginVisitStatus.type field.Jan 12 2021, 6:29 PM
ardumont updated the task description. (Show Details)Jan 12 2021, 6:30 PM
ardumont removed douardda as the assignee of this task.Jan 13 2021, 9:39 AM
ardumont added a subscriber: douardda.
ardumont assigned this task to douardda.Jan 13 2021, 9:52 AM
ardumont changed the status of subtask T2963: Add visit_type field to OriginVisitStatus model object from Open to Work in Progress.
ardumont changed the status of subtask T2965: Adapt storage to actually write the visit_type in the origin_visit_status topic from Open to Work in Progress.Jan 13 2021, 11:36 AM
ardumont changed the status of subtask T2967: Write journal client subcribed to origin_visit_status topics from Open to Work in Progress.Jan 13 2021, 11:38 AM
douardda changed the status of subtask T2964: Adapt origin_visit_status_(get|add) api to deal with the visit_type from Open to Work in Progress.Jan 13 2021, 11:39 AM
vsellier added a commit: rDSCHa62003397d6e: Add an new origin visit info model object and related backend api.Jan 13 2021, 11:49 AM
vsellier closed subtask T2965: Adapt storage to actually write the visit_type in the origin_visit_status topic as Resolved.Jan 13 2021, 4:51 PM
vsellier mentioned this in rDSTO0b44b3725497: Adapt cassandra storage to ignore the new OriginVisitStatus.type field.Jan 14 2021, 11:00 AM
vsellier added a commit: rDMOD1ca92a5ce003: Add an optional type field on OriginVisitStatus object.Jan 14 2021, 2:12 PM
ardumont closed subtask T2963: Add visit_type field to OriginVisitStatus model object as Resolved.Jan 14 2021, 2:54 PM
vsellier closed subtask T2964: Adapt origin_visit_status_(get|add) api to deal with the visit_type as Resolved.Jan 15 2021, 2:01 PM
vsellier changed the status of subtask T2966: Backfill origin_visit_status **with** the `visit_type` field properly given from Open to Work in Progress.Jan 15 2021, 2:40 PM
vsellier closed subtask T2966: Backfill origin_visit_status **with** the `visit_type` field properly given as Resolved.Jan 18 2021, 12:02 PM
ardumont closed subtask T2967: Write journal client subcribed to origin_visit_status topics as Resolved.Jan 21 2021, 11:43 AM
ardumont closed this task as Resolved.Jan 25 2021, 8:42 AM
ardumont moved this task from in-progress to code review on the Sprint 2021 01 board.
ardumont moved this task from code review to done on the Sprint 2021 01 board.
ardumont added a subscriber: ardumont.

Done (see subtasks for details)

gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 4:30 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T2963: Add visit_type field to OriginVisitStatus model object from Resolved to Migrated.Jan 8 2023, 4:33 PM
gitlab-migration changed the status of subtask T2964: Adapt origin_visit_status_(get|add) api to deal with the visit_type from Resolved to Migrated.
gitlab-migration changed the status of subtask T2965: Adapt storage to actually write the visit_type in the origin_visit_status topic from Resolved to Migrated.
gitlab-migration changed the status of subtask T2967: Write journal client subcribed to origin_visit_status topics from Resolved to Migrated.
gitlab-migration changed the status of subtask T2966: Backfill origin_visit_status **with** the `visit_type` field properly given from Resolved to Migrated.Jan 8 2023, 10:01 PM
About · Developer information · Legal