Page MenuHomeSoftware Heritage
Create Task
Maniphest T359

Indexers: batch content analyzer infrastructure
Closed, MigratedEdits Locked
Actions

Description

We want to be able to analyze, in batch, all the content blobs stored by Software Heritage.

Sample use cases are:

  • compute mime type (service running)
  • detect the license using ninka/fossology (service running)
  • detect the programming language (service stopped)

To this end we need some scheduling tooling that allows to add/remove analyzer, (re)run analysis in batch, incrementally stay up to date with new incoming content blobs.

Related Objects
Search...

Event Timeline

zack created this task.Apr 1 2016, 10:51 AM
zack updated the task description. (Show Details)
olasd changed the visibility from "All Users" to "Public (No Login Required)".May 13 2016, 5:09 PM
zack added a subtask: T424: swh-journal: persistent journal infrastructure to record additions to the swh-storage.May 29 2016, 5:57 PM
zack added a parent task: T439: Indexers: compute (and maintain up-to-date) the filetype of all blobs.Jun 13 2016, 4:06 PM
olasd edited subtasks, added: T528: swh-journal: Create a journal client listing objects of a given type; removed: T424: swh-journal: persistent journal infrastructure to record additions to the swh-storage.Aug 16 2016, 6:35 PM
ardumont added a subscriber: ardumont.Oct 8 2016, 9:42 AM

A POC is ongoing for as T548

ardumont renamed this task from batch blob analyzer infrastructure to Indexers: batch blob analyzer infrastructure.Oct 5 2018, 2:47 PM
ardumont removed a subtask: T528: swh-journal: Create a journal client listing objects of a given type.Oct 18 2018, 3:55 PM
ardumont added a comment.Oct 18 2018, 3:58 PM

Unplugging T528 as per discussion.

We need to rework the current indexer implementation to use range instead (T991).
After that, we can schedule 256 ranges of contents to index using the scheduler stack instead.
And see where that goes.

ardumont renamed this task from Indexers: batch blob analyzer infrastructure to Indexers: batch content analyzer infrastructure.Oct 19 2018, 8:44 AM
ardumont raised the priority of this task from Low to Normal.
ardumont added a subtask: T991: Indexers: Send range of ids instead of list of ids.
ardumont updated the task description. (Show Details)
ardumont added a project: Indexer.
ardumont edited subtasks, added: T1227: General improvments of the indexer: Schedule indexer tasks; removed: T991: Indexers: Send range of ids instead of list of ids.Oct 19 2018, 8:47 AM
vlorentz added a parent task: T1385: Monitor output of metadata indexers.Nov 27 2018, 11:58 AM
ardumont closed subtask T1227: General improvments of the indexer: Schedule indexer tasks as Resolved.Dec 4 2018, 11:47 AM
ardumont closed this task as Resolved.Jan 15 2019, 2:44 PM
ardumont claimed this task.

We need to rework the current indexer implementation to use range instead (T991).
After that, we can schedule 256 ranges of contents to index using the scheduler stack instead.
And see where that goes.

Done.

So in effect:

To this end we need some scheduling tooling that allows to add/remove analyzer, (re)run analysis in batch, incrementally stay up to date with new incoming content blobs.

Done.

gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 4:18 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T1227: General improvments of the indexer: Schedule indexer tasks from Resolved to Migrated.Jan 8 2023, 9:58 PM
About · Developer information · Legal