Page MenuHomeSoftware Heritage
Create Task
Maniphest T1385

Monitor output of metadata indexers
Closed, MigratedEdits Locked
Actions

Revisions and Commits

rDCIDX Metadata indexer
ClosedD951 Fix normalization of npm's 'repository' and 'bugs' fields.
ClosedD949 Better error message on content retrieval error + don't write translated_metadata=None to the idx storage.
ClosedD950 Prevent license normalization from crashing on malformed input.
ClosedD953 Prevent repository normalization from crashing on malformed input.

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT1385 Monitor output of metadata indexers
 Migratedgitlab-migrationT359 Indexers: batch content analyzer infrastructure
 Migratedgitlab-migrationT1227 General improvments of the indexer: Schedule indexer tasks
 Migratedgitlab-migrationT1229 Indexers: Make orchestrators use swh-scheduler for scheduling
 Migratedgitlab-migrationT1290 Indexers: Use swh.scheduler instead of directly relying on Celery
 Migratedgitlab-migrationT1230 Indexers: Improve readme to be more explicit on how to run locally
 Migratedgitlab-migrationT1310 Simplify indexer design: move away from the pipeline approach
 Migratedgitlab-migrationT1311 indexer: Remove orchestrators
 Migratedgitlab-migrationT1312 indexer: Adapt textual content indexer to actually filter textual content themselves
 Migratedgitlab-migrationT1324 Deploy metadata indexers in production
 Migratedgitlab-migrationT1326 metadata indexer: Deploy origin head
 Migratedgitlab-migrationT991 Indexers: Send range of ids instead of list of ids
 Migratedgitlab-migrationT1375 Deploy revision metadata indexer
 Migratedgitlab-migrationT1376 Deploy origin indexer
 Migratedgitlab-migrationT1374 content indexer: Determine the identifier ranges to use to schedule those
 Migratedgitlab-migrationT818 indexer DB should not use bytea for mimetype and encoding columns
 Migratedgitlab-migrationT1466 Deploy swh-indexer-journal-client
 Migratedgitlab-migrationT1455 Add a journal client that schedules oneshot tasks for metadata indexers
 Migratedgitlab-migrationT1456 Make metadata indexers support ranges
 Migratedgitlab-migrationT1458 docker containers for metadata indexers
 Migratedgitlab-migrationT1465 Journal notification of full origin visits
 Migratedgitlab-migrationT1277 swh-journal: Create a journal client for listing origin visits
 Migratedgitlab-migrationT1468 journal: Deploy the publisher

Event Timeline

vlorentz triaged this task as High priority.Nov 27 2018, 11:56 AM
vlorentz created this task.
vlorentz added a subtask: T359: Indexers: batch content analyzer infrastructure.
vlorentz closed subtask T1455: Add a journal client that schedules oneshot tasks for metadata indexers as Resolved.Jan 10 2019, 5:12 PM
ardumont closed subtask T1466: Deploy swh-indexer-journal-client as Resolved.Jan 14 2019, 11:33 AM
ardumont closed subtask T359: Indexers: batch content analyzer infrastructure as Resolved.Jan 15 2019, 2:44 PM
ardumont added a subscriber: ardumont.

I'd say using grafana but first we need to determine what are the sensible metrics ;)

vlorentz added revisions: D953: Prevent repository normalization from crashing on malformed input., D950: Prevent license normalization from crashing on malformed input., D949: Better error message on content retrieval error + don't write translated_metadata=None to the idx storage., D951: Fix normalization of npm's 'repository' and 'bugs' fields..Jan 15 2019, 3:57 PM
vlorentz added a comment.Jan 31 2019, 1:16 PM

The only remaining crashes are caused by T1507.

vlorentz closed this task as Resolved.Jan 31 2019, 1:16 PM
gitlab-migration changed the status of subtask T359: Indexers: batch content analyzer infrastructure from Resolved to Migrated.Jan 8 2023, 4:18 PM
gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 4:26 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T1455: Add a journal client that schedules oneshot tasks for metadata indexers from Resolved to Migrated.Jan 8 2023, 4:26 PM
gitlab-migration changed the status of subtask T1466: Deploy swh-indexer-journal-client from Resolved to Migrated.Jan 8 2023, 9:58 PM
About · Developer information · Legal