Page MenuHomeSoftware Heritage
Create Task
Maniphest T2073

Index extrinsic metadata from the journal in swh-search/Elasticsearch
Closed, MigratedEdits Locked
Actions

Revisions and Commits

rDCIDX Metadata indexer
D8060rDCIDX61b22345e70b Add extrinsic metadata indexer
D8058rDCIDXf26b4c8b1ca7 Add support for origin_extrinsic_metadata to the storage
D8055rDCIDX151a3b8a2b69 github mapping: Add support for more terms from the Codemeta crosswalk
D8054rDCIDX8948c8397251 github mapping: Add support for terms outside the codemeta context
D8053rDCIDX9085cae01009 Add minimal GitHub metadata mapping
D8046rDCIDXe6b03db91c99 Remove SingleFileMapping from JsonMapping's base classes

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT3097 Expose metadata in the WebApp and make it searchable
 Migratedgitlab-migrationT2064 Add metadata from deposits to metadata search
 Migratedgitlab-migrationT2073 Index extrinsic metadata from the journal in swh-search/Elasticsearch
 Migratedgitlab-migrationT2074 Publish extrinsic metadata to swh-journal/Kafka
 Migratedgitlab-migrationT2075 Implement metadata authority specification
 Migratedgitlab-migrationT1737 Define and specify metadata providers
 Migratedgitlab-migrationT2343 Decide where/how to store extrinsinc metadata
 Migratedgitlab-migrationT2306 Generic storage for extrinsic, qualified metadata related to any node of the swh archive
 Migratedgitlab-migrationT2876 metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types
 Migratedgitlab-migrationT4249 Choose/define an ontology to use for indexed extrinsic origin metadata
 Migratedgitlab-migrationT4354 Contribute terms to ForgeFed
 Migratedgitlab-migrationT4273 Rewrite indexers as journal clients when relevant
 Migratedgitlab-migrationT4282 Deploy new origin intrinsic metadata journal client indexer > v1.1
 Migratedgitlab-migrationT4274 Resolve all known crashes in the metadata indexer
 Migratedgitlab-migrationT4275 CffMapping: Add checks for value types
 Migratedgitlab-migrationT4276 CffMapping: ignore invalid yaml files
 Migratedgitlab-migrationT4277 Deal with null characters in the output of the metadata indexer
 Migratedgitlab-migrationT4333 test_npm_adversarial fails
 Migratedgitlab-migrationT4395 Migrate azure worker vms to cheaper and more efficient vms
Restricted Maniphest Task
Restricted Maniphest Task
 Migratedgitlab-migrationT4401 Index metadata from the deposit
 Migratedgitlab-migrationT4459 Deploy swh-indexer > v2.6 on staging then production
 Migratedgitlab-migrationT4429 Deploy swh-indexer v2.3.0 on production and staging
 Migratedgitlab-migrationT4477 staging origin intrinsic metadata indexer are stuck
 Migratedgitlab-migrationT4606 Deploy swh-indexer v2.7.0
 Migratedgitlab-migrationT4694 Use directory metadata in origin search

Event Timeline

vlorentz triaged this task as Normal priority.Nov 8 2019, 5:17 PM
vlorentz created this task.
vlorentz claimed this task.Jul 20 2020, 11:20 AM
vlorentz closed subtask T2074: Publish extrinsic metadata to swh-journal/Kafka as Resolved.Jul 29 2020, 7:36 PM
moranegg moved this task from Backlog to Implementation on the Metadata workflow board.Sep 18 2020, 1:48 PM
vlorentz added a subtask: T2876: metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types.Dec 21 2020, 12:23 PM
vlorentz closed subtask T2876: metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types as Resolved.Dec 22 2020, 10:52 AM
vlorentz mentioned this in D5179: Use elasticsearch aliases to simplify maintenance operations.Mar 2 2021, 11:59 AM
rdicosmo added a parent task: T3097: Expose metadata in the WebApp and make it searchable.Mar 8 2021, 10:33 AM
vlorentz renamed this task from Index extrinsic metadata in swh-search/Elasticsearch from the journal to Index extrinsic metadata from the journal in swh-search/Elasticsearch.Apr 9 2021, 9:33 AM
vlorentz added a comment.Sep 10 2021, 4:05 PM

I'm tempted to postpone this issue until we resolve T2063...

olasd added a subscriber: olasd.Sep 13 2021, 11:36 AM
In T2073#70230, @vlorentz wrote:

I'm tempted to postpone this issue until we resolve T2063...

I don't think we're "resolving" T2063 until we need to process tasks that can't reasonably be handled by celery. On the other hand, journal clients are sort of a resolution to T2063.

Would a "simple" journal client, which would be able to mangle known metadata types into something that can be fed into elasticsearch be appropriate here?

vlorentz added a comment.Sep 13 2021, 2:42 PM

On the other hand, journal clients are sort of a resolution to T2063.

Yes, they are. I just want to make sure we explicitly decide to go that way.

Would a "simple" journal client, which would be able to mangle known metadata types into something that can be fed into elasticsearch be appropriate here?

Yes.

vlorentz added revisions: D8060: Add extrinsic metadata indexer, D8058: Add support for origin_extrinsic_metadata to the storage, D8055: github mapping: Add support for more terms from the Codemeta crosswalk, D8054: github mapping: Add support for terms outside the codemeta context, D8053: Add minimal GitHub metadata mapping, D8046: Remove SingleFileMapping from JsonMapping's base classes.Jun 30 2022, 3:31 PM
vlorentz added a commit: rDCIDXe6b03db91c99: Remove SingleFileMapping from JsonMapping's base classes.Jul 4 2022, 12:29 PM
vlorentz added a commit: rDCIDX9085cae01009: Add minimal GitHub metadata mapping.Jul 4 2022, 12:47 PM
vlorentz added a commit: rDCIDX8948c8397251: github mapping: Add support for terms outside the codemeta context.
vlorentz added a commit: rDCIDX151a3b8a2b69: github mapping: Add support for more terms from the Codemeta crosswalk.
vlorentz added a commit: rDCIDX61b22345e70b: Add extrinsic metadata indexer.
vlorentz added a commit: rDCIDXf26b4c8b1ca7: Add support for origin_extrinsic_metadata to the storage.
vlorentz added a parent task: Restricted Maniphest Task.Jul 19 2022, 1:01 PM
vlorentz removed a parent task: Restricted Maniphest Task.Aug 5 2022, 10:55 AM
vlorentz added a subtask: Restricted Maniphest Task.
vlorentz closed subtask T4273: Rewrite indexers as journal clients when relevant as Resolved.Aug 8 2022, 9:59 AM
vlorentz closed subtask Restricted Maniphest Task as Resolved.Nov 2 2022, 11:50 AM
vlorentz closed this task as Resolved.Dec 1 2022, 4:16 PM
gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 4:28 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T4273: Rewrite indexers as journal clients when relevant from Resolved to Migrated.Jan 8 2023, 4:36 PM
gitlab-migration changed the status of subtask Restricted Maniphest Task from Resolved to Migrated.
gitlab-migration closed subtask T4249: Choose/define an ontology to use for indexed extrinsic origin metadata as Migrated.Jan 8 2023, 5:04 PM
gitlab-migration changed the status of subtask T2074: Publish extrinsic metadata to swh-journal/Kafka from Resolved to Migrated.Jan 8 2023, 10:00 PM
gitlab-migration changed the status of subtask T2876: metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types from Resolved to Migrated.
About · Developer information · Legal