Maniphest T2073

Index extrinsic metadata from the journal in swh-search/Elasticsearch
Open, NormalPublic
Actions

Assigned To

Authored By

	vlorentz
	Nov 8 2019, 5:17 PM

Revisions and Commits

rDCIDX Metadata indexer
	D8060	rDCIDX61b22345e70b Add extrinsic metadata indexer
	D8058	rDCIDXf26b4c8b1ca7 Add support for origin_extrinsic_metadata to the storage
	D8055	rDCIDX151a3b8a2b69 github mapping: Add support for more terms from the Codemeta crosswalk
	D8054	rDCIDX8948c8397251 github mapping: Add support for terms outside the codemeta context
	D8053	rDCIDX9085cae01009 Add minimal GitHub metadata mapping
	D8046	rDCIDXe6b03db91c99 Remove SingleFileMapping from JsonMapping's base classes

Related Objects
Search...

Status	Assigned	Task
Open	None	T4081 Show metadata on Web UI
Work in Progress	vlorentz	T3097 Expose metadata in the WebApp and make it searchable
Open	vlorentz	T2064 Add metadata from deposits to metadata search
Open	vlorentz	T2073 Index extrinsic metadata from the journal in swh-search/Elasticsearch
Resolved	vlorentz	T2074 Publish extrinsic metadata to swh-journal/Kafka
Resolved	vlorentz	T2075 Implement metadata authority specification
Resolved	vlorentz	T1737 Define and specify metadata providers
Resolved	vlorentz	T2343 Decide where/how to store extrinsinc metadata
Resolved	vlorentz	T2306 Generic storage for extrinsic, qualified metadata related to any node of the swh archive
Resolved	vlorentz	T2876 metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types
Open	vlorentz	T4249 Choose/define an ontology to use for indexed extrinsic origin metadata
Open	vlorentz	T4354 Contribute terms to ForgeFed
Resolved	vlorentz	T4273 Rewrite indexers as journal clients when relevant
Resolved	ardumont	T4282 Deploy new origin intrinsic metadata journal client indexer > v1.1
Resolved	vlorentz	T4274 Resolve all known crashes in the metadata indexer
Resolved	VickyMerzOwn	T4275 CffMapping: Add checks for value types
Resolved	VickyMerzOwn	T4276 CffMapping: ignore invalid yaml files
Resolved	vlorentz	T4277 Deal with null characters in the output of the metadata indexer
Resolved	VickyMerzOwn	T4333 test_npm_adversarial fails
Resolved	ardumont	T4395 Migrate azure worker vms to cheaper and more efficient vms
		Restricted Maniphest Task
		Restricted Maniphest Task
Open	vlorentz	T4401 Index metadata from the deposit
Resolved	ardumont	T4459 Deploy swh-indexer > v2.6 on staging then production
Resolved	ardumont	T4429 Deploy swh-indexer v2.3.0 on production and staging
Resolved	ardumont	T4477 staging origin intrinsic metadata indexer are stuck

Event Timeline

vlorentz triaged this task as Normal priority.Nov 8 2019, 5:17 PM

vlorentz created this task.

vlorentz claimed this task.Jul 20 2020, 11:20 AM

vlorentz closed subtask T2074: Publish extrinsic metadata to swh-journal/Kafka as Resolved.Jul 29 2020, 7:36 PM

moranegg moved this task from Backlog to Implementation on the Metadata workflow board.Sep 18 2020, 1:48 PM

vlorentz added a subtask: T2876: metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types.Dec 21 2020, 12:23 PM

vlorentz closed subtask T2876: metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types as Resolved.Dec 22 2020, 10:52 AM

vlorentz mentioned this in D5179: Use elasticsearch aliases to simplify maintenance operations.Mar 2 2021, 11:59 AM

rdicosmo added a parent task: T3097: Expose metadata in the WebApp and make it searchable.Mar 8 2021, 10:33 AM

vlorentz renamed this task from Index extrinsic metadata in swh-search/Elasticsearch from the journal to Index extrinsic metadata from the journal in swh-search/Elasticsearch.Apr 9 2021, 9:33 AM

I'm tempted to postpone this issue until we resolve T2063...

In T2073#70230, @vlorentz wrote:

I'm tempted to postpone this issue until we resolve T2063...

I don't think we're "resolving" T2063 until we need to process tasks that can't reasonably be handled by celery. On the other hand, journal clients are sort of a resolution to T2063.

Would a "simple" journal client, which would be able to mangle known metadata types into something that can be fed into elasticsearch be appropriate here?

On the other hand, journal clients are sort of a resolution to T2063.

Yes, they are. I just want to make sure we explicitly decide to go that way.

Would a "simple" journal client, which would be able to mangle known metadata types into something that can be fed into elasticsearch be appropriate here?

Yes.