Write specs about metadata workflow
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	moranegg
	Nov 14 2018, 4:03 PM

Description

(opening tasks for myself on my last day might not be the best idea, but this should be written somewhere)

The metadata workflow and strategy is about recovering descriptive metadata on the artifacts in the archive.
This metadata can be found:

in the content itself -> intrinsic metadata (implemented with T1232)
not in the content -> extrinsic metadata
- extrinsic metadata can be found with the content when listing or loading the content
- or in a software registry (e.g Wikidata, swMath, ASCL..)

The different components and the storage infrastructure that was put in place to keep this information
should be specified and documented.

A discussion started over the metadata_provider in D637.

Status	Assigned	Task
Migrated	gitlab-migration	T2201 Indexing / mining
Migrated	gitlab-migration	T2202 Collect extrinsic metadata
Migrated	gitlab-migration	T833 When listing an origin, add origin level metadata to RMD storage
Migrated	gitlab-migration	T4283 Load https://github.com/chromium/chromium with a higher packfile size limit
Migrated	gitlab-migration	T3273 Use "fork" relationships to speed-up initial load of large repositories
Migrated	gitlab-migration	T1102 Handle all GitHub elements
Migrated	gitlab-migration	T1740 fetch extrinsic origin metadata from GitHub
		Unknown Object (Maniphest Task)
Migrated	gitlab-migration	T1344 Write specs about metadata workflow
Migrated	gitlab-migration	T1738 Define and specify extrinsic origin metadata
Migrated	gitlab-migration	T1739 Define an architecture to fetch extrinsic metadata outside listers and loaders
Migrated	gitlab-migration	T1737 Define and specify metadata providers
Migrated	gitlab-migration	T1747 Review APIs to get metadata from supported origins
Migrated	gitlab-migration	T1748 Review which extrinsic metadata we want to fetch and archive
Migrated	gitlab-migration	T3542 Decide what metadata we want to / can collect from GitHub
Migrated	gitlab-migration	T3681 Review extrinsic metadata specification

moranegg triaged this task as Normal priority.Nov 14 2018, 4:03 PM

moranegg created this task.

Once this task is handled, an improved docstring for Storage.metadata_provider_add would be necessary.

Where would you put this type of specs?

Either in the docs (like the persistent identifiers spec) or on the wiki (like the snapshot spec).

What about writing it on the wiki, and moving it to the docs when it's finished?

EDIT: nevermind, it's discussed in T1683

moranegg added a parent task: Unknown Object (Maniphest Task).Jun 14 2019, 12:33 PM

This can be resolved :-)

It's still missing " Define an architecture to fetch extrinsic metadata outside listers and loaders"

but this is an overly broad task, so indeed, let's close it so it doesn't clutter dashboards