Page MenuHomeSoftware Heritage
Create Task
Maniphest T2202

Collect extrinsic metadata
Closed, MigratedEdits Locked
Actions

Description

This is a meta-task that tracks activities related to collecting, storing, indexing and showing (API and WebApp) metadata coming from external sources.

We want a working pipeline (see T1344 and T1739 about specs), tested on at least a real-world instance, e.g.:

  • extrinsic origin metadata from GitHub T1740
  • extrinsic metadata from SwMath T2031
  • ClearlyDefined metadata

Metadata workflow boards can be found at (do we need two separate workboards?):

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT3097 Expose metadata in the WebApp and make it searchable
 Migratedgitlab-migrationT2064 Add metadata from deposits to metadata search
 Migratedgitlab-migrationT2073 Index extrinsic metadata from the journal in swh-search/Elasticsearch
Restricted Maniphest Task
 Migratedgitlab-migrationT4401 Index metadata from the deposit
 Migratedgitlab-migrationT4459 Deploy swh-indexer > v2.6 on staging then production
 Migratedgitlab-migrationT4429 Deploy swh-indexer v2.3.0 on production and staging
 Migratedgitlab-migrationT4477 staging origin intrinsic metadata indexer are stuck
 Migratedgitlab-migrationT4694 Use directory metadata in origin search
Unknown Object (Maniphest Task)
 Migratedgitlab-migrationT2201 Indexing / mining
 Migratedgitlab-migrationT2202 Collect extrinsic metadata
Restricted Maniphest Task
 Migratedgitlab-migrationT734 Create strategy for extracting metadata from external-catalogs
 Migratedgitlab-migrationT2328 Collect metadata about software from ScanR
 Migratedgitlab-migrationT2329 Extend CodeMeta vocabulary to qualify author relationships
 Migratedgitlab-migrationT2311 Review the deposit of CodeMeta metadata in xml (following SWORD V2 specs)
Migratedgitlab-migrationT2312 Review metadata deposit specs for metadata-only deposit
 Migratedgitlab-migrationT2512 Make all loaders write their extrinsic metadata to the appropriate storage
 Migratedgitlab-migrationT2496 Write deposit metadata for revisions in the generic metadata storage
 Migratedgitlab-migrationT2514 Add raw_extrinsic_metadata to the journal backfiller
 Migratedgitlab-migrationT2074 Publish extrinsic metadata to swh-journal/Kafka
 Migratedgitlab-migrationT2075 Implement metadata authority specification
 Migratedgitlab-migrationT2343 Decide where/how to store extrinsinc metadata
 Migratedgitlab-migrationT2306 Generic storage for extrinsic, qualified metadata related to any node of the swh archive
 Migratedgitlab-migrationT833 When listing an origin, add origin level metadata to RMD storage
 Migratedgitlab-migrationT1740 fetch extrinsic origin metadata from GitHub
 Migratedgitlab-migrationT1344 Write specs about metadata workflow
Migratedgitlab-migrationT1738 Define and specify extrinsic origin metadata
 Migratedgitlab-migrationT4186 Allow loaders to fetch extrinsic metadata
 Migratedgitlab-migrationT4187 Pass forge type to loaders
 Migratedgitlab-migrationT4188 Make swh-loader-core run metadata fetchers before loading an origin
 Migratedgitlab-migrationT4193 staging: Deploy metadata loader
 Migratedgitlab-migrationT4194 staging: Deploy swh-scheduler 1.1.0
 Migratedgitlab-migrationT4195 staging: Deploy swh-loader-core 3.1.0
 Migratedgitlab-migrationT4189 Deploy swh.loader.core v3.0 (and other impacted loaders)
 Migratedgitlab-migrationT4206 prod: Deploy metadata loader v0.0.2
 Migratedgitlab-migrationT4204 prod: Deploy swh-scheduler 1.1.1
 Migratedgitlab-migrationT4205 prod: Deploy swh-loader-core 3.2.1
 Migratedgitlab-migrationT3490 Collect metadata from ClearlyDefined
 Migratedgitlab-migrationT2693 fetch extrinsic origin metadata from GitLab instances
 Migratedgitlab-migrationT1739 Define an architecture to fetch extrinsic metadata outside listers and loaders
 Migratedgitlab-migrationT1737 Define and specify metadata providers
 Migratedgitlab-migrationT1747 Review APIs to get metadata from supported origins
 Migratedgitlab-migrationT1748 Review which extrinsic metadata we want to fetch and archive
 Migratedgitlab-migrationT3542 Decide what metadata we want to / can collect from GitHub
 Migratedgitlab-migrationT3681 Review extrinsic metadata specification
 Migratedgitlab-migrationT4252 Schedule recurring fetches of origin metadata
 Migratedgitlab-migrationT4394 Add support for running metadata fetchers without a VCS/package loaders
 Migratedgitlab-migrationT4728 Add monitoring of API token usage
 Migratedgitlab-migrationT4377 Create API endpoint to expose raw extrinsic metadata on origins
 Migratedgitlab-migrationT4451 Fetch extrinsic metadata from Gitea/Gogs
 Migratedgitlab-migrationT4605 Deploy swh-loader-metadata v0.0.3
 Migratedgitlab-migrationT4606 Deploy swh-indexer v2.7.0

Event Timeline

douardda created this task.Jan 20 2020, 2:12 PM
vlorentz added subtasks: T1740: fetch extrinsic origin metadata from GitHub, Restricted Maniphest Task.Jan 20 2020, 2:54 PM
vlorentz triaged this task as Normal priority.Jan 22 2020, 4:19 PM
moranegg added a subtask: T734: Create strategy for extracting metadata from external-catalogs.Jan 30 2020, 1:52 PM
rdicosmo added a subtask: T2328: Collect metadata about software from ScanR.Mar 20 2020, 7:02 PM
vlorentz added a subtask: T2311: Review the deposit of CodeMeta metadata in xml (following SWORD V2 specs) .Mar 24 2020, 12:05 PM
ardumont changed the status of subtask T2306: Generic storage for extrinsic, qualified metadata related to any node of the swh archive from Open to Work in Progress.Jul 1 2020, 3:50 PM
vlorentz closed subtask T2306: Generic storage for extrinsic, qualified metadata related to any node of the swh archive as Resolved.Jul 29 2020, 7:33 PM
vlorentz closed subtask T2514: Add raw_extrinsic_metadata to the journal backfiller as Resolved.Jul 31 2020, 2:25 PM
vlorentz changed the status of subtask T2512: Make all loaders write their extrinsic metadata to the appropriate storage from Open to Work in Progress.Aug 17 2020, 2:38 PM
vlorentz changed the status of subtask T2513: Copy metadata on revisions to the extrinsic metadata storage from Open to Work in Progress.Aug 28 2020, 1:20 PM
vlorentz closed subtask T2512: Make all loaders write their extrinsic metadata to the appropriate storage as Resolved.Sep 15 2020, 1:58 PM
moranegg added a subtask: T833: When listing an origin, add origin level metadata to RMD storage.Sep 18 2020, 2:35 PM
moranegg edited projects, added Extrinsic metadata; removed Metadata workflow.Sep 18 2020, 2:39 PM
moranegg closed subtask T2311: Review the deposit of CodeMeta metadata in xml (following SWORD V2 specs) as Resolved.Dec 7 2020, 4:03 PM
rdicosmo mentioned this in T3097: Expose metadata in the WebApp and make it searchable.Mar 8 2021, 10:44 AM
rdicosmo updated the task description. (Show Details)Mar 15 2021, 9:08 PM
rdicosmo edited projects, added Roadmap 2021, meta-task; removed Roadmap 2020.
rdicosmo added subscribers: moranegg, rdicosmo, vlorentz.
vlorentz claimed this task.Apr 2 2021, 10:12 AM
vlorentz renamed this task from Extrinsic metadata to Collect extrinsic metadata.Jun 29 2021, 10:53 AM
vlorentz removed a subtask: T2513: Copy metadata on revisions to the extrinsic metadata storage.Aug 19 2021, 9:20 AM
bchauvet added a project: Roadmap 2022.Mar 23 2022, 4:48 PM
bchauvet raised the priority of this task from Normal to High.Mar 25 2022, 5:30 PM
vlorentz added a subtask: T2693: fetch extrinsic origin metadata from GitLab instances.Apr 21 2022, 9:01 AM
vlorentz closed subtask T1740: fetch extrinsic origin metadata from GitHub as Resolved.May 3 2022, 11:08 AM
vlorentz added a parent task: T3273: Use "fork" relationships to speed-up initial load of large repositories.May 3 2022, 11:16 AM
vlorentz removed a parent task: T3273: Use "fork" relationships to speed-up initial load of large repositories.
vlorentz moved this task from Backlog to Work in progress on the Roadmap 2022 board.Jun 2 2022, 9:57 AM
vlorentz closed subtask T833: When listing an origin, add origin level metadata to RMD storage as Wontfix.Jul 5 2022, 6:33 PM
vlorentz closed subtask T4377: Create API endpoint to expose raw extrinsic metadata on origins as Resolved.Jul 18 2022, 10:31 AM
vlorentz closed subtask T4451: Fetch extrinsic metadata from Gitea/Gogs as Resolved.Oct 24 2022, 10:13 AM
vlorentz moved this task from Work in progress to Pending validation on the Roadmap 2022 board.Nov 24 2022, 11:24 AM
gitlab-migration changed the status of subtask T833: When listing an origin, add origin level metadata to RMD storage from Wontfix to Migrated.Jan 8 2023, 4:23 PM
gitlab-migration changed the status of subtask T2311: Review the deposit of CodeMeta metadata in xml (following SWORD V2 specs) from Resolved to Migrated.Jan 8 2023, 4:29 PM
gitlab-migration changed the status of subtask T4377: Create API endpoint to expose raw extrinsic metadata on origins from Resolved to Migrated.Jan 8 2023, 4:37 PM
gitlab-migration closed subtask T2328: Collect metadata about software from ScanR as Migrated.Jan 8 2023, 5:01 PM
gitlab-migration changed the status of subtask T1740: fetch extrinsic origin metadata from GitHub from Resolved to Migrated.Jan 8 2023, 9:59 PM
gitlab-migration changed the status of subtask T2306: Generic storage for extrinsic, qualified metadata related to any node of the swh archive from Resolved to Migrated.
gitlab-migration changed the status of subtask T2512: Make all loaders write their extrinsic metadata to the appropriate storage from Resolved to Migrated.
gitlab-migration changed the status of subtask T2514: Add raw_extrinsic_metadata to the journal backfiller from Resolved to Migrated.
gitlab-migration changed the status of subtask T4451: Fetch extrinsic metadata from Gitea/Gogs from Resolved to Migrated.Jan 8 2023, 10:04 PM
gitlab-migration closed this task as Migrated.Jan 8 2023, 10:21 PM
gitlab-migration claimed this task.
gitlab-migration closed subtask T734: Create strategy for extracting metadata from external-catalogs as Migrated.
gitlab-migration closed subtask Restricted Maniphest Task as Migrated.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration closed subtask T2329: Extend CodeMeta vocabulary to qualify author relationships as Migrated.Jan 8 2023, 10:22 PM
gitlab-migration closed subtask T2693: fetch extrinsic origin metadata from GitLab instances as Migrated.
gitlab-migration closed subtask T3490: Collect metadata from ClearlyDefined as Migrated.
gitlab-migration closed subtask T4252: Schedule recurring fetches of origin metadata as Migrated.Jan 8 2023, 10:24 PM
About · Developer information · Legal