Changeset View
Changeset View
Standalone View
Standalone View
docs/roadmap/roadmap-2021.rst
- This file was added.
| .. _roadmap-2021: | |||||
| Roadmap 2021 | |||||
| ============ | |||||
| (Version 1.0, last modified 5/5/2021) | |||||
| This document provides an overview of the technical roadmap of Software Heritage for | |||||
| 2021. | |||||
| The `Kanban board <https://forge.softwareheritage.org/project/board/160/query/all/>`__ | |||||
| is seen through our forge. | |||||
| .. contents:: | |||||
| :depth: 3 | |||||
| .. | |||||
| Collect | |||||
| ------- | |||||
| Faster and more reliable save code now | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: openscience | |||||
| - task: `T3082 <https://forge.softwareheritage.org/T3082>`__ | |||||
| - lead: ardumont | |||||
| - effort: 1 PM | |||||
| Includes work: | |||||
| - set up dedicated fast track pipeline for save code now | |||||
| - improve save code now monitoring (user and admin) | |||||
| Improve deposit integration, management and display | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: openscience | |||||
| - task: `T3128 <https://forge.softwareheritage.org/T3128>`__ | |||||
| - lead: moranegg | |||||
| - effort: 3 PM | |||||
| Includes work: | |||||
| - full invenioRDM integration `T2344 <https://forge.softwareheritage.org/T2344>`__ | |||||
| - metadata only deposit `T2540 <https://forge.softwareheritage.org/T2540>`__ | |||||
| Save forge now | |||||
| ^^^^^^^^^^^^^^ | |||||
| - tags: expand | |||||
| - task: `T1538 <https://forge.softwareheritage.org/T1538>`__ | |||||
| - lead: ardumont | |||||
| - effort: 1 PM - tooling & process | |||||
| Admin tooling for takedown notices (URLs) | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: contract, compliance | |||||
| - task: `T3087 <https://forge.softwareheritage.org/T3087>`__ | |||||
| - lead: anlambert | |||||
| - effort: 2 PM | |||||
| Includes work: | |||||
| - admin interface | |||||
| - journal of operations | |||||
| - web page with list of accepted TDN | |||||
| Preserve | |||||
| -------- | |||||
| Complete and up-to-date archive copy on S3 | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: stability | |||||
| - task: `T3085 <https://forge.softwareheritage.org/T3085>`__ | |||||
| - lead: douardda | |||||
| - effort: 1 PM | |||||
| Includes work: | |||||
| - live update of the objects | |||||
| - regular dumps of the (anonymized) Merkle graph | |||||
| Scale-out graph storage in production | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: scalability | |||||
| - task: `T2214 <https://forge.softwareheritage.org/T2214>`__ | |||||
| - lead: vlorentz | |||||
| - effort: 3 PM | |||||
| Includes work: | |||||
| - post-Postgres database | |||||
| - Cassandra: `T1892 <https://forge.softwareheritage.org/T1892>`__ (*maybe with external help*) | |||||
| Scale-out object storage prototype | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: stability, scalability, *externalized* | |||||
| - task: `T3054 <https://forge.softwareheritage.org/T3054>`__ | |||||
| - lead: dachary | |||||
| - effort: 3 PM | |||||
| Cold storage archive in Vitam instance at CINES | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: contract | |||||
| - task: `T3113 <https://forge.softwareheritage.org/T3113>`__ | |||||
| - lead: douardda | |||||
| - effort: 4 PM | |||||
| Mirrors | |||||
| ^^^^^^^ | |||||
| - tags: stability, scalability | |||||
| - depends: scale-out object storage | |||||
| - task: `T3116 <https://forge.softwareheritage.org/T3116>`__ | |||||
| - lead: douardda | |||||
| - effort: 3 PM | |||||
| Includes work: | |||||
| - get up and running at least one mirror | |||||
| SWHID v2 | |||||
| ^^^^^^^^ | |||||
| - tags: stability, evolution, datamodel | |||||
| - task: `T3134 <https://forge.softwareheritage.org/T3134>`__ | |||||
| - lead: zack | |||||
| - effort: 6 PM | |||||
| Includes work: | |||||
| - complete on paper spec | |||||
| - align with new git hashes | |||||
| - including migration plan from v1 | |||||
| - understand impact on internal microservice architecture | |||||
| - keep correspondence with v1 (there may be multiple v2 for one v1!) | |||||
| - reviewed by crypto experts | |||||
| Integrity | |||||
| ^^^^^^^^^ | |||||
| - tags: stability, reliability | |||||
| - task: `T3135 <https://forge.softwareheritage.org/T3135>`__ | |||||
| - lead: olasd | |||||
| - effort: 2 PM | |||||
| Includes work: | |||||
| - making sure objects aren’t corrupted before insertion `T399 <https://forge.softwareheritage.org/T399>`__ - … and that existing ones aren’t `T75 <https://forge.softwareheritage.org/T75>`__, and recheck from time to time | |||||
| Share | |||||
| ----- | |||||
| swh-graph in production | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: scalability | |||||
| - task: `T2220 <https://forge.softwareheritage.org/T2220>`__ | |||||
| - lead: zack | |||||
| - effort: 2 PM | |||||
| Efficient and reliable Vault download | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: stability | |||||
| - task: `T3096 <https://forge.softwareheritage.org/T3096>`__ | |||||
| - lead: vlorentz | |||||
| - effort: 3 PM | |||||
| Includes work: | |||||
| - swh-graph may speed up a lot operations | |||||
| Web API 2.0 | |||||
| ^^^^^^^^^^^ | |||||
| - tags: reliability, interoperability | |||||
| - task: `T2194 <https://forge.softwareheritage.org/T2194>`__ | |||||
| - lead: anlambert | |||||
| - effort: 4 PM | |||||
| Includes work: | |||||
| - OpenAPI specification - implementation | |||||
| Expose metadata and make them searchable | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: openscience | |||||
| - task: `T3097 <https://forge.softwareheritage.org/T3097>`__ | |||||
| - lead: vlorentz | |||||
| - effort: 3 PM | |||||
| Includes work: | |||||
| - index extrinsic metadata in swh-search/Elasticsearch from the journal `T2073 <https://forge.softwareheritage.org/T2073>`__ | |||||
| - create API endpoint to access raw_extrinsic_metadata `T2938 <https://forge.softwareheritage.org/T2938>`__ | |||||
| - show metadata in the web UI `T2088 <https://forge.softwareheritage.org/T2088>`__ | |||||
| Full text search prototype | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: feature, wishlist | |||||
| - task: `T2204 <https://forge.softwareheritage.org/T2204>`__ | |||||
| - lead: anlambert | |||||
| - effort: 3 PM | |||||
| Includes work: | |||||
| z | |||||
| - requires integration with swh-graph and/or provenance index | |||||
| Organize | |||||
| -------- | |||||
| Collect extrinsic metadata | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: compliance | |||||
| - task: `T2202 <https://forge.softwareheritage.org/T2202>`__ | |||||
| - lead: vlorentz | |||||
| - effort: 3 PM | |||||
| Includesd work: | |||||
| - working pipeline | |||||
| - at least 1 instance running ClearlyDefined | |||||
| - forge metadata (info on the main page, etc.) | |||||
| Provenance in production | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: contract, feature | |||||
| - task: `T3112 <https://forge.softwareheritage.org/T3112>`__ | |||||
| - lead: zack | |||||
| - effort: 6 PM | |||||
| Prior art | |||||
| ^^^^^^^^^ | |||||
| - tags: compliance | |||||
| - depends: provenance \| swh-graph in production | |||||
| - task: `T3136 <https://forge.softwareheritage.org/T3136>`__ | |||||
| - lead: zack | |||||
| - effort: 3 PM | |||||
| Includes work: | |||||
| - pinpoint origin of selected source code artifacts | |||||
| - possibly integrated with swh-scanner | |||||
| Measurement | |||||
| ----------- | |||||
| Efficient archive counters (HyperLogLog) | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: measure, comm | |||||
| - task: `T2912 <https://forge.softwareheritage.org/T2912>`__ | |||||
| - lead: vsellier | |||||
| - effort: 1 PM | |||||
| Distribution of origins by forge | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: measure, comm | |||||
| - task: `T3127 <https://forge.softwareheritage.org/T3127>`__ | |||||
| - lead: anlambert | |||||
| - effort: 1 PM | |||||
| Stats on regular crawling by forge | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: measure, comm | |||||
| - task: `T1363 <https://forge.softwareheritage.org/T1363>`__ | |||||
| - lead: olasd | |||||
| - effort: 1 PM | |||||
| Includes work: | |||||
| - lag, periodicity, # of changes since last visit, etc. | |||||
| View deposits per user (admin and user) | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: measure, support | |||||
| - task: `T3128 <https://forge.softwareheritage.org/T3128>`__ | |||||
| - lead: ardumont | |||||
| - effort: 1 PM | |||||
| Reliable user-level monitoring of services | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: stability | |||||
| - task: `T3129 <https://forge.softwareheritage.org/T3129>`__ | |||||
| - lead: vsellier | |||||
| - effort: 2 PM | |||||
| Includes work: | |||||
| - status.softwareheritage.org | |||||
| Documentation | |||||
| ------------- | |||||
| Write use case-specific documentation | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: comm, web, doc | |||||
| - task: `T2234 <https://forge.softwareheritage.org/T2234>`__ | |||||
| - lead: moranegg | |||||
| - effort: 2 PM | |||||
| Includes FAQ for: - users - ambassadors | |||||
| Improve quality of code documentation | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: doc, *externalized* | |||||
| - task: TODO | |||||
| - lead: TBD | |||||
| - effort: 2PM | |||||
| Includes work: | |||||
| - doc(string) audit - team training about doc writing | |||||
| Documentation strategy | |||||
| ^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: doc | |||||
| - task: `T2624 <https://forge.softwareheritage.org/T2624>`__ | |||||
| - lead: moranegg | |||||
| - effort: 1 PM | |||||
| Includes work: | |||||
| - respective role of docs.s.o, wiki, www.s.o, etc. | |||||
| Community | |||||
| --------- | |||||
| Tooling for fundraising campaigns | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: web | |||||
| - task: `T3077 <https://forge.softwareheritage.org/T3077>`__ | |||||
| - lead: anlambert | |||||
| - effort: 1 PM | |||||
| Dedicated page to list status of supported listers/loaders | |||||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: web, doc | |||||
| - task: `T3117 <https://forge.softwareheritage.org/T3117>`__ | |||||
| - lead: anlambert | |||||
| - effort: 1 PM | |||||
| Includes work: | |||||
| - `T1870 <https://forge.softwareheritage.org/T1870>`__ | |||||
| - design web page | |||||
| - process to maintain up to date | |||||
| - make clearly visible and link to Sloan subgrants | |||||
| Tooling | |||||
| ------- | |||||
| Migration to GitLab | |||||
| ^^^^^^^^^^^^^^^^^^^ | |||||
| - tags: forge, development | |||||
| - task: `T2225 <https://forge.softwareheritage.org/T2225>`__ | |||||
| - lead: olasd | |||||
| - effort: 1PM | |||||