Changeset View
Changeset View
Standalone View
Standalone View
docs/roadmap/roadmap-2021.rst
- This file was added.
.. _roadmap-2021: | |||||
Roadmap 2021 | |||||
============ | |||||
(Version 1.0, last modified 5/5/2021) | |||||
This document provides an overview of the technical roadmap of Software Heritage for | |||||
2021. | |||||
The `Kanban board <https://forge.softwareheritage.org/project/board/160/query/all/>`_ | |||||
is seen through our forge. | |||||
.. contents:: | |||||
:depth: 3 | |||||
.. | |||||
Collect | |||||
------- | |||||
Faster and more reliable save code now | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: openscience | |||||
- task: `T3082 <https://forge.softwareheritage.org/T3082>`_ | |||||
- lead: ardumont | |||||
- effort: 1 PM | |||||
Includes work: | |||||
- set up dedicated fast track pipeline for save code now | |||||
- improve save code now monitoring (user and admin) | |||||
Improve deposit integration, management and display | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: openscience | |||||
- task: `T3128 <https://forge.softwareheritage.org/T3128>`_ | |||||
- lead: moranegg | |||||
- effort: 3 PM | |||||
Includes work: | |||||
- full invenioRDM integration `T2344 <https://forge.softwareheritage.org/T2344>`_ | |||||
- metadata only deposit `T2540 <https://forge.softwareheritage.org/T2540>`_ | |||||
Save forge now | |||||
^^^^^^^^^^^^^^ | |||||
- tags: expand | |||||
- task: `T1538 <https://forge.softwareheritage.org/T1538>`_ | |||||
- lead: ardumont | |||||
- effort: 1 PM - tooling & process | |||||
Admin tooling for takedown notices (URLs) | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: contract, compliance | |||||
- task: `T3087 <https://forge.softwareheritage.org/T3087>`_ | |||||
- lead: anlambert | |||||
- effort: 2 PM | |||||
Includes work: | |||||
- admin interface | |||||
- journal of operations | |||||
- web page with list of accepted TDN | |||||
Preserve | |||||
-------- | |||||
Complete and up-to-date archive copy on S3 | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: stability | |||||
- task: `T3085 <https://forge.softwareheritage.org/T3085>`_ | |||||
- lead: douardda | |||||
- effort: 1 PM | |||||
Includes work: | |||||
- live update of the objects | |||||
- regular dumps of the (anonymized) Merkle graph | |||||
Scale-out graph storage in production | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: scalability | |||||
- task: `T2214 <https://forge.softwareheritage.org/T2214>`_ | |||||
- lead: vlorentz | |||||
- effort: 3 PM | |||||
Includes work: | |||||
- Cassandra: `T1892 <https://forge.softwareheritage.org/T1892>`_ (*maybe with external help*) | |||||
Scale-out object storage prototype | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: stability, scalability, *externalized* | |||||
- task: `T3054 <https://forge.softwareheritage.org/T3054>`_ | |||||
- lead: dachary | |||||
- effort: 3 PM | |||||
Cold storage archive in Vitam instance at CINES | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: contract | |||||
- task: `T3113 <https://forge.softwareheritage.org/T3113>`_ | |||||
- lead: douardda | |||||
- effort: 4 PM | |||||
Mirrors | |||||
^^^^^^^ | |||||
- tags: stability, scalability | |||||
- depends: scale-out object storage | |||||
- task: `T3116 <https://forge.softwareheritage.org/T3116>`_ | |||||
- lead: douardda | |||||
- effort: 3 PM | |||||
Includes work: | |||||
- get up and running at least one mirror | |||||
SWHID v2 | |||||
^^^^^^^^ | |||||
- tags: stability, evolution, datamodel | |||||
- task: `T3134 <https://forge.softwareheritage.org/T3134>`_ | |||||
- lead: zack | |||||
- effort: 6 PM | |||||
Includes work: | |||||
- complete on paper spec | |||||
- align with new git hashes | |||||
- including migration plan from v1 | |||||
- understand impact on internal microservice architecture | |||||
- keep correspondence with v1 (there may be multiple v2 for one v1!) | |||||
- reviewed by crypto experts | |||||
Integrity | |||||
^^^^^^^^^ | |||||
- tags: stability, reliability | |||||
- task: `T3135 <https://forge.softwareheritage.org/T3135>`_ | |||||
- lead: olasd | |||||
- effort: 2 PM | |||||
Includes work: | |||||
- making sure objects aren’t corrupted before insertion `T399 <https://forge.softwareheritage.org/T399>`_ | |||||
- ... and that existing ones are not part of `T75 <https://forge.softwareheritage.org/T75>`_ | |||||
- make corruption check periodically | |||||
Share | |||||
----- | |||||
swh-graph in production | |||||
^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: scalability | |||||
- task: `T2220 <https://forge.softwareheritage.org/T2220>`_ | |||||
- lead: zack | |||||
- effort: 2 PM | |||||
Efficient and reliable Vault download | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: stability | |||||
- task: `T3096 <https://forge.softwareheritage.org/T3096>`_ | |||||
- lead: vlorentz | |||||
- effort: 3 PM | |||||
Includes work: | |||||
- swh-graph may speed up a lot operations | |||||
Web API 2.0 | |||||
^^^^^^^^^^^ | |||||
- tags: reliability, interoperability | |||||
- task: `T2194 <https://forge.softwareheritage.org/T2194>`_ | |||||
- lead: anlambert | |||||
- effort: 4 PM | |||||
Includes work: | |||||
- OpenAPI specification - implementation | |||||
Expose metadata and make them searchable | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: openscience | |||||
- task: `T3097 <https://forge.softwareheritage.org/T3097>`_ | |||||
- lead: vlorentz | |||||
- effort: 3 PM | |||||
Includes work: | |||||
- index extrinsic metadata in swh-search/Elasticsearch from the journal `T2073 <https://forge.softwareheritage.org/T2073>`_ | |||||
- create API endpoint to access raw_extrinsic_metadata `T2938 <https://forge.softwareheritage.org/T2938>`_ | |||||
- show metadata in the web UI `T2088 <https://forge.softwareheritage.org/T2088>`_ | |||||
Full text search prototype | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: feature, wishlist | |||||
- task: `T2204 <https://forge.softwareheritage.org/T2204>`_ | |||||
- lead: anlambert | |||||
- effort: 3 PM | |||||
Includes work: | |||||
- requires integration with swh-graph and/or provenance index | |||||
Organize | |||||
-------- | |||||
Collect extrinsic metadata | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: compliance | |||||
- task: `T2202 <https://forge.softwareheritage.org/T2202>`_ | |||||
- lead: vlorentz | |||||
- effort: 3 PM | |||||
Includesd work: | |||||
- working pipeline | |||||
- at least 1 instance running ClearlyDefined | |||||
- forge metadata (info on the main page, etc.) | |||||
Provenance in production | |||||
^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: contract, feature | |||||
- task: `T3112 <https://forge.softwareheritage.org/T3112>`_ | |||||
- lead: zack | |||||
- effort: 6 PM | |||||
Prior art | |||||
^^^^^^^^^ | |||||
- tags: compliance | |||||
- depends: provenance \| swh-graph in production | |||||
- task: `T3136 <https://forge.softwareheritage.org/T3136>`_ | |||||
- lead: zack | |||||
- effort: 3 PM | |||||
Includes work: | |||||
- pinpoint origin of selected source code artifacts | |||||
- possibly integrated with swh-scanner | |||||
Measurement | |||||
----------- | |||||
Efficient archive counters (HyperLogLog) | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: measure, comm | |||||
- task: `T2912 <https://forge.softwareheritage.org/T2912>`_ | |||||
- lead: vsellier | |||||
- effort: 1 PM | |||||
Distribution of origins by forge | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: measure, comm | |||||
- task: `T3127 <https://forge.softwareheritage.org/T3127>`_ | |||||
- lead: anlambert | |||||
- effort: 1 PM | |||||
Stats on regular crawling by forge | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: measure, comm | |||||
- task: `T1363 <https://forge.softwareheritage.org/T1363>`_ | |||||
- lead: olasd | |||||
- effort: 1 PM | |||||
Includes work: | |||||
- lag, periodicity, # of changes since last visit, etc. | |||||
View deposits per user (admin and user) | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: measure, support | |||||
- task: `T3128 <https://forge.softwareheritage.org/T3128>`_ | |||||
- lead: ardumont | |||||
- effort: 1 PM | |||||
Reliable user-level monitoring of services | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: stability | |||||
- task: `T3129 <https://forge.softwareheritage.org/T3129>`_ | |||||
- lead: vsellier | |||||
- effort: 2 PM | |||||
Includes work: | |||||
- status.softwareheritage.org | |||||
Documentation | |||||
------------- | |||||
Write use case-specific documentation | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: comm, web, doc | |||||
- task: `T2234 <https://forge.softwareheritage.org/T2234>`_ | |||||
- lead: moranegg | |||||
- effort: 2 PM | |||||
Includes FAQ for: - users - ambassadors | |||||
Improve quality of code documentation | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: doc, *externalized* | |||||
- task: TODO | |||||
- lead: TBD | |||||
- effort: 2PM | |||||
Includes work: | |||||
- doc(string) audit - team training about doc writing | |||||
Documentation strategy | |||||
^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: doc | |||||
- task: `T2624 <https://forge.softwareheritage.org/T2624>`_ | |||||
- lead: moranegg | |||||
- effort: 1 PM | |||||
Includes work: | |||||
- respective role of docs.s.o, wiki, www.s.o, etc. | |||||
Community | |||||
--------- | |||||
Tooling for fundraising campaigns | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: web | |||||
- task: `T3077 <https://forge.softwareheritage.org/T3077>`_ | |||||
- lead: anlambert | |||||
- effort: 1 PM | |||||
Dedicated page to list status of supported listers/loaders | |||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||||
- tags: web, doc | |||||
- task: `T3117 <https://forge.softwareheritage.org/T3117>`_ | |||||
- lead: anlambert | |||||
- effort: 1 PM | |||||
Includes work: | |||||
- `T1870 <https://forge.softwareheritage.org/T1870>`_ | |||||
- design web page | |||||
- process to maintain up to date | |||||
- make clearly visible and link to Sloan subgrants | |||||
Tooling | |||||
------- | |||||
Migration to GitLab | |||||
^^^^^^^^^^^^^^^^^^^ | |||||
- tags: forge, development | |||||
- task: `T2225 <https://forge.softwareheritage.org/T2225>`_ | |||||
- lead: olasd | |||||
- effort: 1PM | |||||