Page MenuHomeSoftware Heritage
Differential D4524

Remove <sword:request>.
ClosedPublic
Actions

Authored by vlorentz on Nov 19 2020, 1:57 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDDEP405b577f42ef: Remove <sword:request>.
Summary

Motivation:

  1. it is not a SWORD tag
  2. I don't think there is any use for this
  3. Its content is buggy, as it only handles one level of hierarchy for the metadata values.

Diff Detail

Repository
rDDEP Push deposit
Branch
reorganize-sword
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 17240
Build 26616: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 26615: arc lint + arc unit

Event Timeline

vlorentz created this revision.Nov 19 2020, 1:57 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptNov 19 2020, 1:57 PM
swh-public-ci added a comment.Nov 19 2020, 2:01 PM

Build is green

Patch application report for D4524 (id=16034)

Rebasing onto 85416fb165...

Current branch diff-target is up to date.
Changes applied before test
commit 918d39057abe4a9194b0f7a0cc04e39bd297ec13
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 13:56:39 2020 +0100

    Remove <sword:request>.
    
    Motivation:
    
    1. it is not a SWORD tag
    2. I don't think there is any use for this
    3. Its content is buggy, as it only handles one level of hierarchy for the metadata values.

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/385/ for more details.

Harbormaster completed remote builds in B17240: Diff 16034.Nov 19 2020, 2:01 PM
ardumont added a subscriber: ardumont.Nov 19 2020, 3:58 PM

i don't recall what this is... ¯\_(ツ)_/¯
Let's see if removing it make people scream i guess.

ardumont accepted this revision.Nov 19 2020, 3:58 PM
This revision is now accepted and ready to land.Nov 19 2020, 3:58 PM
vlorentz updated this revision to Diff 16097.Nov 20 2020, 11:05 AM

rebase + update doc

swh-public-ci added a comment.Nov 20 2020, 11:08 AM

Build is green

Patch application report for D4524 (id=16097)

Rebasing onto cb733dd49a...

Current branch diff-target is up to date.
Changes applied before test
commit 8ff2e68d8c32ebed6b11f062f643612e6a569725
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 13:56:39 2020 +0100

    Remove <sword:request>.
    
    Motivation:
    
    1. it is not a SWORD tag
    2. I don't think there is any use for this
    3. Its content is buggy, as it only handles one level of hierarchy for the metadata values.

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/399/ for more details.

Harbormaster completed remote builds in B17295: Diff 16097.Nov 20 2020, 11:08 AM
vlorentz updated this revision to Diff 16126.Nov 20 2020, 3:37 PM

rebase

swh-public-ci added a comment.Nov 20 2020, 3:40 PM

Build is green

Patch application report for D4524 (id=16126)

Rebasing onto f4b4c25e44...

Current branch diff-target is up to date.
Changes applied before test
commit 405b577f42efdb678aefa044b10599c7d2a1e97b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 13:56:39 2020 +0100

    Remove <sword:request>.
    
    Motivation:
    
    1. it is not a SWORD tag
    2. I don't think there is any use for this
    3. Its content is buggy, as it only handles one level of hierarchy for the metadata values.

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/409/ for more details.

Harbormaster completed remote builds in B17326: Diff 16126.Nov 20 2020, 3:40 PM
Closed by commit rDDEP405b577f42ef: Remove <sword:request>. (authored by vlorentz). · Explain WhyNov 20 2020, 4:52 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDDEP405b577f42ef: Remove <sword:request>..

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
deposit/
templates/
deposit/
10 lines--
tests/
api/
2 lines100%
Loading...

Diff 16034

View Options

swh/deposit/templates/deposit/content.xml

  • This file was added.
.. _planning-a-mirror:
Hosting a mirror
================
This section present and discuss the technical requirements needed to host a
|SWH| mirror.
There are many different options to host a mirror, but there are common overall
requirements that needs to be fulfilled.
Namely, hosting a mirror requires:
- a dedicated infrastructure with enough compute (s/computing) power and storage
- enough network bandwidth (both ingress and egress)
- good IT tooling (supervision, alerting).
The mirror operator is not required to run the Software Heritage `full software
stack <https://docs.softwareheritage.org/devel>`_, however it is possible to
use it.
.. Warning::
Volumes given in this section are estimations and numbers from **January
2022**.
The global raw hardware requirements are:
- a database system for the main storage of the archive (the graph structure);
the current volume is about 17TB, with an increase rate of about
280GB/month,
- an object storage system for the objects (archived software source code
files); the current volume is about 800TB with an increase rate of
about 21TB/month,
- an elasticsearch engine; the current main index is about 180M entries
(origins) for an index size of 360GB; the increase rate is about 2M
entries/month,
- a web/application server for the main web application and public API,
- a few compute nodes for the application services.
A mirror should provision machines or cloud-based resources with these numbers
in mind. This should include the usual robustness margins (RAID-like storage,
replication, backup etc.).
ardumontUnsubmitted
Done
Inline Actions
- a few compute nodes for the application services.
A mirror should provision machines or cloud-based resources with these numbers
- in mind, and should include usual the robustness margin (RAID-like storage,
+ in mind. This should include the usual robustness margin (RAID-like storage,
replication, backup etc.).
General hardware requirements
ardumont:
General hardware requirements
-----------------------------
When deploying a mirror based on the Software Heritage software stack, one will
need:
Core services
^^^^^^^^^^^^^
- a database for the storage; this can be either a
`Postgresql <https://postgresql.org>`_ database (single machine)
or a `Cassandra <https://cassandra.apache.org>`_ cluster (at least 3 nodes),
- an object storage system; this can be any
:py:mod:`supported backend <swh.objstorage.backends>`
-- a public cloud-based obstorage (e.g. s3), any private supported object storage,
an ad-hoc filesystem storage system, etc.
- an `elasticsearch <https://www.elastic.co>`_ instance,
- a few nodes for backend applications
(:py:mod:`swh-storage <swh.storage>`, :py:mod:`swh-objstorage <swh.objstorage>`)
- the web frontend (:py:mod:`swh-web <swh.web>`)
serving the main web app and the `public
API <https://docs.softwareheritage.org/devel/swh-web/uri-scheme-api.html>`_)
Replaying services
^^^^^^^^^^^^^^^^^^
- `graph
replayers <https://docs.softwareheritage.org/devel/swh-storage/cli.html#swh-storage-replay>`_
as mirroring workers (increase parallelism to increase speed)
- `content
ardumontUnsubmitted
Done
Inline Actions
^^^^^^^^^^^^^^^^^^
- - a fair amount of `graph
+ - `graph
replayers <https://docs.softwareheritage.org/devel/swh-storage/cli.html#swh-storage-replay>`_
- mirroring workers
+ as mirroring workers (increase parallelism to increase speed)
- a fair amount of `content

or something

ardumont: or something
replayers <https://docs.softwareheritage.org/devel/swh-objstorage-replayer/cli.html>`_
as mirroring workers (id.)
ardumontUnsubmitted
Done
Inline Actions

^ same remark

ardumont: ^ same remark
Vault service
^^^^^^^^^^^^^
- a node for the :ref:`swh-vault <swh-vault>` backend service,
- a node for the :ref:`swh-vault <swh-vault>` worker service
Sizing a mirror infrastructure
vlorentzUnsubmitted
Done
Inline Actions
^^^^^^^^^^^^^
- - a node for the `swh-vault`_ backend service,
- - a node for the `swh-vault`_ worker service
-
- .. _`swh-vault`: https://docs.softwareheritage.org/devel/swh-vault
+ - a node for the :ref:`swh-vault <swh-vault>` backend service,
+ - a node for the :ref:`swh-vault <swh-vault>` worker service
Sizing a mirror infrastructure
vlorentz:
------------------------------
.. Note:: solutions with a star (*) in the tables below are still under test or
validation.
Common components
^^^^^^^^^^^^^^^^^
vlorentzUnsubmitted
Not Done
Inline Actions

nice

vlorentz: nice
================ ====================== ========= ===== ============== ==============
SWH Service Tool Instances RAM Storage Type Storage Volume
================ ====================== ========= ===== ============== ==============
storage swh-storage 16 16GB regular 10GB
search elasticsearch 3 32GB fast / zfs 6TB
web swh-web 1 32GB regular 100GB
---------------- ---------------------- --------- ----- -------------- --------------
graph replayer swh-storage 32 4GB regular 10GB
content replayer swh-obstorage-replayer 32 4GB regular 10GB
replayer redis 1 8GB regular 100GB
---------------- ---------------------- --------- ----- -------------- --------------
vault swh-vault 1 4GB regular 10GB
vault worker swh-vault 1 16GB fast 1TB
vault rabbitmq 1 8GB regular 10GB
================ ====================== ========= ===== ============== ==============
Storage backend
^^^^^^^^^^^^^^^
.. tabbed:: Postgresql
================ ====================== ========= ===== ============== ==============
SWH Service Tool Instances RAM Storage Type Storage Volume
================ ====================== ========= ===== ============== ==============
storage postgresql 1 512GB fast+zfs (lz4) 40TB
================ ====================== ========= ===== ============== ==============
.. tabbed:: Cassandra (min.)*
================ ====================== ========= ===== ============== ==============
SWH Service Tool Instances RAM Storage Type Storage Volume
================ ====================== ========= ===== ============== ==============
storage cassandra 3 32GB fast 30TB
================ ====================== ========= ===== ============== ==============
.. tabbed:: Cassandra (typ.)*
================ ====================== ========= ===== ============== ==============
SWH Service Tool Instances RAM Storage Type Storage Volume
================ ====================== ========= ===== ============== ==============
storage cassandra 6+ 32GB fast 20TB
================ ====================== ========= ===== ============== ==============
Objstorage backend
^^^^^^^^^^^^^^^^^^
.. tabbed:: FS
================ ====================== ========= ===== ============== ==============
SWH Service Tool Instances RAM Storage Type Storage Volume
================ ====================== ========= ===== ============== ==============
objstorage swh-objstorage 1 [#f1]_ 512GB zfs (with lz4) 1PB
================ ====================== ========= ===== ============== ==============
.. tabbed:: Winery - Ceph*
vlorentzUnsubmitted
Not Done
Inline Actions

Should probably add a note that it can be split between multiple nodes.

vlorentz: Should probably add a note that it can be split between multiple nodes.
================ ====================== ========= ===== ============== ==============
SWH Service Tool Instances RAM Storage Type Storage Volume
================ ====================== ========= ===== ============== ==============
objstorage swh-objstorage 2 [#f2]_ 32GB standard 100GB
winery-db postgresql 2 [#f2]_ 512GB fast 10TB
ceph-mon ceph 3 4GB fast 60GB
ceph-osd ceph 12+ 4GB mix fast+HDD 1PB (total)
================ ====================== ========= ===== ============== ==============
.. tabbed:: Seaweedfs*
vlorentzUnsubmitted
Done
Inline Actions
winery-db postgresql 3 ?? ?? ??
- ceph-monitor ceph 3 ?? ?? ??
- ceph-osb ceph 12+ ?? ?? ??
+ ceph-mon ceph 3 ?? ?? ??
+ ceph-osd ceph 12+ ?? ?? ??
================ ====================== ===== ===== ============== ==============
vlorentz: * ceph-mon: 4GB RAM should be enough; 60GB of disk according to https://docs.ceph.
================ ====================== ========= ===== ============== ==============
SWH Service Tool Instances RAM Storage Type Storage Volume
================ ====================== ========= ===== ============== ==============
objstorage swh-objstorage 3 32GB standard 100GB
seaweed LB nginx 1 32GB fast 100GB
seaweed-master seaweedfs 3 8GB standard 10GB
seaweed-filer seaweedfs 3 32GB fast 1TB
seaweed-volume seaweedfs 3+ 32GB standard 1PB (total)
================ ====================== ========= ===== ============== ==============
.. rubric:: Notes
.. [#f1] An swh-objstorage using :py:mod:`simple filesystem
<swh.objstorage.backends.pathslicing>` as backend can actually be
split on several machines using the
:py:mod:`swh.objstorage.multiplexer` backend.
.. [#f2] The swh-objstorage RPC service and the index database can be hosted on
the same machine.
View Options

swh/deposit/tests/api/test_get_file.py

  • This file was added.
.. _planning-a-mirror:
Hosting a mirror
================
This section present and discuss the technical requirements needed to host a
|SWH| mirror.
There are many different options to host a mirror, but there are common overall
requirements that needs to be fulfilled.
Namely, hosting a mirror requires:
- a dedicated infrastructure with enough compute (s/computing) power and storage
- enough network bandwidth (both ingress and egress)
- good IT tooling (supervision, alerting).
The mirror operator is not required to run the Software Heritage `full software
stack <https://docs.softwareheritage.org/devel>`_, however it is possible to
use it.
.. Warning::
Volumes given in this section are estimations and numbers from **January
2022**.
The global raw hardware requirements are:
- a database system for the main storage of the archive (the graph structure);
the current volume is about 17TB, with an increase rate of about
280GB/month,
- an object storage system for the objects (archived software source code
files); the current volume is about 800TB with an increase rate of
about 21TB/month,
- an elasticsearch engine; the current main index is about 180M entries
(origins) for an index size of 360GB; the increase rate is about 2M
entries/month,
- a web/application server for the main web application and public API,
- a few compute nodes for the application services.
A mirror should provision machines or cloud-based resources with these numbers
in mind. This should include the usual robustness margins (RAID-like storage,
replication, backup etc.).
ardumontUnsubmitted
Done
Inline Actions
- a few compute nodes for the application services.
A mirror should provision machines or cloud-based resources with these numbers
- in mind, and should include usual the robustness margin (RAID-like storage,
+ in mind. This should include the usual robustness margin (RAID-like storage,
replication, backup etc.).
General hardware requirements
ardumont:
General hardware requirements
-----------------------------
When deploying a mirror based on the Software Heritage software stack, one will
need:
Core services
^^^^^^^^^^^^^
- a database for the storage; this can be either a
`Postgresql <https://postgresql.org>`_ database (single machine)
or a `Cassandra <https://cassandra.apache.org>`_ cluster (at least 3 nodes),
- an object storage system; this can be any
:py:mod:`supported backend <swh.objstorage.backends>`
-- a public cloud-based obstorage (e.g. s3), any private supported object storage,
an ad-hoc filesystem storage system, etc.
- an `elasticsearch <https://www.elastic.co>`_ instance,
- a few nodes for backend applications
(:py:mod:`swh-storage <swh.storage>`, :py:mod:`swh-objstorage <swh.objstorage>`)
- the web frontend (:py:mod:`swh-web <swh.web>`)
serving the main web app and the `public
API <https://docs.softwareheritage.org/devel/swh-web/uri-scheme-api.html>`_)
Replaying services
^^^^^^^^^^^^^^^^^^
- `graph
replayers <https://docs.softwareheritage.org/devel/swh-storage/cli.html#swh-storage-replay>`_
as mirroring workers (increase parallelism to increase speed)
- `content
ardumontUnsubmitted
Done
Inline Actions
^^^^^^^^^^^^^^^^^^
- - a fair amount of `graph
+ - `graph
replayers <https://docs.softwareheritage.org/devel/swh-storage/cli.html#swh-storage-replay>`_
- mirroring workers
+ as mirroring workers (increase parallelism to increase speed)
- a fair amount of `content

or something

ardumont: or something
replayers <https://docs.softwareheritage.org/devel/swh-objstorage-replayer/cli.html>`_
as mirroring workers (id.)
ardumontUnsubmitted
Done
Inline Actions

^ same remark

ardumont: ^ same remark
Vault service
^^^^^^^^^^^^^
- a node for the :ref:`swh-vault <swh-vault>` backend service,
- a node for the :ref:`swh-vault <swh-vault>` worker service
Sizing a mirror infrastructure
vlorentzUnsubmitted
Done
Inline Actions
^^^^^^^^^^^^^
- - a node for the `swh-vault`_ backend service,
- - a node for the `swh-vault`_ worker service
-
- .. _`swh-vault`: https://docs.softwareheritage.org/devel/swh-vault
+ - a node for the :ref:`swh-vault <swh-vault>` backend service,
+ - a node for the :ref:`swh-vault <swh-vault>` worker service
Sizing a mirror infrastructure
vlorentz:
------------------------------
.. Note:: solutions with a star (*) in the tables below are still under test or
validation.
Common components
^^^^^^^^^^^^^^^^^
vlorentzUnsubmitted
Not Done
Inline Actions

nice

vlorentz: nice
================ ====================== ========= ===== ============== ==============
SWH Service Tool Instances RAM Storage Type Storage Volume
================ ====================== ========= ===== ============== ==============
storage swh-storage 16 16GB regular 10GB
search elasticsearch 3 32GB fast / zfs 6TB
web swh-web 1 32GB regular 100GB
---------------- ---------------------- --------- ----- -------------- --------------
graph replayer swh-storage 32 4GB regular 10GB
content replayer swh-obstorage-replayer 32 4GB regular 10GB
replayer redis 1 8GB regular 100GB
---------------- ---------------------- --------- ----- -------------- --------------
vault swh-vault 1 4GB regular 10GB
vault worker swh-vault 1 16GB fast 1TB
vault rabbitmq 1 8GB regular 10GB
================ ====================== ========= ===== ============== ==============
Storage backend
^^^^^^^^^^^^^^^
.. tabbed:: Postgresql
================ ====================== ========= ===== ============== ==============
SWH Service Tool Instances RAM Storage Type Storage Volume
================ ====================== ========= ===== ============== ==============
storage postgresql 1 512GB fast+zfs (lz4) 40TB
================ ====================== ========= ===== ============== ==============
.. tabbed:: Cassandra (min.)*
================ ====================== ========= ===== ============== ==============
SWH Service Tool Instances RAM Storage Type Storage Volume
================ ====================== ========= ===== ============== ==============
storage cassandra 3 32GB fast 30TB
================ ====================== ========= ===== ============== ==============
.. tabbed:: Cassandra (typ.)*
================ ====================== ========= ===== ============== ==============
SWH Service Tool Instances RAM Storage Type Storage Volume
================ ====================== ========= ===== ============== ==============
storage cassandra 6+ 32GB fast 20TB
================ ====================== ========= ===== ============== ==============
Objstorage backend
^^^^^^^^^^^^^^^^^^
.. tabbed:: FS
================ ====================== ========= ===== ============== ==============
SWH Service Tool Instances RAM Storage Type Storage Volume
================ ====================== ========= ===== ============== ==============
objstorage swh-objstorage 1 [#f1]_ 512GB zfs (with lz4) 1PB
================ ====================== ========= ===== ============== ==============
.. tabbed:: Winery - Ceph*
vlorentzUnsubmitted
Not Done
Inline Actions

Should probably add a note that it can be split between multiple nodes.

vlorentz: Should probably add a note that it can be split between multiple nodes.
================ ====================== ========= ===== ============== ==============
SWH Service Tool Instances RAM Storage Type Storage Volume
================ ====================== ========= ===== ============== ==============
objstorage swh-objstorage 2 [#f2]_ 32GB standard 100GB
winery-db postgresql 2 [#f2]_ 512GB fast 10TB
ceph-mon ceph 3 4GB fast 60GB
ceph-osd ceph 12+ 4GB mix fast+HDD 1PB (total)
================ ====================== ========= ===== ============== ==============
.. tabbed:: Seaweedfs*
vlorentzUnsubmitted
Done
Inline Actions
winery-db postgresql 3 ?? ?? ??
- ceph-monitor ceph 3 ?? ?? ??
- ceph-osb ceph 12+ ?? ?? ??
+ ceph-mon ceph 3 ?? ?? ??
+ ceph-osd ceph 12+ ?? ?? ??
================ ====================== ===== ===== ============== ==============
vlorentz: * ceph-mon: 4GB RAM should be enough; 60GB of disk according to https://docs.ceph.
================ ====================== ========= ===== ============== ==============
SWH Service Tool Instances RAM Storage Type Storage Volume
================ ====================== ========= ===== ============== ==============
objstorage swh-objstorage 3 32GB standard 100GB
seaweed LB nginx 1 32GB fast 100GB
seaweed-master seaweedfs 3 8GB standard 10GB
seaweed-filer seaweedfs 3 32GB fast 1TB
seaweed-volume seaweedfs 3+ 32GB standard 1PB (total)
================ ====================== ========= ===== ============== ==============
.. rubric:: Notes
.. [#f1] An swh-objstorage using :py:mod:`simple filesystem
<swh.objstorage.backends.pathslicing>` as backend can actually be
split on several machines using the
:py:mod:`swh.objstorage.multiplexer` backend.
.. [#f2] The swh-objstorage RPC service and the index database can be hosted on
the same machine.
About · Developer information · Legal