Page Menu
Home
Software Heritage
Search
Configure Global Search
Log In
Files
F9345097
archive-copies.rst
No One
Temporary
Actions
View File
Edit File
Delete File
View Transforms
Subscribe
Mute Notifications
Award Token
Flag For Later
Size
2 KB
Subscribers
None
archive-copies.rst
View Options
:orphan:
..
_archive-copies:
Archive copies
==============
..
_swh-storage-copies-layout:
..
figure
::
images/swh-archive-copies.svg
:width:
1024px
:align:
center
Layout of Software Heritage archive copies (click to zoom).
The Software Heritage archive exists in several copies, to minimize the risk of
losing archived source code artifacts. The layout of existing copies, their
relationships, as well as their geographical and administrative domains are
shown in the layout diagram above.
We recall that the archive is conceptually organized as a graph, and
specifically a Merkle DAG, see
:ref:
`data-model`
for more information.
Ingested source code artifacts land directly on the
**primary copy**
, which is
updated live and also used as reference for deduplication purposes. There,
different parts of the Merkle DAG as stored using different backend
technologies. The leaves of the graph, i.e.,
*content objects*
(or "blobs"),
are stored in a key-value object storage, using their SHA1 identifiers as keys
(see
:ref:
`persistent-identifiers`
). SHA1 collision avoidance is enforced by
the
:mod:
`swh.storage`
module. The
*rest of the graph*
is stored in a Postgres
database (see
:ref:
`sql-storage`
).
At the time of writing, the primary object storage contains about 5 billion
blobs with a median size of 3 KB---yes, that is *a lot of very small
files*---for a total compressed size of about 200 TB. The Postgres database
takes about 8 TB, half of which required by indexes. In terms of graph metrics,
the Merkle DAG has about 10 B nodes and 100 B edges.
The
**secondary copy**
is hosted on Microsoft Azure cloud, using its native
blob storage for the object storage and a large virtual machine to run a
Postgres instance there. The database is kept up-to-date w.r.t. the primary
copy using Postgres WAL replication. The object storage is kept up-to-date
using
:mod:
`swh.archiver`
.
Archive copies (as opposed to archive mirrors) are operated by the Software
Heritage Team at Inria. The primary archived copy is geographically located at
Rocquencourt, France; the secondary copy hosted in the Europe West region of
the Azure cloud.
File Metadata
Details
Attached
Mime Type
text/plain
Expires
Fri, Jul 4, 3:05 PM (6 d, 14 h ago)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
3298951
Attached To
rDSTO Storage manager
Event Timeline
Log In to Comment