diff --git a/docs/archiver-blueprint.md b/docs/archiver-blueprint.md --- a/docs/archiver-blueprint.md +++ b/docs/archiver-blueprint.md @@ -12,24 +12,32 @@ Requirements ------------ -* **Master/slave architecture** +* **Peer-to-peer topology** - There is 1 master copy and 1 or more slave copies of each object. A retention - policy specifies the minimum number of copies that are required to be "safe". + Every storage involved in the archival process can be used as a source or a + destination for the archival, depending on the blobs it contains. A + retention policy specifies the minimum number of copies that are required + to be "safe". + + Althoug the servers are totally equals the coordination of which content + should be copied and from where to where is centralized. * **Append-only archival** - The archiver treats master as read-only storage and slaves as append-only - storages. The archiver never deletes any object. If removals are needed, in - either master or slaves, they will be dealt with by other means. + The archiver treats involved storages as append-only storages. The archiver + never deletes any object. If removals are needed, they will be dealt with + by other means. * **Asynchronous archival.** Periodically (e.g., via cron), the archiver runs, produces a list of objects - that need to be copied from master to slaves, and starts copying them over. - Very likely, during any given archival run other new objects will be added to - master; it will be the responsibility of *future* archiver runs, and not the - current one, to copy new objects over. + that need to have more copies, and starts copying them over. The decision of + which storages are choosen to be sources and destinations are not up to the + storages themselves. + + Very likely, during any given archival run, other new objects will be added + to storages; it will be the responsibility of *future* archiver runs, and + not the current one, to copy new objects over if needed. * **Integrity at archival time.** @@ -40,7 +48,7 @@ reporting about the corruption will be emitted. Note that archival-time integrity checks are *not meant to replace periodic - integrity checks* on both master and slave copies. + integrity checks*. * **Parallel archival** @@ -56,7 +64,8 @@ information: * **status**: 3-state: *missing* (copy not present at destination), *ongoing* - (copy to destination ongoing), *present* (copy present at destination) + (copy to destination ongoing), *present* (copy present at destination), + *corrupted* (content detected as corrupted during an archival). * **mtime**: timestamp of last status change. This is either the destination archival time (when status=present), or the timestamp of the last archival request (status=ongoing); the timestamp is undefined when status=missing. @@ -86,22 +95,14 @@ At each execution the director: 1. for each object: retrieve its archival status -2. for each object that is in the master storage but has fewer copies than - those requested by the retention policy: - 1. if status=ongoing and mtime is not older than archival max age - then continue to next object - 2. check object integrity (e.g., with swh.storage.ObjStorage.check(obj_id)) - 3. mark object as needing archival -3. group objects in need of archival in batches of archival batch size +2. for each object that has fewer copies than those requested by the + retention policy: + 1. mark object as needing archival +3. group objects in need of archival in batches of `archival batch size` 4. for each batch: - 1. set status=ongoing and mtime=now() for each object in the batch - 2. spawn an archive worker on the whole batch (e.g., submitting the relevant + 1. spawn an archive worker on the whole batch (e.g., submitting the relevant celery task) -Note that if an archiver worker task takes a long time (t > archival max age) -to complete, it is possible for another task to be scheduled on the same batch, -or an overlapping one. - ### Archiver worker @@ -111,47 +112,61 @@ Runtime parameters: * objects to archive +* archival policies (retention & archival max age) At each execution a worker: -1. create empty map { destinations -> objects that need to be copied there } +1. for each object in the batch + 1. check that the object still need to be archived + (#present copies < retention policy) + 2. if an object has status=ongoing but the elapsed time from task submission + is less than the *archival max age*, it count as present, as we assume + that it will be copied in the futur. If the delay is elapsed, otherwise, + is count as a missing copy. 2. for each object to archive: 1. retrieve current archive status for all destinations - 2. update the map noting where the object needs to be copied -3. for each destination: - 1. look up in the map objects that need to be copied there - 2. copy all objects to destination using the copier - 3. set status=present and mtime=now() for each copied object + 2. create a map noting where the object is present and where it can be copied + 3. Randomly choose couples (source, destination), where destinations are all + differents, to make enough copies +3. for each (content, source, destination): + 1. Join the contents by key (source, destination) to have a map + {(source, destination) -> [contents]} +4. for each (source, destination) -> contents + 1. for each content in content, check its integrity on the source storage + * if the object if corrupted or missing + * update its status in the database + * remove it from the current contents list +5. start the copy of the batces by launching for each transfer tuple a copier + * if an error occured on one of the content that should have been valid, + consider the whole batch as a failure. +6. set status=present and mtime=now for each successfully copied object Note that: * In case multiple jobs where tasked to archive the same of overlapping - objects, step (2.2) might decide that some/all objects of this batch no - longer need to be archived to some/all destinations. + objects, step (1) might decide that some/all objects of this batch no + longer need to be archived. -* Due to parallelism, it is also possible that the same objects will be copied - over at the same time by multiple workers. +* Due to parallelism, it is possible that the same objects will be copied + over at the same time by multiple workers. Also, the same object could end + having more copies than the minimal number required. ### Archiver copier The copier is run on demand by archiver workers, to transfer file batches from -master to a given destination. +a given source to a given destination. -The copier transfers all files together with a single network connection. The -copying process is atomic at the file granularity (i.e., individual files might -be visible on the destination before *all* files have been transferred) and -ensures that *concurrent transfer of the same files by multiple copier -instances do not result in corrupted files*. Note that, due to this and the -fact that timestamps are updated by the director, all files copied in the same -batch will have the same mtime even though the actual file creation times on a -given destination might differ. +The copier transfers files one by one. The copying process is atomic at the file +granularity (i.e., individual files might be visible on the destination before +*all* files have been transferred) and ensures that *concurrent transfer of the +same files by multiple copier instances do not result in corrupted files*. Note +that, due to this and the fact that timestamps are updated by the worker, all +files copied in the same batch will have the same mtime even though the actual +file creation times on a given destination might differ. -As a first approximation, the copier can be implemented using rsync, but a -dedicated protocol can be devised later. In the case of rsync, one should use ---files-from to list the file to be copied. Rsync atomically renames files -one-by-one during transfer; so as long as --inplace is *not* used, concurrent -rsync of the same files should not be a problem. +The copier is implemented using the ObjStorage API for the sources and +destinations. DB structure @@ -159,9 +174,13 @@ Postgres SQL definitions for the archival status: - CREATE DOMAIN archive_id AS TEXT; + -- Those names are sample of archives server names + CREATE TYPE archive_id AS ENUM ( + 'uffizi', + 'banco' + ); - CREATE TABLE archives ( + CREATE TABLE archive ( id archive_id PRIMARY KEY, url TEXT ); @@ -169,13 +188,40 @@ CREATE TYPE archive_status AS ENUM ( 'missing', 'ongoing', - 'present' + 'present', + 'corrupted' ); CREATE TABLE content_archive ( - content_id sha1 REFERENCES content(sha1), - archive_id archive_id REFERENCES archives(id), - status archive_status, - mtime timestamptz, - PRIMARY KEY (content_id, archive_id) + content_id sha1 unique, + copies jsonb ); + + +Where the content_archive.copies field is of type jsonb and contains datas +about the storages that contains (or not) the content represented by the sha1 + + { + "$schema": "http://json-schema.org/schema#", + "title": "Copies data", + "description": "Data about the presence of a content into the storages", + "type": "object", + "Patternproperties": { + "^[a-zA-Z1-9]+$": { + "description": "archival status for the server named by key", + "type": "object", + "properties": { + "status": { + "description": "Archival status on this copy", + "type": "string", + "enum": ["missing", "ongoing", "present", "corrupted"] + }, + "mtime": { + "description": "Last time of the status update", + "type": "float" + } + } + } + }, + "additionalProperties": false + }