Page MenuHomeSoftware Heritage
Feed Advanced Search

Aug 12 2016

qcampos closed D95: Improve the ObjStorage inheritance tree and its documentation by committing rDOBJSd53c6b90f34c: Reference base class in implementations docstrings.
Aug 12 2016, 2:01 PM
qcampos updated the diff for D93: Make the archiver use content_archive.num_present.

Merge with origin

Aug 12 2016, 1:47 PM
qcampos added inline comments to D95: Improve the ObjStorage inheritance tree and its documentation.
Aug 12 2016, 1:45 PM
qcampos updated the diff for D95: Improve the ObjStorage inheritance tree and its documentation.

Use hashutil.hashdata for hashcode computation in ObjStorage

Aug 12 2016, 1:42 PM
qcampos committed rDOBJS553c299916e4: Add first example of cloud object storages (authored by qcampos).
Add first example of cloud object storages
Aug 12 2016, 1:35 PM
qcampos committed rDOBJSbd574b6982f2: Cloud-based objstorage using Apache Libcloud (authored by qcampos).
Cloud-based objstorage using Apache Libcloud
Aug 12 2016, 1:35 PM
qcampos closed D94: Cloud-based objstorage using Apache Libcloud by committing rDOBJSbd574b6982f2: Cloud-based objstorage using Apache Libcloud.
Aug 12 2016, 1:35 PM
qcampos updated the diff for D93: Make the archiver use content_archive.num_present.

Rename previous_content to last_content.

Aug 12 2016, 1:34 PM
qcampos added inline comments to D95: Improve the ObjStorage inheritance tree and its documentation.
Aug 12 2016, 12:40 PM
qcampos updated the diff for D94: Cloud-based objstorage using Apache Libcloud.

See previous diff; I made the update from the wrong branch.

Aug 12 2016, 12:36 PM
qcampos removed a task from D94: Cloud-based objstorage using Apache Libcloud: T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time.
Aug 12 2016, 12:35 PM
qcampos removed a revision from T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time: D94: Cloud-based objstorage using Apache Libcloud.
Aug 12 2016, 12:35 PM · Storage manager
qcampos updated the diff for D94: Cloud-based objstorage using Apache Libcloud.

No changes, but current diff needs to be updated as the parent commit as changed due to a rebase in top of previously pushed diff.

Aug 12 2016, 12:35 PM
qcampos added a revision to T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time: D94: Cloud-based objstorage using Apache Libcloud.
Aug 12 2016, 12:35 PM · Storage manager
qcampos committed rDOBJS9850ab415559: Add a new checker implementation (authored by qcampos).
Add a new checker implementation
Aug 12 2016, 12:32 PM
qcampos committed rDOBJSe8b541b647d2: Update tests to keep up-to-date with the class (authored by qcampos).
Update tests to keep up-to-date with the class
Aug 12 2016, 12:32 PM
qcampos committed rDOBJSfd075db3d93f: Make the content integrity checker more extendable (authored by qcampos).
Make the content integrity checker more extendable
Aug 12 2016, 12:32 PM
qcampos committed rDOBJScc35ceaf3a3d: Remove a TODO forgotten in D91 (authored by qcampos).
Remove a TODO forgotten in D91
Aug 12 2016, 12:32 PM
qcampos closed D92: Make the content checker easy to extend and add an implementation by committing rDOBJScc35ceaf3a3d: Remove a TODO forgotten in D91.
Aug 12 2016, 12:32 PM

Aug 10 2016

qcampos added inline comments to D93: Make the archiver use content_archive.num_present.
Aug 10 2016, 2:32 PM
qcampos updated the diff for D93: Make the archiver use content_archive.num_present.

Perform some minor corrections.
There is still some work to do about the prototypes.

Aug 10 2016, 2:31 PM
qcampos updated the diff for D95: Improve the ObjStorage inheritance tree and its documentation.

Remove docstring in subclasses

Aug 10 2016, 2:17 PM
qcampos updated the diff for D94: Cloud-based objstorage using Apache Libcloud.

Remove pointless docstrings when abc's one can be used

Aug 10 2016, 1:59 PM
qcampos updated the diff for D92: Make the content checker easy to extend and add an implementation.

Make the BaseContentChecker class an Abtract Base Class

Aug 10 2016, 1:49 PM

Aug 9 2016

qcampos added a subtask for T424: swh-journal: persistent journal infrastructure to record additions to the swh-storage: T494: swh-journal: archiver-client: Keep archiver table in sync with new contents.
Aug 9 2016, 5:47 PM · Journal
qcampos added a parent task for T494: swh-journal: archiver-client: Keep archiver table in sync with new contents: T424: swh-journal: persistent journal infrastructure to record additions to the swh-storage.
Aug 9 2016, 5:47 PM · Journal, Restricted Project, Storage manager
qcampos updated D95: Improve the ObjStorage inheritance tree and its documentation.
Aug 9 2016, 5:31 PM
qcampos updated D95: Improve the ObjStorage inheritance tree and its documentation.
Aug 9 2016, 5:31 PM
qcampos updated D95: Improve the ObjStorage inheritance tree and its documentation.
Aug 9 2016, 5:31 PM
qcampos updated the diff for D95: Improve the ObjStorage inheritance tree and its documentation.
  • Factorize hash computation into objstorage
  • Rename an argument that didn't follow the API
Aug 9 2016, 5:30 PM
qcampos retitled D95: Improve the ObjStorage inheritance tree and its documentation from to Improve the ObjStorage inheritance tree and its documentation.
Aug 9 2016, 4:02 PM
qcampos updated the diff for D94: Cloud-based objstorage using Apache Libcloud.

Correct some docstring and use ABC for the base class.

Aug 9 2016, 2:59 PM
qcampos added inline comments to D94: Cloud-based objstorage using Apache Libcloud.
Aug 9 2016, 1:36 PM
qcampos added inline comments to D94: Cloud-based objstorage using Apache Libcloud.
Aug 9 2016, 1:24 PM

Aug 5 2016

qcampos added a task to D94: Cloud-based objstorage using Apache Libcloud: T454: object storage backend that can read from/write to S3.
Aug 5 2016, 4:32 PM
qcampos added a revision to T454: object storage backend that can read from/write to S3: D94: Cloud-based objstorage using Apache Libcloud.
Aug 5 2016, 4:32 PM · Storage manager
qcampos added a task to D93: Make the archiver use content_archive.num_present: T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time.
Aug 5 2016, 4:27 PM
qcampos added a revision to T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time: D93: Make the archiver use content_archive.num_present.
Aug 5 2016, 4:27 PM · Storage manager
qcampos updated the diff for D92: Make the content checker easy to extend and add an implementation.

Correct some mistakes in docstrings

Aug 5 2016, 12:08 PM

Aug 4 2016

qcampos updated the diff for D92: Make the content checker easy to extend and add an implementation.

Remove a forgotten TODO

Aug 4 2016, 4:42 PM
qcampos updated D92: Make the content checker easy to extend and add an implementation.
Aug 4 2016, 4:40 PM
qcampos retitled D92: Make the content checker easy to extend and add an implementation from Make the content checker extandable and add an implementation to Make the content checker easy to extend and add an implementation.
Aug 4 2016, 4:40 PM
qcampos committed rDOBJS38d4238e391c: Remove direct exposition of PathSlicingObjStorage (authored by qcampos).
Remove direct exposition of PathSlicingObjStorage
Aug 4 2016, 4:39 PM
qcampos closed D91: Remove direct exposition of PathSlicingObjStorage by committing rDOBJS38d4238e391c: Remove direct exposition of PathSlicingObjStorage.
Aug 4 2016, 4:39 PM
qcampos added a task to D92: Make the content checker easy to extend and add an implementation: T304: content integrity checker.
Aug 4 2016, 4:38 PM
qcampos added a revision to T304: content integrity checker: D92: Make the content checker easy to extend and add an implementation.
Aug 4 2016, 4:38 PM · Storage manager
qcampos retitled D91: Remove direct exposition of PathSlicingObjStorage from to Remove direct exposition of PathSlicingObjStorage.
Aug 4 2016, 4:31 PM

Aug 3 2016

qcampos added a comment to T454: object storage backend that can read from/write to S3.

I've made some investigations : S3 does have GET and PUT operations on blobs that are referenced by a name (could be sha1). This is sufficient for our ObjStorage needs.

Aug 3 2016, 5:21 PM · Storage manager
qcampos updated the task description for T523: Figure out what to do with corrupted copies detected by the archiver.
Aug 3 2016, 5:01 PM · Storage manager
qcampos committed rDSTOc73941c4d5c9: Remove cli arguments for the archiver launching (authored by qcampos).
Remove cli arguments for the archiver launching
Aug 3 2016, 12:36 PM
qcampos committed rDSTO801ff72244e3: archiver: correct the worker when handling ongoing status (authored by qcampos).
archiver: correct the worker when handling ongoing status
Aug 3 2016, 12:36 PM
qcampos committed rDSTOed9af69596f1: Remove outdated manual test (authored by qcampos).
Remove outdated manual test
Aug 3 2016, 12:36 PM
qcampos committed rDSTOf79a679d7c29: Correct config override in the archiver's test (authored by qcampos).
Correct config override in the archiver's test
Aug 3 2016, 12:36 PM
qcampos committed rDSTOf76cd4d93861: Perform a complete rewriting of the archiver (authored by qcampos).
Perform a complete rewriting of the archiver
Aug 3 2016, 12:36 PM
qcampos committed rDSTO287da80f7c9d: archiver.worker: Remove additional config (authored by qcampos).
archiver.worker: Remove additional config
Aug 3 2016, 12:36 PM
qcampos committed rDSTO212d9030b443: archiver: Director & Worker arguments are now parsed from conf file (authored by qcampos).
archiver: Director & Worker arguments are now parsed from conf file
Aug 3 2016, 12:36 PM
qcampos committed rDSTOdd49151734f6: Make the worker correclty catch corrupted contents (authored by qcampos).
Make the worker correclty catch corrupted contents
Aug 3 2016, 12:36 PM
qcampos closed T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time as Resolved by committing rDSTOf76cd4d93861: Perform a complete rewriting of the archiver.
Aug 3 2016, 12:36 PM · Storage manager
qcampos closed D86: Complete refactoring of the archiver by committing rDSTOf76cd4d93861: Perform a complete rewriting of the archiver.
Aug 3 2016, 12:36 PM
qcampos closed T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time, a subtask of T401: Content archiver - Asynchronous version, as Resolved.
Aug 3 2016, 12:36 PM · Restricted Project, Storage manager
qcampos updated the diff for D86: Complete refactoring of the archiver.

Submit identical diff due to merge with master

Aug 3 2016, 11:07 AM

Aug 2 2016

qcampos updated the diff for D86: Complete refactoring of the archiver.

Remove cli arguments & correct config override in tests

Aug 2 2016, 3:45 PM
qcampos removed a task from D86: Complete refactoring of the archiver: T240: content archiver.
Aug 2 2016, 2:05 PM
qcampos removed a revision from T240: content archiver: D86: Complete refactoring of the archiver.
Aug 2 2016, 2:05 PM · Storage manager
qcampos updated the diff for D86: Complete refactoring of the archiver.

Change the way Archiver's config is overriden in the tests

Aug 2 2016, 2:03 PM
qcampos updated the diff for D86: Complete refactoring of the archiver.

Remove outdated manual test for the archiver

Aug 2 2016, 1:13 PM
qcampos added a revision to T240: content archiver: D86: Complete refactoring of the archiver.
Aug 2 2016, 1:13 PM · Storage manager
qcampos updated the diff for D86: Complete refactoring of the archiver.
  • archiver.worker: Remove additional config
Aug 2 2016, 1:02 PM

Aug 1 2016

qcampos added inline comments to D86: Complete refactoring of the archiver.
Aug 1 2016, 5:36 PM
qcampos updated the diff for D86: Complete refactoring of the archiver.
  • The Director and Worker configuration are now parsed using swh.core.config.SWHConfig mixin class
  • A bug where ongoing contents could be used as sources have been corrected
Aug 1 2016, 5:36 PM
qcampos committed rDSTO591e886ada07: Update the archiver specification (authored by qcampos).
Update the archiver specification
Aug 1 2016, 5:24 PM
qcampos closed D88: Update the archiver specification by committing rDSTO591e886ada07: Update the archiver specification.
Aug 1 2016, 5:24 PM
qcampos committed rDSTO87386de54bc6: Add the 'corrupted' state in the archive_status enum (authored by qcampos).
Add the 'corrupted' state in the archive_status enum
Aug 1 2016, 5:23 PM
qcampos closed D89: Add the 'corrupted' state in the archive_status enum by committing rDSTO87386de54bc6: Add the 'corrupted' state in the archive_status enum.
Aug 1 2016, 5:23 PM
qcampos added inline comments to D88: Update the archiver specification.
Aug 1 2016, 2:48 PM
qcampos updated the diff for D88: Update the archiver specification.

Correct json format and move integrity check from copier to worker (previous position was a mistake)

Aug 1 2016, 2:48 PM
qcampos added inline comments to D88: Update the archiver specification.
Aug 1 2016, 2:20 PM
qcampos updated the diff for D88: Update the archiver specification.

Corrections of the specs and add the json schema description

Aug 1 2016, 2:20 PM
qcampos updated the diff for D89: Add the 'corrupted' state in the archive_status enum.

Add the update in sql/archiver/swh-archiver-schema.sql

Aug 1 2016, 12:38 PM

Jul 29 2016

qcampos updated the diff for D86: Complete refactoring of the archiver.

Correct an import

Jul 29 2016, 4:48 PM
qcampos updated the diff for D86: Complete refactoring of the archiver.

Make the worker correclty catch corrupted or missing contents that were referenced as 'present' into the db

Jul 29 2016, 4:44 PM
qcampos renamed T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time from Make archiver have a more symmetrical behavior with storages as potential sources & destinations at the same time to Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time.
Jul 29 2016, 4:40 PM · Storage manager
qcampos updated the task description for T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time.
Jul 29 2016, 4:40 PM · Storage manager
qcampos renamed T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time from Change archiver behavior to Make archiver have a more symmetrical behavior with storages as potential sources & destinations at the same time.
Jul 29 2016, 4:39 PM · Storage manager
qcampos added a task to D89: Add the 'corrupted' state in the archive_status enum: T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time.
Jul 29 2016, 4:02 PM
qcampos added a revision to T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time: D89: Add the 'corrupted' state in the archive_status enum.
Jul 29 2016, 4:02 PM · Storage manager
qcampos added a task to D88: Update the archiver specification: T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time.
Jul 29 2016, 3:33 PM
qcampos added a revision to T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time: D88: Update the archiver specification.
Jul 29 2016, 3:33 PM · Storage manager
qcampos added a comment to D86: Complete refactoring of the archiver.

Note that the current version does not yet change the status of a content when it is found to be corrupted.

Jul 29 2016, 3:32 PM
qcampos updated D86: Complete refactoring of the archiver.
Jul 29 2016, 3:01 PM
qcampos added a revision to T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time: D86: Complete refactoring of the archiver.
Jul 29 2016, 2:59 PM · Storage manager
qcampos added a task to D86: Complete refactoring of the archiver: T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time.
Jul 29 2016, 2:59 PM
qcampos closed T513: Add methods to easily instantiate ObjStorage of different types as Resolved.

Closed by D83 & D84

Jul 29 2016, 2:58 PM · Restricted Project, Storage manager
qcampos closed T513: Add methods to easily instantiate ObjStorage of different types, a subtask of T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time, as Resolved.
Jul 29 2016, 2:58 PM · Storage manager
qcampos committed rDSTO65bae82a774c: Update tests to follow RemoteObjStorage api fixes (authored by qcampos).
Update tests to follow RemoteObjStorage api fixes
Jul 29 2016, 2:57 PM
qcampos closed D84: Update tests to follow RemoteObjStorage api fixes by committing rDSTO65bae82a774c: Update tests to follow RemoteObjStorage api fixes.
Jul 29 2016, 2:57 PM

Jul 28 2016

qcampos committed rDOBJSb7032be4ee49: Align the RemoteObjStorage api to ObjStorage base (authored by qcampos).
Align the RemoteObjStorage api to ObjStorage base
Jul 28 2016, 3:52 PM
qcampos committed rDOBJS28f96c42d309: Update ObjStorages tests so they use the new initialization (authored by qcampos).
Update ObjStorages tests so they use the new initialization
Jul 28 2016, 3:52 PM
qcampos committed rDOBJS457f4aefdaa0: Add method to get contents in bulk in the base API (authored by qcampos).
Add method to get contents in bulk in the base API
Jul 28 2016, 3:52 PM
qcampos committed rDOBJS64d08a4446b0: Add an easy way to instantiate specific objstorage (authored by qcampos).
Add an easy way to instantiate specific objstorage
Jul 28 2016, 3:52 PM