Page MenuHomeSoftware Heritage
Feed Advanced Search

Advanced Search
Use Results
Hide Query

Aug 18 2016

ardumont edited parent tasks for T538: Add origin_visit API entry points to permit creation/update, added: T534: Add completion information to softwareheritage.origin_visit table; removed: T533: Allow loaders to register partial state (meta task).
Aug 18 2016, 2:29 PM · Storage manager
ardumont created T538: Add origin_visit API entry points to permit creation/update.
Aug 18 2016, 2:24 PM · Storage manager
ardumont created T534: Add completion information to softwareheritage.origin_visit table.
Aug 18 2016, 1:54 PM · Storage manager

Aug 16 2016

olasd added a comment to T526: Add notifications support to swh.storage.

The SQL side of this is now deployed.

Aug 16 2016, 6:55 PM · Storage manager
olasd created T526: Add notifications support to swh.storage.
Aug 16 2016, 6:29 PM · Storage manager
olasd closed T525: Allow bulk-listing of objects by content-id as Wontfix.

That's not something we're going to use in the end.

Aug 16 2016, 12:35 PM · Storage manager

Aug 12 2016

qcampos removed a revision from T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time: D94: Cloud-based objstorage using Apache Libcloud.
Aug 12 2016, 12:35 PM · Storage manager
qcampos added a revision to T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time: D94: Cloud-based objstorage using Apache Libcloud.
Aug 12 2016, 12:35 PM · Storage manager

Aug 11 2016

olasd created T525: Allow bulk-listing of objects by content-id.
Aug 11 2016, 4:34 PM · Storage manager

Aug 9 2016

qcampos added a parent task for T494: swh-journal: archiver-client: Keep archiver table in sync with new contents: T424: swh-journal: persistent journal infrastructure to record additions to the swh-storage.
Aug 9 2016, 5:47 PM · Journal, Restricted Project, Storage manager

Aug 5 2016

qcampos added a revision to T454: object storage backend that can read from/write to S3: D94: Cloud-based objstorage using Apache Libcloud.
Aug 5 2016, 4:32 PM · Storage manager
qcampos added a revision to T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time: D93: Make the archiver use content_archive.num_present.
Aug 5 2016, 4:27 PM · Storage manager

Aug 4 2016

qcampos added a revision to T304: content integrity checker: D92: Make the content checker easy to extend and add an implementation.
Aug 4 2016, 4:38 PM · Storage manager

Aug 3 2016

qcampos added a comment to T454: object storage backend that can read from/write to S3.

I've made some investigations : S3 does have GET and PUT operations on blobs that are referenced by a name (could be sha1). This is sufficient for our ObjStorage needs.

Aug 3 2016, 5:21 PM · Storage manager
qcampos updated the task description for T523: Figure out what to do with corrupted copies detected by the archiver.
Aug 3 2016, 5:01 PM · Storage manager
olasd created T523: Figure out what to do with corrupted copies detected by the archiver.
Aug 3 2016, 3:31 PM · Storage manager
qcampos closed T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time as Resolved by committing rDSTOf76cd4d93861: Perform a complete rewriting of the archiver.
Aug 3 2016, 12:36 PM · Storage manager
qcampos closed T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time, a subtask of T401: Content archiver - Asynchronous version, as Resolved.
Aug 3 2016, 12:36 PM · Restricted Project, Storage manager

Aug 2 2016

qcampos removed a revision from T240: content archiver: D86: Complete refactoring of the archiver.
Aug 2 2016, 2:05 PM · Storage manager
qcampos added a revision to T240: content archiver: D86: Complete refactoring of the archiver.
Aug 2 2016, 1:13 PM · Storage manager

Aug 1 2016

zack removed a project from T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time: Restricted Project.
Aug 1 2016, 10:39 AM · Storage manager

Jul 29 2016

qcampos renamed T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time from Make archiver have a more symmetrical behavior with storages as potential sources & destinations at the same time to Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time.
Jul 29 2016, 4:40 PM · Storage manager
qcampos updated the task description for T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time.
Jul 29 2016, 4:40 PM · Storage manager
qcampos renamed T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time from Change archiver behavior to Make archiver have a more symmetrical behavior with storages as potential sources & destinations at the same time.
Jul 29 2016, 4:39 PM · Storage manager
qcampos added a revision to T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time: D89: Add the 'corrupted' state in the archive_status enum.
Jul 29 2016, 4:02 PM · Storage manager
qcampos added a revision to T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time: D88: Update the archiver specification.
Jul 29 2016, 3:33 PM · Storage manager
qcampos added a revision to T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time: D86: Complete refactoring of the archiver.
Jul 29 2016, 2:59 PM · Storage manager
qcampos closed T513: Add methods to easily instantiate ObjStorage of different types as Resolved.

Closed by D83 & D84

Jul 29 2016, 2:58 PM · Restricted Project, Storage manager
qcampos closed T513: Add methods to easily instantiate ObjStorage of different types, a subtask of T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time, as Resolved.
Jul 29 2016, 2:58 PM · Storage manager
zack removed a subtask for T412: Bootstrap archiver's database: Unknown Object (Maniphest Task).
Jul 29 2016, 12:04 AM · Storage manager

Jul 28 2016

jbertran renamed T518: Batch-search content according to sha1, sha256 and sha1_git at once from Batch-search files according to sha1, sha256 and sha1_git at once to Batch-search content according to sha1, sha256 and sha1_git at once.
Jul 28 2016, 11:14 AM · Storage manager
jbertran added a parent task for T518: Batch-search content according to sha1, sha256 and sha1_git at once: T517: Batch-search files according to sha1, sha256 AND sha1_git in all web-ui layers.
Jul 28 2016, 11:14 AM · Storage manager
jbertran created T518: Batch-search content according to sha1, sha256 and sha1_git at once.
Jul 28 2016, 11:13 AM · Storage manager

Jul 27 2016

qcampos added a revision to T513: Add methods to easily instantiate ObjStorage of different types: D84: Update tests to follow RemoteObjStorage api fixes.
Jul 27 2016, 4:08 PM · Restricted Project, Storage manager
qcampos added a revision to T513: Add methods to easily instantiate ObjStorage of different types: D83: Add new functionnality to the ObjStorage and align remote API to the local one.
Jul 27 2016, 3:45 PM · Restricted Project, Storage manager
qcampos created T513: Add methods to easily instantiate ObjStorage of different types.
Jul 27 2016, 1:15 PM · Restricted Project, Storage manager
qcampos created T512: Make archiver have a more symmetrical behavior treating storages as potential sources & destinations at the same time.
Jul 27 2016, 1:11 PM · Storage manager
qcampos removed a subtask for T240: content archiver: T406: Make the archiver asynchronous mode optional.
Jul 27 2016, 1:09 PM · Storage manager
qcampos added a subtask for T401: Content archiver - Asynchronous version: T406: Make the archiver asynchronous mode optional.
Jul 27 2016, 1:09 PM · Restricted Project, Storage manager
qcampos edited parent tasks for T406: Make the archiver asynchronous mode optional, added: T401: Content archiver - Asynchronous version; removed: T240: content archiver.
Jul 27 2016, 1:09 PM · Restricted Project, Storage manager
qcampos edited parent tasks for T403: Improve content archiver testing coverage, added: T401: Content archiver - Asynchronous version; removed: T240: content archiver.
Jul 27 2016, 1:08 PM · Restricted Project, Storage manager
qcampos removed a subtask for T240: content archiver: T403: Improve content archiver testing coverage.
Jul 27 2016, 1:08 PM · Storage manager
qcampos added a subtask for T401: Content archiver - Asynchronous version: T403: Improve content archiver testing coverage.
Jul 27 2016, 1:08 PM · Restricted Project, Storage manager
qcampos removed a parent task for T402: Dev: Improve starting server routine: T240: content archiver.
Jul 27 2016, 1:08 PM · Restricted Project, Storage manager
qcampos removed a subtask for T240: content archiver: T402: Dev: Improve starting server routine.
Jul 27 2016, 1:08 PM · Storage manager
qcampos edited parent tasks for T405: Deploying objstorage API server, added: T381: HTTP client/server version of swh.storage.objstorage; removed: T240: content archiver.
Jul 27 2016, 1:07 PM · System administration, Language-Puppet, Storage manager
qcampos removed a subtask for T240: content archiver: T405: Deploying objstorage API server.
Jul 27 2016, 1:07 PM · Storage manager
qcampos added a subtask for T381: HTTP client/server version of swh.storage.objstorage: T405: Deploying objstorage API server.
Jul 27 2016, 1:07 PM · Storage manager
qcampos edited parent tasks for T404: Debian packaging update, added: T401: Content archiver - Asynchronous version; removed: T240: content archiver.
Jul 27 2016, 1:07 PM · Restricted Project, Storage manager
qcampos removed a subtask for T240: content archiver: T404: Debian packaging update.
Jul 27 2016, 1:07 PM · Storage manager
qcampos added a subtask for T401: Content archiver - Asynchronous version: T404: Debian packaging update.
Jul 27 2016, 1:07 PM · Restricted Project, Storage manager
qcampos added a parent task for T402: Dev: Improve starting server routine: T381: HTTP client/server version of swh.storage.objstorage.
Jul 27 2016, 1:06 PM · Restricted Project, Storage manager
qcampos added a subtask for T381: HTTP client/server version of swh.storage.objstorage: T402: Dev: Improve starting server routine.
Jul 27 2016, 1:06 PM · Storage manager

Jul 26 2016

qcampos added a revision to T240: content archiver: D81: Refactor and optimize the archiver.
Jul 26 2016, 4:42 PM · Storage manager
olasd added a comment to T482: First swh-storage-archiver run to catch up uffizi.

The hdd database is horribly, horribly slow. Currently dumping and reimporting it to SSD-based storage...

Jul 26 2016, 12:27 PM · Storage manager

Jul 25 2016

olasd changed the status of T482: First swh-storage-archiver run to catch up uffizi, a subtask of T240: content archiver, from Open to Work in Progress.
Jul 25 2016, 12:53 PM · Storage manager
olasd changed the status of T482: First swh-storage-archiver run to catch up uffizi from Open to Work in Progress.

The archiver director and a worker on uffizi are now running.

Jul 25 2016, 12:53 PM · Storage manager
olasd closed T412: Bootstrap archiver's database as Resolved.

The primary key has been added and the database should now match the schema.

Jul 25 2016, 12:53 PM · Storage manager
olasd closed T412: Bootstrap archiver's database, a subtask of T482: First swh-storage-archiver run to catch up uffizi, as Resolved.
Jul 25 2016, 12:53 PM · Storage manager

Jul 23 2016

ardumont added a comment to T412: Bootstrap archiver's database.

status:

  • contents present in uffizi and missing in banco added in softwareheritage-archiver.content_archive
  • index currently being built on content_archive (still).
Jul 23 2016, 5:37 AM · Storage manager

Jul 22 2016

qcampos removed a subtask for T240: content archiver: T401: Content archiver - Asynchronous version.
Jul 22 2016, 12:30 PM · Storage manager
qcampos edited parent tasks for T401: Content archiver - Asynchronous version, added: T400: Content archiver synchronous version; removed: T240: content archiver.
Jul 22 2016, 12:30 PM · Restricted Project, Storage manager
qcampos added a subtask for T400: Content archiver synchronous version: T401: Content archiver - Asynchronous version.
Jul 22 2016, 12:30 PM · Restricted Project, Storage manager
qcampos removed a parent task for T412: Bootstrap archiver's database: T240: content archiver.
Jul 22 2016, 12:29 PM · Storage manager
qcampos removed a subtask for T240: content archiver: T412: Bootstrap archiver's database.
Jul 22 2016, 12:29 PM · Storage manager

Jul 21 2016

ardumont reopened T412: Bootstrap archiver's database, a subtask of T482: First swh-storage-archiver run to catch up uffizi, as Work in Progress.
Jul 21 2016, 1:10 PM · Storage manager
ardumont reopened T412: Bootstrap archiver's database as "Work in Progress".

After some discussion, we need to rework the schema some more.

Jul 21 2016, 1:10 PM · Storage manager
ardumont reopened T412: Bootstrap archiver's database, a subtask of T240: content archiver, as Work in Progress.
Jul 21 2016, 1:10 PM · Storage manager
ardumont closed T486: DB storage cleanup as Resolved.
Jul 21 2016, 11:19 AM · Storage manager
ardumont closed T489: Reduce current log configuration of various services as Resolved.
Jul 21 2016, 11:19 AM · Storage manager
ardumont closed T489: Reduce current log configuration of various services, a subtask of T486: DB storage cleanup, as Resolved.
Jul 21 2016, 11:19 AM · Storage manager
ardumont changed the status of T485: Synchronously catchup backup from uffizi to banco from Resolved to Invalid.
Jul 21 2016, 11:12 AM · Object storage, Storage manager
ardumont changed the status of T485: Synchronously catchup backup from uffizi to banco, a subtask of T240: content archiver, from Resolved to Invalid.
Jul 21 2016, 11:12 AM · Storage manager
ardumont changed the status of T485: Synchronously catchup backup from uffizi to banco, a subtask of T240: content archiver, from Invalid to Resolved.
Jul 21 2016, 11:10 AM · Storage manager
ardumont changed the status of T485: Synchronously catchup backup from uffizi to banco from Invalid to Resolved by committing rDOBJS84bc789758a6: Revert "Client to backup contents to an objstorage server instance".
Jul 21 2016, 11:10 AM · Object storage, Storage manager

Jul 20 2016

ardumont updated subscribers of T482: First swh-storage-archiver run to catch up uffizi.

@qcampos Looks like we ready to rock'n roll ^^

Jul 20 2016, 7:49 PM · Storage manager
ardumont closed T485: Synchronously catchup backup from uffizi to banco as Invalid.
Jul 20 2016, 7:48 PM · Object storage, Storage manager
ardumont closed T485: Synchronously catchup backup from uffizi to banco, a subtask of T240: content archiver, as Invalid.
Jul 20 2016, 7:48 PM · Storage manager
ardumont closed T412: Bootstrap archiver's database, a subtask of T240: content archiver, as Resolved.
Jul 20 2016, 7:47 PM · Storage manager
ardumont closed T412: Bootstrap archiver's database, a subtask of T482: First swh-storage-archiver run to catch up uffizi, as Resolved.
Jul 20 2016, 7:47 PM · Storage manager
ardumont closed T412: Bootstrap archiver's database as Resolved.
Jul 20 2016, 7:47 PM · Storage manager
ardumont added a comment to T412: Bootstrap archiver's database.

Sum up:

Jul 20 2016, 7:47 PM · Storage manager
ardumont added a comment to T412: Bootstrap archiver's database.

And we are done

Jul 20 2016, 7:45 PM · Storage manager
ardumont added a comment to T481: Deploying archiver on uffizi.

After T499 and T502, archiver got redeployed.

Jul 20 2016, 7:33 PM · Storage manager
ardumont added a comment to T412: Bootstrap archiver's database.

Primary key done

Jul 20 2016, 7:31 PM · Storage manager
ardumont updated the task description for T502: Update archiver's profile manifest.
Jul 20 2016, 7:22 PM · Storage manager
ardumont added a comment to T502: Update archiver's profile manifest.

@qcampos FYI ^^

Jul 20 2016, 7:22 PM · Storage manager
ardumont closed T502: Update archiver's profile manifest as Resolved by committing rSPPROFba98d23ed6b3: Update archiver.ini template according to latest development.
Jul 20 2016, 7:21 PM · Storage manager
ardumont closed T502: Update archiver's profile manifest, a subtask of T499: Change the swh.storage of the archiver to a swh.objstorage, as Resolved.
Jul 20 2016, 7:21 PM · Storage manager
ardumont closed T499: Change the swh.storage of the archiver to a swh.objstorage as Resolved by committing rDSTOc779f05b2fcf: Fix - Remove unused parameter.
Jul 20 2016, 7:04 PM · Storage manager
ardumont closed T499: Change the swh.storage of the archiver to a swh.objstorage, a subtask of T240: content archiver, as Resolved.
Jul 20 2016, 7:04 PM · Storage manager
qcampos added a revision to T499: Change the swh.storage of the archiver to a swh.objstorage: D78: archiver: change the master's storage to an object storage.
Jul 20 2016, 5:03 PM · Storage manager
ardumont added a comment to T482: First swh-storage-archiver run to catch up uffizi.

uffizi ready to run some archive-worker tasks.

Jul 20 2016, 4:37 PM · Storage manager
ardumont renamed T482: First swh-storage-archiver run to catch up uffizi from First swh-storage-archiver synchroneous mode to catch up uffizi to First swh-storage-archiver run to catch up uffizi.
Jul 20 2016, 3:53 PM · Storage manager
ardumont created T502: Update archiver's profile manifest.
Jul 20 2016, 3:06 PM · Storage manager
qcampos created T499: Change the swh.storage of the archiver to a swh.objstorage.
Jul 20 2016, 3:02 PM · Storage manager
ardumont added a comment to T412: Bootstrap archiver's database.

And it's done.
It took around 45min, awesome.

Jul 20 2016, 12:35 PM · Storage manager
ardumont added a comment to T412: Bootstrap archiver's database.

Ok so, status, faster but still too slow.
One third done in ~24h or so.

Jul 20 2016, 11:55 AM · Storage manager
qcampos renamed T494: swh-journal: archiver-client: Keep archiver table in sync with new contents from Content_archiver update to Add a way to update content_archive table when a new content is added.
Jul 20 2016, 11:52 AM · Journal, Restricted Project, Storage manager
qcampos renamed T494: swh-journal: archiver-client: Keep archiver table in sync with new contents from Improve storage.content_add function with a way to notify the archival db that a new content have been added to Content_archiver update.
Jul 20 2016, 11:51 AM · Journal, Restricted Project, Storage manager
First
Prev
Next
About · Developer information · Legal