Page MenuHomeSoftware Heritage

Object storageFolder
ActivePublic

Members

  • This project does not have any members.

Watchers

  • This project does not have any watchers.

Recent Activity

Mon, Aug 19

vlorentz updated the task description for T1954: Up-to-date objstorage mirror on S3.
Mon, Aug 19, 11:47 AM · System administration, Object storage
vlorentz updated the task description for T1954: Up-to-date objstorage mirror on S3.
Mon, Aug 19, 11:45 AM · System administration, Object storage
vlorentz triaged T1954: Up-to-date objstorage mirror on S3 as High priority.
Mon, Aug 19, 11:44 AM · System administration, Object storage

Jun 19 2019

olasd closed T1823: make DB/FS transactions nest properly as Resolved by committing rDOBJS67197802d5aa: pathslicing: Make sure data is flushed to disk before renaming the tempfile.
Jun 19 2019, 4:11 PM · Object storage, Storage manager
olasd added a revision to T1823: make DB/FS transactions nest properly: D1611: pathslicing: Make sure data is flushed to disk before renaming the tempfile.
Jun 19 2019, 1:46 PM · Object storage, Storage manager

Jun 18 2019

zack triaged T1823: make DB/FS transactions nest properly as High priority.
Jun 18 2019, 12:38 PM · Object storage, Storage manager

May 23 2019

vlorentz triaged T1736: Update the "Archive copies" documentation as Normal priority.
May 23 2019, 11:17 AM · Object storage, Development documentation

Mar 26 2019

olasd added a parent task for T1577: Compare/benchmark objstorage backends : T1608: Write all new objects to azure synchronously.
Mar 26 2019, 6:58 PM · Object storage

Mar 22 2019

seirl placed T805: objstorage: allow use of file-like objects for streaming methods up for grabs.
Mar 22 2019, 1:31 PM · Object storage
seirl triaged T1596: ObjStorage: investigate the per-request CPU overhead of aiohttp as Normal priority.
Mar 22 2019, 1:27 PM · Object storage
seirl triaged T1595: ObjStorage: compression pass-through as Normal priority.
Mar 22 2019, 1:24 PM · Object storage
seirl triaged T1594: ObjStorage: investigate HTTP pipelining issues across all the different backends as Normal priority.
Mar 22 2019, 11:35 AM · Object storage
seirl triaged T1593: ObjStorage: fast mass-retrieval of objects as Normal priority.
Mar 22 2019, 11:33 AM · Object storage

Mar 12 2019

vlorentz removed a project from T1447: Add support for slices when getting objects from the objstorage.: Easy hack.

I didn't know objects are compressed. That indeed makes the issue harder.

Mar 12 2019, 11:51 AM · Object storage
douardda triaged T1577: Compare/benchmark objstorage backends as Normal priority.
Mar 12 2019, 9:53 AM · Object storage
douardda added a comment to T1447: Add support for slices when getting objects from the objstorage..

Same here. Not that much an easy hack. And what is the real life use case that drive this feature request? YAGNI?

Mar 12 2019, 9:46 AM · Object storage

Feb 25 2019

ardumont closed T1533: Make sure api server uses explicit configurations as Resolved.
Feb 25 2019, 12:15 PM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault
ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Feb 25 2019, 12:15 PM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault

Feb 23 2019

ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Feb 23 2019, 1:58 AM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault
ardumont added a project to T1533: Make sure api server uses explicit configurations: Scheduling utilities.
Feb 23 2019, 12:43 AM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault
ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Feb 23 2019, 12:42 AM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault

Feb 22 2019

ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Feb 22 2019, 11:22 AM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault

Feb 21 2019

ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Feb 21 2019, 8:34 PM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault
ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Feb 21 2019, 2:11 PM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault

Feb 20 2019

ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Feb 20 2019, 7:11 PM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault

Feb 19 2019

ardumont renamed T1533: Make sure api server uses explicit configurations from Use explicit configuration for all modules to Make sure api server uses explicit configurations.
Feb 19 2019, 11:36 AM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault

Jan 31 2019

vlorentz added a project to T1487: Add a public API endpoint to retrieve a set of files with a given name: Easy hack.
Jan 31 2019, 1:29 PM · Easy hack, Storage manager, Object storage

Jan 21 2019

vlorentz added a project to T1487: Add a public API endpoint to retrieve a set of files with a given name: Storage manager.

The object storage doesn't have content names, so it cannot address this feature as stated.

Jan 21 2019, 4:55 PM · Easy hack, Storage manager, Object storage
zack updated subscribers of T1487: Add a public API endpoint to retrieve a set of files with a given name.

The object storage doesn't have content names, so it cannot address this feature as stated.

Jan 21 2019, 4:50 PM · Easy hack, Storage manager, Object storage
vlorentz added a comment to T1487: Add a public API endpoint to retrieve a set of files with a given name.

A crude script doing this: https://forge.softwareheritage.org/source/snippets/browse/master/vlorentz/get_metadata_files_examples.py

Jan 21 2019, 4:29 PM · Easy hack, Storage manager, Object storage
vlorentz triaged T1487: Add a public API endpoint to retrieve a set of files with a given name as Low priority.
Jan 21 2019, 4:28 PM · Easy hack, Storage manager, Object storage

Dec 19 2018

ardumont added a comment to T803: Indexer - Retrieval error when contents is too big.

I suppose it all depends on the current storage's configuration.

Dec 19 2018, 10:21 AM · Indexer, Object storage

Dec 18 2018

olasd added a comment to T1447: Add support for slices when getting objects from the objstorage..

objstorage_pathslicing manipulates a *gzipped* file object, which means that TTBOMK seek is not supported, and we will have to decompress the complete beginning of the file to get to the range that we really want to read.

Dec 18 2018, 6:36 PM · Object storage
ardumont added a comment to T803: Indexer - Retrieval error when contents is too big.

In the objstorage's pathslicing implementation, there is the get_stream implementation which is not used [1]

Dec 18 2018, 5:10 PM · Indexer, Object storage
vlorentz triaged T1447: Add support for slices when getting objects from the objstorage. as Low priority.
Dec 18 2018, 4:40 PM · Object storage
vlorentz added a subtask for T803: Indexer - Retrieval error when contents is too big: T1446: Add support for slices in Storage.content_get.
Dec 18 2018, 4:35 PM · Indexer, Object storage

Nov 19 2018

vlorentz claimed T1307: Remove mock storages used in tests..
Nov 19 2018, 5:11 PM · Storage manager
vlorentz closed T1306: Write an in-memory backend for swh.storage for tests., a subtask of T1307: Remove mock storages used in tests., as Resolved.
Nov 19 2018, 5:11 PM · Storage manager

Nov 7 2018

vlorentz renamed T1306: Write an in-memory backend for swh.storage for tests. from Write in-memory backends for swh.storage and swh.objstorage for tests. to Write an in-memory backend for swh.storage for tests..
Nov 7 2018, 12:04 PM · Storage manager
ardumont added a comment to T1306: Write an in-memory backend for swh.storage for tests..

objstorage already has an in-memory implementation?

Nov 7 2018, 11:57 AM · Storage manager

Nov 6 2018

olasd added a comment to T1306: Write an in-memory backend for swh.storage for tests..

objstorage already has an in-memory implementation?

Nov 6 2018, 3:43 PM · Storage manager
vlorentz renamed T1306: Write an in-memory backend for swh.storage for tests. from Write an in-memory backend for swh.storage for tests. to Write in-memory backends for swh.storage and swh.objstorage for tests..
Nov 6 2018, 2:40 PM · Storage manager
vlorentz added a subtask for T1307: Remove mock storages used in tests.: T1306: Write an in-memory backend for swh.storage for tests..
Nov 6 2018, 2:40 PM · Storage manager
vlorentz triaged T1307: Remove mock storages used in tests. as Normal priority.
Nov 6 2018, 2:39 PM · Storage manager

Oct 11 2018

ardumont closed D514: tests: Add rados requirements.
Oct 11 2018, 11:24 AM · Object storage
ardumont updated the summary of D514: tests: Add rados requirements.
Oct 11 2018, 11:23 AM · Object storage
vlorentz accepted D514: tests: Add rados requirements.
Oct 11 2018, 11:21 AM · Object storage
ardumont added a project to D514: tests: Add rados requirements: Object storage.
Oct 11 2018, 11:20 AM · Object storage

Aug 3 2018

ftigeot added a comment to T1048: Clean striped object storages from objects they should not be containing.

Given Uffizi already uses 64GB of RAM (more than some physical machines), this should be a no brainer.
I am not sure if this would really improve I/O performance, though.

Aug 3 2018, 3:03 PM · Object storage

Jul 17 2018

olasd changed the status of T1047: Write all contents synchronously to the ceph cluster, a subtask of T1043: handle the uffizi content store being full, from Open to Work in Progress.
Jul 17 2018, 2:15 PM · Object storage