Page MenuHomeSoftware Heritage

Object storageFolder
ActivePublic

Members

  • This project does not have any members.

Watchers

  • This project does not have any watchers.

Recent Activity

Wed, Jan 22

vlorentz added projects to T2215: Streaming support everywhere: Storage manager, Object storage, Web app.
Wed, Jan 22, 4:24 PM · Web app, Object storage, Storage manager, Restricted Project
vlorentz added a project to T2216: Packing object storage: Object storage.
Wed, Jan 22, 4:20 PM · Object storage, Restricted Project

Dec 7 2019

olasd added a comment to T1954: Up-to-date objstorage mirror on S3.

I've grown tired of babysitting this, so I've added systemd notify calls to the journal replayer, allowing us to just use the systemd watchdog to restart hung processes.

Dec 7 2019, 6:37 PM · System administration, Object storage
olasd merged T1899: complete object storage mirror on AWS into T1954: Up-to-date objstorage mirror on S3.
Dec 7 2019, 6:30 PM · System administration, Object storage

Nov 25 2019

olasd added a comment to T1954: Up-to-date objstorage mirror on S3.
In T1954#39027, @zack wrote:

So, the amount of contents on S3 went up fairly quickly during in between Nov 10th and Nov 20th, but then it stopped again, is it expected/normal?
(thanks for the metric, it really helps)

Nov 25 2019, 3:30 PM · System administration, Object storage

Nov 24 2019

zack added a comment to T1954: Up-to-date objstorage mirror on S3.

So, the amount of contents on S3 went up fairly quickly during in between Nov 10th and Nov 20th, but then it stopped again, is it expected/normal?

Nov 24 2019, 8:26 PM · System administration, Object storage

Nov 8 2019

olasd added a comment to T1954: Up-to-date objstorage mirror on S3.

I've added a metric with the S3 objects to https://grafana.softwareheritage.org/d/jScG7g6mk/objstorage-object-counts. There's... "some" work to do still.

Nov 8 2019, 7:12 PM · System administration, Object storage
olasd changed the status of T1954: Up-to-date objstorage mirror on S3 from Open to Work in Progress.

So I've deployed this (by hand for now) on uffizi and it seems to be doing its job.

Nov 8 2019, 11:41 AM · System administration, Object storage

Oct 29 2019

vlorentz updated the task description for T1954: Up-to-date objstorage mirror on S3.
Oct 29 2019, 11:34 AM · System administration, Object storage

Aug 19 2019

vlorentz updated the task description for T1954: Up-to-date objstorage mirror on S3.
Aug 19 2019, 11:47 AM · System administration, Object storage
vlorentz updated the task description for T1954: Up-to-date objstorage mirror on S3.
Aug 19 2019, 11:45 AM · System administration, Object storage
vlorentz triaged T1954: Up-to-date objstorage mirror on S3 as High priority.
Aug 19 2019, 11:44 AM · System administration, Object storage

Jun 19 2019

olasd closed T1823: make DB/FS transactions nest properly as Resolved by committing rDOBJS67197802d5aa: pathslicing: Make sure data is flushed to disk before renaming the tempfile.
Jun 19 2019, 4:11 PM · Object storage, Storage manager
olasd added a revision to T1823: make DB/FS transactions nest properly: D1611: pathslicing: Make sure data is flushed to disk before renaming the tempfile.
Jun 19 2019, 1:46 PM · Object storage, Storage manager

Jun 18 2019

zack triaged T1823: make DB/FS transactions nest properly as High priority.
Jun 18 2019, 12:38 PM · Object storage, Storage manager

May 23 2019

vlorentz triaged T1736: Update the "Archive copies" documentation as Normal priority.
May 23 2019, 11:17 AM · Object storage, Development documentation

Mar 26 2019

olasd added a parent task for T1577: Compare/benchmark objstorage backends : T1608: Write all new objects to azure synchronously.
Mar 26 2019, 6:58 PM · Object storage

Mar 22 2019

seirl placed T805: objstorage: allow use of file-like objects for streaming methods up for grabs.
Mar 22 2019, 1:31 PM · Object storage
seirl triaged T1596: ObjStorage: investigate the per-request CPU overhead of aiohttp as Normal priority.
Mar 22 2019, 1:27 PM · Object storage
seirl triaged T1595: ObjStorage: compression pass-through as Normal priority.
Mar 22 2019, 1:24 PM · Object storage
seirl triaged T1594: ObjStorage: investigate HTTP pipelining issues across all the different backends as Normal priority.
Mar 22 2019, 11:35 AM · Object storage
seirl triaged T1593: ObjStorage: fast mass-retrieval of objects as Normal priority.
Mar 22 2019, 11:33 AM · Object storage

Mar 12 2019

vlorentz removed a project from T1447: Add support for slices when getting objects from the objstorage.: Easy hack.

I didn't know objects are compressed. That indeed makes the issue harder.

Mar 12 2019, 11:51 AM · Object storage
douardda triaged T1577: Compare/benchmark objstorage backends as Normal priority.
Mar 12 2019, 9:53 AM · Object storage
douardda added a comment to T1447: Add support for slices when getting objects from the objstorage..

Same here. Not that much an easy hack. And what is the real life use case that drive this feature request? YAGNI?

Mar 12 2019, 9:46 AM · Object storage

Feb 25 2019

ardumont closed T1533: Make sure api server uses explicit configurations as Resolved.
Feb 25 2019, 12:15 PM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault
ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Feb 25 2019, 12:15 PM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault

Feb 23 2019

ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Feb 23 2019, 1:58 AM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault
ardumont added a project to T1533: Make sure api server uses explicit configurations: Scheduling utilities.
Feb 23 2019, 12:43 AM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault
ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Feb 23 2019, 12:42 AM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault

Feb 22 2019

ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Feb 22 2019, 11:22 AM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault

Feb 21 2019

ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Feb 21 2019, 8:34 PM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault
ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Feb 21 2019, 2:11 PM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault

Feb 20 2019

ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Feb 20 2019, 7:11 PM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault

Feb 19 2019

ardumont renamed T1533: Make sure api server uses explicit configurations from Use explicit configuration for all modules to Make sure api server uses explicit configurations.
Feb 19 2019, 11:36 AM · Scheduling utilities, Web app, SWORD deposit, Object storage, Storage manager, Vault

Jan 31 2019

vlorentz added a project to T1487: Add a public API endpoint to retrieve a set of files with a given name: Easy hack.
Jan 31 2019, 1:29 PM · Easy hack, Storage manager, Object storage

Jan 21 2019

vlorentz added a project to T1487: Add a public API endpoint to retrieve a set of files with a given name: Storage manager.

The object storage doesn't have content names, so it cannot address this feature as stated.

Jan 21 2019, 4:55 PM · Easy hack, Storage manager, Object storage
zack updated subscribers of T1487: Add a public API endpoint to retrieve a set of files with a given name.

The object storage doesn't have content names, so it cannot address this feature as stated.

Jan 21 2019, 4:50 PM · Easy hack, Storage manager, Object storage
vlorentz added a comment to T1487: Add a public API endpoint to retrieve a set of files with a given name.

A crude script doing this: https://forge.softwareheritage.org/source/snippets/browse/master/vlorentz/get_metadata_files_examples.py

Jan 21 2019, 4:29 PM · Easy hack, Storage manager, Object storage
vlorentz triaged T1487: Add a public API endpoint to retrieve a set of files with a given name as Low priority.
Jan 21 2019, 4:28 PM · Easy hack, Storage manager, Object storage

Dec 19 2018

ardumont added a comment to T803: Indexer - Retrieval error when contents is too big.

I suppose it all depends on the current storage's configuration.

Dec 19 2018, 10:21 AM · Indexer, Object storage

Dec 18 2018

olasd added a comment to T1447: Add support for slices when getting objects from the objstorage..

objstorage_pathslicing manipulates a *gzipped* file object, which means that TTBOMK seek is not supported, and we will have to decompress the complete beginning of the file to get to the range that we really want to read.

Dec 18 2018, 6:36 PM · Object storage
ardumont added a comment to T803: Indexer - Retrieval error when contents is too big.

In the objstorage's pathslicing implementation, there is the get_stream implementation which is not used [1]

Dec 18 2018, 5:10 PM · Indexer, Object storage
vlorentz triaged T1447: Add support for slices when getting objects from the objstorage. as Low priority.
Dec 18 2018, 4:40 PM · Object storage
vlorentz added a subtask for T803: Indexer - Retrieval error when contents is too big: T1446: Add support for slices in Storage.content_get.
Dec 18 2018, 4:35 PM · Indexer, Object storage

Nov 19 2018

vlorentz claimed T1307: Remove mock storages used in tests..
Nov 19 2018, 5:11 PM · Storage manager
vlorentz closed T1306: Write an in-memory backend for swh.storage for tests., a subtask of T1307: Remove mock storages used in tests., as Resolved.
Nov 19 2018, 5:11 PM · Storage manager

Nov 7 2018

vlorentz renamed T1306: Write an in-memory backend for swh.storage for tests. from Write in-memory backends for swh.storage and swh.objstorage for tests. to Write an in-memory backend for swh.storage for tests..
Nov 7 2018, 12:04 PM · Storage manager
ardumont added a comment to T1306: Write an in-memory backend for swh.storage for tests..

objstorage already has an in-memory implementation?

Nov 7 2018, 11:57 AM · Storage manager

Nov 6 2018

olasd added a comment to T1306: Write an in-memory backend for swh.storage for tests..

objstorage already has an in-memory implementation?

Nov 6 2018, 3:43 PM · Storage manager