Page MenuHomeSoftware Heritage

Object storageFolder
ActivePublic

Members

  • This project does not have any members.

Watchers

  • This project does not have any watchers.

Recent Activity

Mon, Nov 19

vlorentz claimed T1307: Remove mock storages used in tests..
Mon, Nov 19, 5:11 PM · Storage manager
vlorentz closed T1306: Write an in-memory backend for swh.storage for tests., a subtask of T1307: Remove mock storages used in tests., as Resolved.
Mon, Nov 19, 5:11 PM · Storage manager

Nov 7 2018

vlorentz renamed T1306: Write an in-memory backend for swh.storage for tests. from Write in-memory backends for swh.storage and swh.objstorage for tests. to Write an in-memory backend for swh.storage for tests..
Nov 7 2018, 12:04 PM · Storage manager
ardumont added a comment to T1306: Write an in-memory backend for swh.storage for tests..

objstorage already has an in-memory implementation?

Nov 7 2018, 11:57 AM · Storage manager

Nov 6 2018

olasd added a comment to T1306: Write an in-memory backend for swh.storage for tests..

objstorage already has an in-memory implementation?

Nov 6 2018, 3:43 PM · Storage manager
vlorentz renamed T1306: Write an in-memory backend for swh.storage for tests. from Write an in-memory backend for swh.storage for tests. to Write in-memory backends for swh.storage and swh.objstorage for tests..
Nov 6 2018, 2:40 PM · Storage manager
vlorentz added a subtask for T1307: Remove mock storages used in tests.: T1306: Write an in-memory backend for swh.storage for tests..
Nov 6 2018, 2:40 PM · Storage manager
vlorentz triaged T1307: Remove mock storages used in tests. as Normal priority.
Nov 6 2018, 2:39 PM · Storage manager

Oct 11 2018

ardumont closed D514: tests: Add rados requirements.
Oct 11 2018, 11:24 AM · Object storage
ardumont updated the summary of D514: tests: Add rados requirements.
Oct 11 2018, 11:23 AM · Object storage
vlorentz accepted D514: tests: Add rados requirements.
Oct 11 2018, 11:21 AM · Object storage
ardumont added a project to D514: tests: Add rados requirements: Object storage.
Oct 11 2018, 11:20 AM · Object storage

Aug 3 2018

ftigeot added a comment to T1048: Clean striped object storages from objects they should not be containing.

Given Uffizi already uses 64GB of RAM (more than some physical machines), this should be a no brainer.
I am not sure if this would really improve I/O performance, though.

Aug 3 2018, 3:03 PM · Object storage

Jul 17 2018

olasd changed the status of T1047: Write all contents synchronously to the ceph cluster, a subtask of T1043: handle the uffizi content store being full, from Open to Work in Progress.
Jul 17 2018, 2:15 PM · Object storage
olasd changed the status of T1047: Write all contents synchronously to the ceph cluster from Open to Work in Progress.

The scaffolding to do this has been setup in Puppet. However, our naive ceph objstorage implementation uses around 7-8 times the space that would be used by the objects stored individually, instead of 1.4 times.

Jul 17 2018, 2:15 PM · Object storage
olasd changed the status of T1048: Clean striped object storages from objects they should not be containing, a subtask of T1043: handle the uffizi content store being full, from Open to Work in Progress.
Jul 17 2018, 2:15 PM · Object storage
olasd changed the status of T1048: Clean striped object storages from objects they should not be containing, a subtask of T1044: Write all contents synchronously to azure, from Open to Work in Progress.
Jul 17 2018, 2:15 PM · Object storage
olasd changed the status of T1048: Clean striped object storages from objects they should not be containing, a subtask of T1046: Stripe local contents between uffizi and banco, from Open to Work in Progress.
Jul 17 2018, 2:15 PM · Object storage
olasd changed the status of T1048: Clean striped object storages from objects they should not be containing from Open to Work in Progress.

This has been running for a while. It's quite taxing on uffizi even with nice and ionice set very high: load has been consistently around 45.

Jul 17 2018, 2:15 PM · Object storage

Jun 4 2018

olasd closed T1046: Stripe local contents between uffizi and banco, a subtask of T1045: Implement a striping object storage, as Resolved.
Jun 4 2018, 3:58 PM · Object storage
olasd closed T1046: Stripe local contents between uffizi and banco as Resolved.

This has been deployed now.

Jun 4 2018, 3:58 PM · Object storage
olasd closed T1046: Stripe local contents between uffizi and banco, a subtask of T1043: handle the uffizi content store being full, as Resolved.
Jun 4 2018, 3:58 PM · Object storage

May 7 2018

olasd closed T1045: Implement a striping object storage as Resolved by committing rDOBJS60fb5a0c5611: Add a striping objstorage which will distribute objects across backends.
May 7 2018, 1:44 PM · Object storage
olasd closed T1045: Implement a striping object storage, a subtask of T1043: handle the uffizi content store being full, as Resolved.
May 7 2018, 1:44 PM · Object storage

May 3 2018

olasd closed T1044: Write all contents synchronously to azure as Resolved by committing rSPSITE259ed288cf76: uffizi: write objects synchronously to azure.
May 3 2018, 5:02 PM · Object storage
olasd closed T1044: Write all contents synchronously to azure, a subtask of T1043: handle the uffizi content store being full, as Resolved.
May 3 2018, 5:02 PM · Object storage
olasd added a parent task for T1048: Clean striped object storages from objects they should not be containing: T1044: Write all contents synchronously to azure.
May 3 2018, 4:58 PM · Object storage
olasd added a subtask for T1044: Write all contents synchronously to azure: T1048: Clean striped object storages from objects they should not be containing.
May 3 2018, 4:58 PM · Object storage
zack added a comment to T1048: Clean striped object storages from objects they should not be containing.

FWIW, the proposed policy looks good to me. Green light!

May 3 2018, 4:57 PM · Object storage
olasd added a revision to T1045: Implement a striping object storage: D316: Add a striping objstorage which will distribute objects across backends.
May 3 2018, 4:57 PM · Object storage
olasd added a subtask for T1046: Stripe local contents between uffizi and banco: T1048: Clean striped object storages from objects they should not be containing.
May 3 2018, 4:54 PM · Object storage
olasd added a parent task for T1048: Clean striped object storages from objects they should not be containing: T1046: Stripe local contents between uffizi and banco.
May 3 2018, 4:54 PM · Object storage
olasd triaged T1048: Clean striped object storages from objects they should not be containing as High priority.
May 3 2018, 4:53 PM · Object storage
olasd triaged T1047: Write all contents synchronously to the ceph cluster as Normal priority.
May 3 2018, 4:46 PM · Object storage
olasd added a subtask for T1045: Implement a striping object storage: T1046: Stripe local contents between uffizi and banco.
May 3 2018, 4:44 PM · Object storage
olasd added a parent task for T1046: Stripe local contents between uffizi and banco: T1045: Implement a striping object storage.
May 3 2018, 4:44 PM · Object storage
olasd triaged T1046: Stripe local contents between uffizi and banco as High priority.
May 3 2018, 4:44 PM · Object storage
olasd changed the status of T1045: Implement a striping object storage from Open to Work in Progress.
May 3 2018, 4:42 PM · Object storage
olasd triaged T1044: Write all contents synchronously to azure as High priority.
May 3 2018, 4:41 PM · Object storage
olasd changed the status of T1043: handle the uffizi content store being full from Open to Work in Progress.
May 3 2018, 4:40 PM · Object storage

Feb 28 2018

olasd closed T760: swh api clients often fail with a BadStatusLine exception as Resolved.

So, that was a beast of a puppet refactoring, but the end result is: all our RPC servers are now nicely tucked behind an instance of nginx.

Feb 28 2018, 7:06 PM · Vault, Object storage, Storage manager

Feb 16 2018

seirl added a comment to T760: swh api clients often fail with a BadStatusLine exception.

Here is a pcap of the issue observed while a Vault cooker was calling set_progress on the Vault backend. The BadStatusLine packet that only contains "\r\n" is packet 306.

Feb 16 2018, 6:26 PM · Vault, Object storage, Storage manager
olasd added a project to T760: swh api clients often fail with a BadStatusLine exception: Vault.

The BadStatusLine error reared back its ugly head on the vault backend. It was reproducible. Putting a nginx in front of the vault backend fixed it. I guess it's time to puppetize the configuration of a http frontend to all our RPC servers.

Feb 16 2018, 6:17 PM · Vault, Object storage, Storage manager

Dec 12 2017

seirl added a parent task for T805: objstorage: allow use of file-like objects for streaming methods: T885: Vault: use objstorage streaming to store and fetch bundles.
Dec 12 2017, 3:17 PM · Object storage

Nov 6 2017

olasd added a parent task for T698: Migrate the content store to a new (internal) primary key scheme: T835: Migrate away from using sha1s as foreign keys in the database.
Nov 6 2017, 2:38 PM · Object storage, Storage manager

Oct 13 2017

seirl triaged T805: objstorage: allow use of file-like objects for streaming methods as Normal priority.
Oct 13 2017, 11:38 AM · Object storage
seirl created T805: objstorage: allow use of file-like objects for streaming methods.
Oct 13 2017, 11:37 AM · Object storage

Oct 10 2017

ardumont created T803: Indexer - Retrieval error when contents is too big.
Oct 10 2017, 3:04 PM · Indexer, Object storage

Sep 18 2017

olasd added a comment to T760: swh api clients often fail with a BadStatusLine exception.

As a proof of concept nginx has been manually deployed to uffizi on port 15003. It does alleviate the BadStatusLine issues the archiver encountered before under high load. This "just" needs to be properly deployed.

Sep 18 2017, 8:50 PM · Vault, Object storage, Storage manager

Sep 15 2017

zack closed T746: Objstorage: add a way to delete items (with a filter for production environments) as Resolved.

this has been fixed in rDOBJS350c2e7bc94e

Sep 15 2017, 10:21 AM · Object storage