Change Details

Leveraging azure infrastructure, trigger the blake2s256 update on the existing contents. This means: - [X] Provisioning azure vms (sizing -> DS2_V2: 7GB ram, 14GB ssd disk, 2 cores; 85.33E/month) -> for now 2 vms - [X] code: configuration composability on storage read/write and objstorage readings adaptation - [X] puppet: swh_indexer_rehash puppetization - [X] Deploying the swh.indexer.rehash module (+ fix bits and pieces along the way) - [ ] Send all contents to the swh_indexer_rehash queue (table archiver.content or swh.content to determine). - [ ] Cost/speed projection as soon as some threshold contents have been updated (~1M sounds reasonable given our size) Note: In regards to the storage stack to use, we can: - either use the azure's objstorage (copy is complete now). This will be the starting point. - or use uffizi's objstorage (or banco) as the azure's in-transit's cost is null if the cost projection is too high.

Leveraging azure infrastructure, trigger the blake2s256 update on the existing contents. This means: - [X] Provisioning azure vms (sizing -> DS2_V2: 7GB ram, 14GB ssd disk, 2 cores; 85.33E/month) -> for now 2 vms - [X] code: configuration composability on storage read/write and objstorage readings adaptation - [X] puppet: swh_indexer_rehash puppetization - [X] Deploying the swh.indexer.rehash module (+ fix bits and pieces along the way) - [-] Compute list of sha1 to rehash from swh.content table (in progress in uffizi:/srv/storage/space/lists/contents-sha1-to-rehash.txt.gz) - [ ] Send all contents to the swh_indexer_rehash queue - [ ] Cost/speed projection as soon as some threshold contents have been updated (~1M sounds reasonable given our size) Note: In regards to the storage stack to use, we can: - either use the azure's objstorage (copy is complete now). This will be the starting point. - or use uffizi's objstorage (or banco) as the azure's in-transit's cost is null if the cost projection is too high.

Leveraging azure infrastructure, trigger the blake2s256 update on the existing contents. This means: - [X] Provisioning azure vms (sizing -> DS2_V2: 7GB ram, 14GB ssd disk, 2 cores; 85.33E/month) -> for now 2 vms - [X] code: configuration composability on storage read/write and objstorage readings adaptation - [X] puppet: swh_indexer_rehash puppetization - [X] Deploying the swh.indexer.rehash module (+ fix bits and pieces along the way) - [ ] Send all contents to the swh_indexer_rehash queue (table archiver.content or swh.content to determine).-] Compute list of sha1 to rehash from swh.content table (in progress in uffizi:/srv/storage/space/lists/contents-sha1-to-rehash.txt.gz) - [ ] Send all contents to the swh_indexer_rehash queue - [ ] Cost/speed projection as soon as some threshold contents have been updated (~1M sounds reasonable given our size) Note: In regards to the storage stack to use, we can: - either use the azure's objstorage (copy is complete now). This will be the starting point. - or use uffizi's objstorage (or banco) as the azure's in-transit's cost is null if the cost projection is too high.