HomeSoftware Heritage

Azure objstorage performance improvements

Description

Azure objstorage performance improvements

Summary:
Most notably:

Add a prefixed azure object storage which directly maps across several accounts

This should reduce the overhead from using the stack of multiplexer and filter
to access our Azure content stores.

Before adding this, stack a few cleanup commits, most notably:

  • Use a request_session for the block blob service to enable https pipelining
  • objstorage_azure: Cleaner mocking of the BlockBlobService class for tests
  • Use get_objstorage instead of instantiating the AzureCloudObjStorage class directly in test

This is related to T1044

Test Plan: Added unit tests

Reviewers: Reviewers!

Differential Revision: https://forge.softwareheritage.org/D318

Details

Provenance
olasdAuthored on May 3 2018, 8:22 PM
olasdPushed on May 7 2018, 1:44 PM
Differential Revision
D318: Azure objstorage performance improvements
Parents
rDOBJS246d6d1cfd01: Use get_objstorage instead of instantiating the AzureCloudObjStorage class…
Branches
Unknown
Tags
Unknown