Page MenuHomeSoftware Heritage

Azure objstorage performance improvements
ClosedPublic

Authored by olasd on May 3 2018, 8:31 PM.

Details

Summary

Most notably:

Add a prefixed azure object storage which directly maps across several accounts

This should reduce the overhead from using the stack of multiplexer and filter
to access our Azure content stores.

Before adding this, stack a few cleanup commits, most notably:

  • Use a request_session for the block blob service to enable https pipelining
  • objstorage_azure: Cleaner mocking of the BlockBlobService class for tests
  • Use get_objstorage instead of instantiating the AzureCloudObjStorage class directly in test

This is related to T1044

Test Plan

Added unit tests

Diff Detail

Repository
rDOBJS Object storage
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.