Page MenuHomeSoftware Heritage

Reimplement the azure objstorage on top of recent azure-storage-blob
ClosedPublic

Authored by olasd on May 4 2020, 3:01 PM.

Details

Summary

The new version is what Debian unstable is carrying, so use that. Unfortunately
the situation with respect to the old API hasn't been clarified by upstream yet.

https://github.com/Azure/azure-storage-python/issues/655

Test Plan

toxed, and also ran the tests on a real azure storage account

Diff Detail

Repository
rDOBJS Object storage
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

olasd created this revision.May 4 2020, 3:01 PM

Build is green

Patch application report for D3116 (id=11074)

Rebasing onto d3dba87634...

Current branch diff-target is up to date.
Changes applied before test
commit 3fad3f8f5b7e6a4047b44afa968b47020deda860
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Apr 30 19:50:17 2020 +0200

    Reimplement the azure objstorage on top of recent azure-storage-blob
    
    The new version is what Debian unstable is carrying, so use that. Unfortunately
    the situation with respect to the old API hasn't been clarified by upstream yet.
    
    https://github.com/Azure/azure-storage-python/issues/655

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/6/ for more details.

douardda accepted this revision.May 4 2020, 3:26 PM
douardda added a subscriber: douardda.

I'd like one or 2 comments (as stated in the comment above), and I don't like the inheritance between PrefixedAzureCloudObjStorage and AzureCloudObjStorage, but that's not the object of this diff.

swh/objstorage/backends/azure.py
135–139

this method looks weird as is. Why does it exists? (not read the whole file yet, but my bet is a dispatching mechanism implemented in a subclass.

I've noted this was already like this before, nonetheless, it could benefit from a comment or something IMHO.

This revision is now accepted and ready to land.May 4 2020, 3:26 PM
olasd updated this revision to Diff 11087.May 4 2020, 7:33 PM
  • Apply @douardda's comments;
  • Add get_container_url helper;
  • Add backwards-compatibility to existing azure storage definitions.
olasd requested review of this revision.May 4 2020, 7:33 PM

Build is green

Patch application report for D3116 (id=11087)

Rebasing onto f7ba8fe19c...

Current branch diff-target is up to date.
Changes applied before test
commit 87f5ab96148664e182e057b000c7ce25537a5ff0
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Apr 30 19:50:17 2020 +0200

    Reimplement the azure objstorage on top of recent azure-storage-blob
    
    The new version is what Debian unstable is carrying, so use that. Unfortunately
    the situation with respect to the old API hasn't been clarified by upstream yet.
    
    https://github.com/Azure/azure-storage-python/issues/655

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/7/ for more details.

olasd updated this revision to Diff 11088.May 4 2020, 7:54 PM

Add tests for backwards-compatible calls

Build is green

Patch application report for D3116 (id=11088)

Rebasing onto f7ba8fe19c...

Current branch diff-target is up to date.
Changes applied before test
commit cc43ada0556b1f7b45229c7a5bd0b90e4969c603
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Apr 30 19:50:17 2020 +0200

    Reimplement the azure objstorage on top of recent azure-storage-blob
    
    The new version is what Debian unstable is carrying, so use that. Unfortunately
    the situation with respect to the old API hasn't been clarified by upstream yet.
    
    https://github.com/Azure/azure-storage-python/issues/655

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/8/ for more details.

douardda accepted this revision.May 5 2020, 10:02 AM

nice work! thx

This revision is now accepted and ready to land.May 5 2020, 10:02 AM