Page MenuHomeSoftware Heritage

Vault: use objstorage streaming to store and fetch bundles
Open, NormalPublic

Event Timeline

seirl claimed this task.Jan 31 2018, 3:17 PM
seirl removed seirl as the assignee of this task.Mar 22 2019, 1:33 PM
zack added a subscriber: zack.EditedThu, Sep 19, 12:15 PM

see T1964 for a concrete example where the lack of streaming is causing problems (after the cooking, when the bundle is ready)

$ wget
--2019-09-19 11:43:50--
Resolving (
Connecting to (||:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 539845226 (515M) [application/gzip]
Saving to: ‘index.html’

index.html                    31%[=============>                                ] 162,18M  2,66MB/s    in 66s     

2019-09-19 11:46:13 (2,46 MB/s) - Connection closed at byte 170059557. Retrying.

--2019-09-19 11:46:14--  (try: 2)
Connecting to (||:443... connected.
HTTP request sent, awaiting response... 400 Bad Request
2019-09-19 11:50:29 ERROR 400: Bad Request.
olasd added a subscriber: olasd.Thu, Sep 19, 1:18 PM

I wonder whether the best solution wouldn't be to just generate a redirect to a direct download url from the azure bucket using a temporary shared access signature.

e.g. in the rocrail case: (expiry today at 19:15 UTC)

olasd added a comment.Thu, Sep 19, 1:19 PM

Of course, the current bundles are double-compressed, which makes this... not great.

olasd added a comment.Mon, Oct 7, 7:06 PM

Pluggable compression has been implemented for all objstorage backends, which means we could

  • store the (compressed) bundles in an uncompressed objstorage on azure
  • when a user requests the bundle