Page MenuHomeSoftware Heritage

docker: add the swh-winery-db and swh-winery services
AcceptedPublic

Authored by dachary on Dec 14 2021, 3:47 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T3533: Winery backend: basic implementation
Summary

Related to: T3533

It depends on D6796 and will fail until it is merged.

Test Plan

It can be tested from sources (assuming swh-objstorage is checked out with https://forge.softwareheritage.org/D6796) with an override like this:

version: "2.1"

services:
  swh-winery:
    volumes:
      - "../swh-objstorage:/src/swh-objstorage"

and

  • cd docker ; docker-compose up swh-winery-db swh-winery

Diff Detail

Repository
rDENV Development environment
Branch
wip-winery (branched from master)
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 25625
Build 40069: arc lint + arc unit

Event Timeline

It depends on https://forge.softwareheritage.org/D6796 and will fail until it is merged. It can be tested from sources with an override like this:

version: "2.1"

services:
  swh-winery:
    volumes:
      - "../swh-objstorage:/src/swh-objstorage"
dachary edited the test plan for this revision. (Show Details)
dachary edited the test plan for this revision. (Show Details)

Instead of defining a new service, could you provide an alternative docker-compose config file? This way, it can be used as to switch all services to use it, just by adding a CLI parameter. eg. we do this to replace the postgres storage backend with cassandra: https://docs.softwareheritage.org/devel/getting-started/using-docker.html#cassandra

Instead of defining a new service, could you provide an alternative docker-compose config file? This way, it can be used as to switch all services to use it, just by adding a CLI parameter. eg. we do this to replace the postgres storage backend with cassandra: https://docs.softwareheritage.org/devel/getting-started/using-docker.html#cassandra

The winery is not intended as a replacement but as an additional service that will run along the existing swh-objstorage. The swh-objstorage may be configured to use swh-winery (or not) using the existing "remote" cls. Does your suggestion still stands knowing this? I'm not against it on principle (that's my first patch here, I have no experience or opinion really). I'm just making sure you got the context right before doing what you suggest.

I don't understand why. Currently, we have: client --> objstorage pathslicing backend (port 5003) --> disk.
What you want to do is: client --> objstorage proxy (port 5003) --> objstorage winery backend (port 5012) --> winery --> ceph, right?
(objstorage winery backend is what is launched by docker/services/swh-winery/entrypoint.sh in this diff)

objstorage winery backend must implement exactly the same interface as objstorage pathslicing backend, otherwise the proxy won't work. So why have the proxy in the first place instead of client --> objstorage winery backend --> winery --> ceph?


Anyway, assuming we use an objstorage proxy, then we will need a dockerfile that changes the objstorage config from {cls: "pathslicing", path: ...} (the pathslicing backend) to {cls: "remote", url: "http://swh-winery:5012"} (a proxy to the winery backend)

Ah wait I got it, docker/services/swh-winery/entrypoint.sh launches winery itself, not an actual objstorage backend; you're only reusing the scafholding.

Alright, let's do this and we'll remove the proxy later if it turns out to be unnecessary. Please allocate port 5012 in https://forge.softwareheritage.org/source/swh-docs/browse/master/docs/services.txt though.

docker/services/swh-winery/entrypoint.sh
14–15

indentation issue

This revision is now accepted and ready to land.Dec 14 2021, 5:03 PM

Ah wait I got it, docker/services/swh-winery/entrypoint.sh launches winery itself, not an actual objstorage backend; you're only reusing the scafholding.

Exactly :-)

Alright, let's do this and we'll remove the proxy later if it turns out to be unnecessary. Please allocate port 5012 in https://forge.softwareheritage.org/source/swh-docs/browse/master/docs/services.txt though.

Oh, thanks for the hint. Done in D6837