Page MenuHomeSoftware Heritage

webapp0: Switch to cassandra storage
ClosedPublic

Authored by ardumont on Jan 27 2020, 4:04 PM.

Details

Reviewers
vlorentz
olasd
Group Reviewers
Reviewers
Summary

Related to T2185

Test Plan

bin/octocatalog storage0.euwest.azure -> nothing
bin/octocatalog webapp0 -> webapp's config update with the storage key changed to the cassandra backend (+ install python3-cassandra)

Diff Detail

Repository
rSPSITE puppet-swh-site
Branch
staging
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 10311
Build 15319: arc lint + arc unit

Event Timeline

olasd requested changes to this revision.Jan 27 2020, 6:56 PM

I believe the vault and the indexers also use storage0.euwest.azure as read-only archive backend. We don't want to switch them over to cassandra (at least not until the replay has completed, if ever).

I think we should inline the cassandra storage config in the webapp0 swh.web config instead.

We should also make sure that the optional cassandra dependencies are pulled in by the storage deployment (unless swh.storage grew hardcoded dependencies on the cassandra stuff, in which case we're fine).

This revision now requires changes to proceed.Jan 27 2020, 6:56 PM

I agree with @olasd. Another possibility is to run an swh-storage instance on webapp0.

Right.

I also wanted to make sure i did not misconfigure. Nobody shouted at it so it seemed fine from that standpoint ;)

Thanks.

I think we should inline the cassandra storage config in the webapp0 swh.web config instead.

Another possibility is to run a swh-storage instance on webapp0.

yes, not sure what's the best here though.

I'll go with the first proposition for no other convention than 'dibs' :D

We should also make sure that the optional cassandra dependencies are pulled in by the storage deployment (unless swh.storage grew hardcoded dependencies on the cassandra stuff, in which case we're fine).

btw, yeah.
I believe i started working on that [1]
For some unknown reasons yet, i don't see the python-cassandra-driver (buster) uploaded in pergamon.
Don't know if it's related or not, It's marked as source, i need to continue the work on that [2]

[1] https://forge.softwareheritage.org/source/python-cassandra-driver/

[2]

root@pergamon:/srv/softwareheritage/repository# reprepro ls python-cassandra-driver
python-cassandra-driver | 3.20.2-1~swh1~bpo10+1 | buster-swh | source

Switch to cassandra storage only for webapp0

ardumont retitled this revision from storage0: Switch to cassandra storage to webapp0: Switch to cassandra storage.Jan 28 2020, 11:01 AM
ardumont edited the summary of this revision. (Show Details)
ardumont edited the test plan for this revision. (Show Details)

For some unknown reasons yet, i don't see the python-cassandra-driver (buster) uploaded in pergamon.
Don't know if it's related or not, It's marked as source, i need to continue the work on that [2]

Thanks to @olasd, we saw everything is fine in the end.
I looked for the wrong package ;)
python-cassandra-driver is the source package
python3-cassandra is what we want though \m/

root@pergamon:/srv/softwareheritage/repository# reprepro ls python3-cassandra
python3-cassandra | 3.20.2-1~swh1~bpo10+1 | buster-swh | amd64
ardumont edited the test plan for this revision. (Show Details)

We should also make sure that the optional cassandra dependencies are pulled in by the storage deployment (unless swh.storage grew hardcoded dependencies on the cassandra stuff, in which case we're fine).

i realize i did not deal with that part though...

  • Properly switch to webapp0's storage to cassandra storage
  • Add conditional to install the python3-cassandra when needed

Fix python3-cassandra package name

you're making the cassandra driver in the same process as the webapp. Are you sure you want to do that?

you're making the cassandra driver in the same process as the webapp. Are you sure you want to do that?

No idea yet, i did what i said i would do.
Also i would like to try and see what happens.
And then improve on what's problematic.
That would still be progress ;)

olasd added inline comments.
site-modules/profile/manifests/swh/deploy/webapp.pp
107–110 ↗(On Diff #9264)

Could you move that package stanza into a new include ::profile::swh::deploy::storage::cassandra instead? That way we can do the same change to the swh::deploy::storage (and avoid dependency conflicts).

This revision is now accepted and ready to land.Jan 28 2020, 3:08 PM
site-modules/profile/manifests/swh/deploy/webapp.pp
107–110 ↗(On Diff #9264)

sure.

Move the cassandra storage's dependencies to a profile

  • Install conditionally on webapp node if configuration requires it (for webapp0).
  • Install conditionally on storage node if configuration requires it (nobody for now)
ardumont edited the test plan for this revision. (Show Details)

Landed.