Page MenuHomeSoftware Heritage

webapp0: Switch to cassandra storage
ClosedPublic

Authored by ardumont on Mon, Jan 27, 4:04 PM.

Details

Reviewers
vlorentz
olasd
Group Reviewers
Reviewers
Summary

Related to T2185

Test Plan

bin/octocatalog storage0.euwest.azure -> nothing
bin/octocatalog webapp0 -> webapp's config update with the storage key changed to the cassandra backend (+ install python3-cassandra)

Diff Detail

Repository
rSPSITE puppet-swh-site
Branch
staging
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 10329
Build 15347: arc lint + arc unit

Event Timeline

ardumont created this revision.Mon, Jan 27, 4:04 PM
olasd requested changes to this revision.Mon, Jan 27, 6:56 PM

I believe the vault and the indexers also use storage0.euwest.azure as read-only archive backend. We don't want to switch them over to cassandra (at least not until the replay has completed, if ever).

I think we should inline the cassandra storage config in the webapp0 swh.web config instead.

We should also make sure that the optional cassandra dependencies are pulled in by the storage deployment (unless swh.storage grew hardcoded dependencies on the cassandra stuff, in which case we're fine).

This revision now requires changes to proceed.Mon, Jan 27, 6:56 PM

I agree with @olasd. Another possibility is to run an swh-storage instance on webapp0.

ardumont added a comment.EditedTue, Jan 28, 10:17 AM

Right.

I also wanted to make sure i did not misconfigure. Nobody shouted at it so it seemed fine from that standpoint ;)

Thanks.

I think we should inline the cassandra storage config in the webapp0 swh.web config instead.

Another possibility is to run a swh-storage instance on webapp0.

yes, not sure what's the best here though.

I'll go with the first proposition for no other convention than 'dibs' :D

We should also make sure that the optional cassandra dependencies are pulled in by the storage deployment (unless swh.storage grew hardcoded dependencies on the cassandra stuff, in which case we're fine).

btw, yeah.
I believe i started working on that [1]
For some unknown reasons yet, i don't see the python-cassandra-driver (buster) uploaded in pergamon.
Don't know if it's related or not, It's marked as source, i need to continue the work on that [2]

[1] https://forge.softwareheritage.org/source/python-cassandra-driver/

[2]

root@pergamon:/srv/softwareheritage/repository# reprepro ls python-cassandra-driver
python-cassandra-driver | 3.20.2-1~swh1~bpo10+1 | buster-swh | source
ardumont updated this revision to Diff 9252.Tue, Jan 28, 11:00 AM

Switch to cassandra storage only for webapp0

ardumont retitled this revision from storage0: Switch to cassandra storage to webapp0: Switch to cassandra storage.Tue, Jan 28, 11:01 AM
ardumont edited the summary of this revision. (Show Details)
ardumont edited the test plan for this revision. (Show Details)

For some unknown reasons yet, i don't see the python-cassandra-driver (buster) uploaded in pergamon.
Don't know if it's related or not, It's marked as source, i need to continue the work on that [2]

Thanks to @olasd, we saw everything is fine in the end.
I looked for the wrong package ;)
python-cassandra-driver is the source package
python3-cassandra is what we want though \m/

root@pergamon:/srv/softwareheritage/repository# reprepro ls python3-cassandra
python3-cassandra | 3.20.2-1~swh1~bpo10+1 | buster-swh | amd64
ardumont edited the test plan for this revision. (Show Details)Tue, Jan 28, 2:15 PM
ardumont edited the test plan for this revision. (Show Details)

We should also make sure that the optional cassandra dependencies are pulled in by the storage deployment (unless swh.storage grew hardcoded dependencies on the cassandra stuff, in which case we're fine).

i realize i did not deal with that part though...

ardumont updated this revision to Diff 9263.Tue, Jan 28, 2:32 PM
  • Properly switch to webapp0's storage to cassandra storage
  • Add conditional to install the python3-cassandra when needed
ardumont updated this revision to Diff 9264.Tue, Jan 28, 2:32 PM

Fix python3-cassandra package name

you're making the cassandra driver in the same process as the webapp. Are you sure you want to do that?

you're making the cassandra driver in the same process as the webapp. Are you sure you want to do that?

No idea yet, i did what i said i would do.
Also i would like to try and see what happens.
And then improve on what's problematic.
That would still be progress ;)

olasd accepted this revision.Tue, Jan 28, 3:08 PM
olasd added inline comments.
site-modules/profile/manifests/swh/deploy/webapp.pp
107–110

Could you move that package stanza into a new include ::profile::swh::deploy::storage::cassandra instead? That way we can do the same change to the swh::deploy::storage (and avoid dependency conflicts).

This revision is now accepted and ready to land.Tue, Jan 28, 3:08 PM
ardumont added inline comments.Tue, Jan 28, 4:19 PM
site-modules/profile/manifests/swh/deploy/webapp.pp
107–110

sure.

ardumont updated this revision to Diff 9270.Tue, Jan 28, 5:28 PM

Move the cassandra storage's dependencies to a profile

  • Install conditionally on webapp node if configuration requires it (for webapp0).
  • Install conditionally on storage node if configuration requires it (nobody for now)
ardumont closed this revision.Tue, Jan 28, 5:29 PM
ardumont edited the test plan for this revision. (Show Details)

Landed.