Page MenuHomeSoftware Heritage
Differential D5974

cassandra: Allow to configure the consistency level to use
ClosedPublic
Actions

Authored by vsellier on Jul 6 2021, 4:58 PM.

Details

Reviewers
ardumont
Maniphest Tasks
T3396: cassandra - allow to configure the consistency level used by the queries
Commits
rDSTO9747aed6cbf1: cassandra: Allow to configure the consistency level to use
Summary

The default ONE level is used to keep the previous behaviour

Related to T3396

Diff Detail

Repository
rDSTO Storage manager
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 22497
Build 35051: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 35050: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
404 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_cassandra.TestCassandraStorage::test_config_consistency_used
self = <swh.storage.tests.test_cassandra.TestCassandraStorage object at 0x7f9174b79438> swh_storage_backend_config = {'cls': 'cassandra', 'hosts': ['127.0.0.1'], 'journal_writer': {'cls': 'memory'}, 'keyspace': 'b450d37fedbc10b9b0a7', ...}
22,982 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_kafka_writer::test_storage_direct_writer
kafka_prefix = 'lzvdwfoflx', kafka_server = '127.0.0.1:37911' consumer = <cimpl.Consumer object at 0x7f913d93f228>
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_content
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin_visit
View Full Test Results (2 Failed · 1,055 Passed · 39 Skipped)

Event Timeline

vsellier created this revision.Jul 6 2021, 4:58 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJul 6 2021, 4:58 PM
vsellier edited the summary of this revision. (Show Details)Jul 6 2021, 4:59 PM
vsellier added a task: T3396: cassandra - allow to configure the consistency level used by the queries.
vsellier updated this revision to Diff 21524.Jul 6 2021, 4:59 PM

Fix the task number in the commit message

swh-public-ci added a comment.Jul 6 2021, 5:06 PM

Build has FAILED

Patch application report for D5974 (id=21523)

Rebasing onto f1cac4fcaa...

Current branch diff-target is up to date.
Changes applied before test
commit 232e76b45788106d6e03d2d0f990347c8f1560a2
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Jul 6 16:54:57 2021 +0200

    cassandra: Allow to configure the consistency level to use
    
    The default ONE level is used to keep the previous behaviour
    
    Related to 3396

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1349/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1349/console

Harbormaster failed remote builds in B22496: Diff 21523!Jul 6 2021, 5:06 PM
vsellier updated this revision to Diff 21525.Jul 6 2021, 5:12 PM

make the tests happy

swh-public-ci added a comment.Jul 6 2021, 5:13 PM

Build has FAILED

Patch application report for D5974 (id=21524)

Rebasing onto f1cac4fcaa...

Current branch diff-target is up to date.
Changes applied before test
commit 22a41b088405b611719a6623ed98e8ecf3769b64
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Jul 6 16:54:57 2021 +0200

    cassandra: Allow to configure the consistency level to use
    
    The default ONE level is used to keep the previous behaviour
    
    Related to T3396

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1350/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1350/console

Harbormaster failed remote builds in B22497: Diff 21524!Jul 6 2021, 5:13 PM
swh-public-ci added a comment.Jul 6 2021, 5:21 PM

Build is green

Patch application report for D5974 (id=21525)

Rebasing onto f1cac4fcaa...

Current branch diff-target is up to date.
Changes applied before test
commit 936001b343d68057069199da2ef06b609b59adc0
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Jul 6 16:54:57 2021 +0200

    cassandra: Allow to configure the consistency level to use
    
    The default ONE level is used to keep the previous behaviour
    
    Related to T3396

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1351/ for more details.

Harbormaster completed remote builds in B22498: Diff 21525.Jul 6 2021, 5:21 PM
vsellier requested review of this revision.Jul 6 2021, 5:21 PM
ardumont added subscribers: vlorentz, ardumont.Jul 7 2021, 11:33 AM

lgtm, one question and suggestions inline.

swh/storage/cassandra/cql.py
84

that's not a logging instruction so you can use f-string here ;)

106

Shouldn't this be consistent with the execution profile the CqlRunner uses?

No idea...

After checking which part is using this, it seems to be mostly a test utility function so i'm confused...
And there is no docstring for that function...

@vlorentz any hints?

swh/storage/tests/test_cassandra.py
251

Not to check the full log of the exception but to check the at the very least, it raises for the right reason.

ardumont accepted this revision as: ardumont.Jul 7 2021, 11:34 AM
ardumont added 1 blocking reviewer(s): Reviewers.
ardumont added inline comments.Jul 7 2021, 11:36 AM
swh/storage/cassandra/cql.py
106

@vsellier showed me a use case [1]

Still, it'd be best this function clearly explains it's administration related in its docstring (outside the scope of this diff)

[1] https://forge.softwareheritage.org/source/snippets/browse/master/sysadmin/grid5000/cassandra/ansible/templates/swhstorage/init-cassandra-keyspace.sh

vsellier added inline comments.Jul 7 2021, 11:40 AM
swh/storage/cassandra/cql.py
106

IMO it's not necessary at it's only to bootstrap the database.
It certainly needs some cares to include this in the standard initialization process.

vsellier added inline comments.Jul 7 2021, 12:59 PM
swh/storage/cassandra/cql.py
106

my previous comment was the reply to

Shouldn't this be consistent with the execution profile the CqlRunner uses?

vsellier updated this revision to Diff 21532.Jul 7 2021, 2:30 PM
  • use a fstring to build the error message
  • check the error message content
ardumont added inline comments.Jul 7 2021, 2:32 PM
swh/storage/cassandra/cql.py
106

yes, thanks, i understood it that way ;)

swh-public-ci added a comment.Jul 7 2021, 2:38 PM

Build is green

Patch application report for D5974 (id=21532)

Rebasing onto f1cac4fcaa...

Current branch diff-target is up to date.
Changes applied before test
commit 9747aed6cbf1b31b87a8c3d00fe0177da357d056
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Jul 6 16:54:57 2021 +0200

    cassandra: Allow to configure the consistency level to use
    
    The default ONE level is used to keep the previous behaviour
    
    Related to T3396

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1352/ for more details.

Harbormaster completed remote builds in B22507: Diff 21532.Jul 7 2021, 2:38 PM
ardumont removed a reviewer: Reviewers.Jul 7 2021, 4:47 PM
This revision is now accepted and ready to land.Jul 7 2021, 4:47 PM
ardumont accepted this revision.Jul 7 2021, 4:47 PM

lgtm

Closed by commit rDSTO9747aed6cbf1: cassandra: Allow to configure the consistency level to use (authored by vsellier). · Explain WhyJul 7 2021, 6:21 PM
This revision was automatically updated to reflect the committed changes.
vsellier added a commit: rDSTO9747aed6cbf1: cassandra: Allow to configure the consistency level to use.

Revision Contents
Changeset List

PathSize
swh/
storage/
cassandra/
35 lines
7 lines
tests/
25 lines

Diff 21524

View Options

swh/storage/cassandra/cql.py

Loading...
View Options

swh/storage/cassandra/storage.py

Loading...
View Options

swh/storage/tests/test_cassandra.py

Loading...
About · Developer information · Legal