Page MenuHomeSoftware Heritage
Differential D5582

cassandra: Add 'allow_overwrite' option, to allow updating objects
ClosedPublic
Actions

Authored by vlorentz on Apr 22 2021, 8:28 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Maniphest Tasks
T2602: Investigate how to upgrade the schema of the Cassandra storage
Commits
rDSTOf23346144dd3: cassandra: Add 'check_missing' option, to allow updating objects
Summary

as part of a migration.

Also write a first test that simulates how a simple migration would go.

This is the easy part of T2602.

Diff Detail

Repository
rDSTO Storage manager
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 20949
Build 32511: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 32510: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.algos.test_snapshot::test_snapshot_get_latest
swh_storage = <swh.storage.in_memory.InMemoryStorage object at 0x7fa91b401a58> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7fa90a911be0>
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.algos.test_snapshot::test_snapshot_get_latest_none
swh_storage = <swh.storage.in_memory.InMemoryStorage object at 0x7fa91bcc8208> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7fa919aeb898>
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.algos.test_snapshot::test_snapshot_id_get_from_revision
swh_storage = <swh.storage.in_memory.InMemoryStorage object at 0x7fa913dfa7b8> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7faa780ed240>
1,682 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.algos.test_snapshot::test_snapshot_large
swh_storage = <swh.storage.in_memory.InMemoryStorage object at 0x7fa9313cb400> @settings(suppress_health_check=function_scoped_fixture_check)
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.algos.test_snapshot::test_snapshot_resolve_alias
swh_storage = <swh.storage.in_memory.InMemoryStorage object at 0x7faa584a8710> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7fa9167a1208>
View Full Test Results (320 Failed · 487 Passed · 23 Skipped)

Event Timeline

vlorentz created this revision.Apr 22 2021, 8:28 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 22 2021, 8:28 PM
swh-public-ci added a comment.Apr 22 2021, 8:33 PM

Build has FAILED

Patch application report for D5582 (id=19943)

Rebasing onto eb8c147a54...

Current branch diff-target is up to date.
Changes applied before test
commit 5eddebc9accf868dc5974608755fc72f01dc9317
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 22 20:27:33 2021 +0200

    cassandra: Add 'check_missing' option, to allow updating objects
    
    as part of a migration.
    
    Also write a first test that simulates how a simple migration would go.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1265/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1265/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 22 2021, 8:33 PM
Harbormaster failed remote builds in B20948: Diff 19943!
vlorentz updated this revision to Diff 19944.Apr 22 2021, 8:36 PM

doc typos

swh-public-ci added a comment.Apr 22 2021, 8:40 PM

Build has FAILED

Patch application report for D5582 (id=19944)

Rebasing onto eb8c147a54...

Current branch diff-target is up to date.
Changes applied before test
commit 9ed28808477549ce75ad9dd7214ef03c88c690e5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 22 20:27:33 2021 +0200

    cassandra: Add 'check_missing' option, to allow updating objects
    
    as part of a migration.
    
    Also write a first test that simulates how a simple migration would go.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1266/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1266/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 22 2021, 8:40 PM
Harbormaster failed remote builds in B20949: Diff 19944!
vlorentz updated this revision to Diff 19951.Apr 23 2021, 9:49 AM

fix in-mem

swh-public-ci added a comment.Apr 23 2021, 9:55 AM

Build is green

Patch application report for D5582 (id=19951)

Rebasing onto eb8c147a54...

Current branch diff-target is up to date.
Changes applied before test
commit 879b240c123dc47afffc6bcb3840be32d6bf1eab
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 22 20:27:33 2021 +0200

    cassandra: Add 'check_missing' option, to allow updating objects
    
    as part of a migration.
    
    Also write a first test that simulates how a simple migration would go.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1267/ for more details.

Harbormaster completed remote builds in B20953: Diff 19951.Apr 23 2021, 9:55 AM
vlorentz requested review of this revision.Apr 23 2021, 9:55 AM
vlorentz added a child revision: D5584: cassandra: Add a test of a 'complex' migration, with a PK update.Apr 23 2021, 10:43 AM
vlorentz added a task: T2602: Investigate how to upgrade the schema of the Cassandra storage.Apr 26 2021, 10:25 AM
douardda added a subscriber: douardda.Apr 27 2021, 10:26 AM
douardda added inline comments.
swh/storage/cassandra/storage.py
186–207

why is the whole collision detection block is now run only if check_missing is True?

vlorentz added inline comments.Apr 27 2021, 10:44 AM
swh/storage/cassandra/storage.py
186–207

Because, if an object already exists with the exact same hashes and it was not filtered out initially, then self._content_get_from_hash(algo, content.get_hash(algo)) will return it and it would be detected as a collision.

I could filter the results of self._content_get_from_hash, but I don't think it's worth it, as _check_missing is pretty niche anyway.

vlorentz added inline comments.Apr 27 2021, 10:45 AM
swh/storage/cassandra/storage.py
186–207

And collision detection in Cassandra isn't guaranteed anyway, because two colliding objects may be inserted at the same time and Cassandra is only eventually consistent.

douardda requested changes to this revision.Apr 29 2021, 9:53 AM

I'm not fond of the 'check_missing' name for the argument, but would prefer 'allow_overwrite'.

Then, is the added test is nice, most of the _add methods are not tested when the config option is false. I'd like to see (stupid) simple tests for those.

This revision now requires changes to proceed.Apr 29 2021, 9:53 AM
vlorentz updated this revision to Diff 20126.Apr 29 2021, 10:17 AM

rename 'check_missing' to 'not allow_overwrite'

vlorentz planned changes to this revision.Apr 29 2021, 10:18 AM
swh-public-ci added a comment.Apr 29 2021, 10:23 AM

Build has FAILED

Patch application report for D5582 (id=20126)

Rebasing onto 98804f9e12...

First, rewinding head to replay your work on top of it...
Applying: cassandra: Add 'check_missing' option, to allow updating objects
Changes applied before test
commit bc0cd4eb0e765002790266b97f25e58b3946eb18
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 22 20:27:33 2021 +0200

    cassandra: Add 'check_missing' option, to allow updating objects
    
    as part of a migration.
    
    Also write a first test that simulates how a simple migration would go.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1279/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1279/console

Harbormaster failed remote builds in B21117: Diff 20126!Apr 29 2021, 10:23 AM
vlorentz updated this revision to Diff 20129.Apr 29 2021, 10:57 AM

fix in-mem

vlorentz planned changes to this revision.Apr 29 2021, 10:57 AM
swh-public-ci added a comment.Apr 29 2021, 11:01 AM

Build has FAILED

Patch application report for D5582 (id=20129)

Rebasing onto 98804f9e12...

First, rewinding head to replay your work on top of it...
Applying: cassandra: Add 'check_missing' option, to allow updating objects
Changes applied before test
commit d3ea8339115170edbdebe48c6586a6380d8725f3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 22 20:27:33 2021 +0200

    cassandra: Add 'check_missing' option, to allow updating objects
    
    as part of a migration.
    
    Also write a first test that simulates how a simple migration would go.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1281/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1281/console

Harbormaster failed remote builds in B21120: Diff 20129!Apr 29 2021, 11:01 AM
vlorentz updated this revision to Diff 20138.Apr 29 2021, 12:24 PM

add test

swh-public-ci added a comment.Apr 29 2021, 12:31 PM

Build has FAILED

Patch application report for D5582 (id=20138)

Rebasing onto 98804f9e12...

First, rewinding head to replay your work on top of it...
Applying: cassandra: Add 'check_missing' option, to allow updating objects
Changes applied before test
commit 185084abe76d6847189fc45b7ddd4529a1dc815d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 22 20:27:33 2021 +0200

    cassandra: Add 'check_missing' option, to allow updating objects
    
    as part of a migration.
    
    Also write a first test that simulates how a simple migration would go.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1283/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1283/console

Harbormaster failed remote builds in B21128: Diff 20138!Apr 29 2021, 12:31 PM
vlorentz updated this revision to Diff 20142.Apr 29 2021, 12:58 PM

fix doc?

swh-public-ci added a comment.Apr 29 2021, 1:06 PM

Build has FAILED

Patch application report for D5582 (id=20142)

Rebasing onto 98804f9e12...

First, rewinding head to replay your work on top of it...
Applying: cassandra: Add 'check_missing' option, to allow updating objects
Changes applied before test
commit da0768cbff30e00f23d07d56d0461b9997ec5927
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 22 20:27:33 2021 +0200

    cassandra: Add 'check_missing' option, to allow updating objects
    
    as part of a migration.
    
    Also write a first test that simulates how a simple migration would go.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1286/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1286/console

Harbormaster failed remote builds in B21131: Diff 20142!Apr 29 2021, 1:06 PM
vlorentz updated this revision to Diff 20144.Apr 29 2021, 2:15 PM

fix doc

swh-public-ci added a comment.Apr 29 2021, 2:22 PM

Build is green

Patch application report for D5582 (id=20144)

Rebasing onto 98804f9e12...

First, rewinding head to replay your work on top of it...
Applying: cassandra: Add 'check_missing' option, to allow updating objects
Changes applied before test
commit c5354b94fd069306df5daf6d09cc85ac82d415a6
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 22 20:27:33 2021 +0200

    cassandra: Add 'check_missing' option, to allow updating objects
    
    as part of a migration.
    
    Also write a first test that simulates how a simple migration would go.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1287/ for more details.

Harbormaster completed remote builds in B21133: Diff 20144.Apr 29 2021, 2:22 PM
vlorentz retitled this revision from cassandra: Add 'check_missing' option, to allow updating objects to cassandra: Add 'allow_overwrite' option, to allow updating objects.Apr 30 2021, 2:25 PM
douardda accepted this revision.May 3 2021, 2:14 PM
This revision is now accepted and ready to land.May 3 2021, 2:14 PM
vlorentz updated this revision to Diff 20230.May 3 2021, 5:41 PM

rebase

swh-public-ci added a comment.May 3 2021, 5:48 PM

Build is green

Patch application report for D5582 (id=20230)

Rebasing onto 92d551a4a5...

Current branch diff-target is up to date.
Changes applied before test
commit f23346144dd3032df0664b3a49b8fe254aed217f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 22 20:27:33 2021 +0200

    cassandra: Add 'check_missing' option, to allow updating objects
    
    as part of a migration.
    
    Also write a first test that simulates how a simple migration would go.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1297/ for more details.

Harbormaster completed remote builds in B21218: Diff 20230.May 3 2021, 5:48 PM
Closed by commit rDSTOf23346144dd3: cassandra: Add 'check_missing' option, to allow updating objects (authored by vlorentz). · Explain WhyMay 4 2021, 10:59 AM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSTOf23346144dd3: cassandra: Add 'check_missing' option, to allow updating objects.

Revision Contents
Changeset List

PathSize
swh/
storage/
cassandra/
145 lines
tests/
163 lines

Diff 19944

View Options

swh/storage/cassandra/storage.py

Loading...
View Options

swh/storage/tests/test_cassandra_migration.py

Loading...
About · Developer information · Legal