Store the value of token(partition_key) in content_by_* table, instead of three hashes.
ClosedPublic
Actions

Authored by vlorentz on Mar 10 2020, 1:51 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T2304: Cassandra storage: Reduce the size of the "secondary lookup tables" for contents

Commits

rDSTOa24ab3f22bf8: Store the value of token(partition_key) in content_by_* table, instead of three…

Summary

That's a big win in terms of disk space, and shouldn't affect performance
negatively (there's only an extra query in content_add on sha1/sha1_git collisions)

Resolves T2304.

When this diff is accepted, I'll do a similar one for skipped_content.

Diff Detail

Repository

rDSTO Storage manager

Branch

content-murmur3

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 11286
Build 17062: tox-on-jenkins	Jenkins
Build 17061: arc lint + arc unit

Event Timeline

vlorentz created this revision.Mar 10 2020, 1:51 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptMar 10 2020, 1:51 PM

vlorentz edited the summary of this revision. (Show Details)Mar 10 2020, 1:53 PM

vlorentz added a task: T2304: Cassandra storage: Reduce the size of the "secondary lookup tables" for contents.

vlorentz edited the summary of this revision. (Show Details)

vlorentz edited the summary of this revision. (Show Details)Mar 10 2020, 1:55 PM

vlorentz edited the summary of this revision. (Show Details)

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/1042/ for more details.

Harbormaster completed remote builds in B11015: Diff 9944.Mar 10 2020, 1:58 PM

Sounds fine.

Remains to look up the tests.

Don't forget to drop the dead (and short-lived) code mentioned ;)

swh/storage/cassandra/storage.py
98	Drop row_to_content_hashes if it's no longer used ;)

ardumont accepted this revision.Mar 10 2020, 2:27 PM

This revision is now accepted and ready to land.Mar 10 2020, 2:27 PM

How do you migrate the tables now?

I'm waiting for the current replay to finish to get the sha1 collision stats. Then I'll drop the tables, recreate them, reset offsets on swh.journal.objects.content and restart the replay

rebase

As the cassandra cluster is paused indefinitely, I'm landing this diff now

This revision was landed with ongoing or failed builds.Mar 23 2020, 3:21 PM

Closed by commit rDSTOa24ab3f22bf8: Store the value of token(partition_key) in content_by_* table, instead of three… (authored by vlorentz). · Explain Why

This revision was automatically updated to reflect the committed changes.

vlorentz added a commit: rDSTOa24ab3f22bf8: Store the value of token(partition_key) in content_by_* table, instead of three….

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/1071/ for more details.

Harbormaster completed remote builds in B11286: Diff 10211.Mar 23 2020, 3:23 PM

vlorentz mentioned this in D2866: Store the value of token(partition_key) in skipped_content_by_* table, instead of three hashes..Mar 23 2020, 3:50 PM