Page MenuHomeSoftware Heritage

in_memory: Add class Table, which emulates a Cassandra table.
ClosedPublic

Authored by vlorentz on Aug 12 2020, 6:01 PM.

Details

Summary

It will be used to implement the in-memory storage as a backend for the
cassandra storage.

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D3766 (id=13251)

Could not rebase; Attempt merge onto 6675286c4f...

Updating 6675286c..d9bfed69
Fast-forward
 swh/storage/cassandra/cql.py        | 17 ++++----
 swh/storage/cassandra/model.py      | 35 +++++++++++++++-
 swh/storage/in_memory.py            | 80 +++++++++++++++++++++++++++++++++++++
 swh/storage/tests/test_in_memory.py | 56 +++++++++++++++++++++++++-
 4 files changed, 176 insertions(+), 12 deletions(-)
Changes applied before test
commit d9bfed69178e07f882bade35b50fa99a2130211c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:24:33 2020 +0200

    in_memory: Add class Table, which emulates a Cassandra table.
    
    It will be used to implement the in-memory storage as a backend for the
    cassandra storage.

commit fc80a7f06ae8e1fccbe3ce2ee7576d2e9c5ff93a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:16:44 2020 +0200

    cassandra.cql: Fix return type of stat_counters.

commit 267f48024bfd257e91aa95f6e1ae99320587add7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:16:13 2020 +0200

    cassandra.model: Add PARTITION_KEY and CLUSTERING_KEY to the model classes.
    
    They will be used by the in-mem implementation of CqlRunner.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/736/ for more details.

Build is green

Patch application report for D3766 (id=13274)

Could not rebase; Attempt merge onto 6675286c4f...

Updating 6675286c..30d65e6b
Fast-forward
 swh/storage/cassandra/cql.py        | 17 +++----
 swh/storage/cassandra/model.py      | 35 +++++++++++++-
 swh/storage/cassandra/storage.py    | 45 +++++++++---------
 swh/storage/in_memory.py            | 91 ++++++++++++++++++++++++++++++++++++-
 swh/storage/tests/test_in_memory.py | 62 ++++++++++++++++++++++++-
 swh/storage/tests/test_storage.py   | 56 ++++++++++++++++++++++-
 6 files changed, 270 insertions(+), 36 deletions(-)
Changes applied before test
commit 30d65e6bd9e290e2eb0b1e029a93be2428fb33fa
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:24:33 2020 +0200

    in_memory: Add class Table, which emulates a Cassandra table.
    
    It will be used to implement the in-memory storage as a backend for the
    cassandra storage.

commit a0037c5d12a5694bfab749896db8546b7176b633
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:16:44 2020 +0200

    cassandra.cql: Fix return type of stat_counters.

commit f53cc6fc8606e9f97af0c3eea0ce0d7ec46d486a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:16:13 2020 +0200

    cassandra.model: Add PARTITION_KEY and CLUSTERING_KEY to the model classes.
    
    They will be used by the in-mem implementation of CqlRunner.

commit dd320a63b32b0e79c6f90a83236d607093219359
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 20:25:23 2020 +0200

    cassandra: Make origin_visit_get_latest filter using any status of a visit, instead of just the last.
    
    This fixes a mismatch in behavior with the pg and the in-mem storages

commit 1d45c5a0debc2e45ceee5effe565aab36905eb1c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 19:08:17 2020 +0200

    cassandra: Fix wrong algo reported in HashCollision, because of variable shadowing.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/759/ for more details.

anlambert added a subscriber: anlambert.

Looks good to me.

swh/storage/in_memory.py
179

of a row, in a row ?

This revision is now accepted and ready to land.Aug 14 2020, 11:48 AM

Build is green

Patch application report for D3766 (id=13321)

Could not rebase; Attempt merge onto 6675286c4f...

Updating 6675286c..20971864
Fast-forward
 swh/storage/cassandra/cql.py        | 17 +++----
 swh/storage/cassandra/model.py      | 35 +++++++++++++-
 swh/storage/cassandra/storage.py    | 47 +++++++++----------
 swh/storage/in_memory.py            | 91 ++++++++++++++++++++++++++++++++++++-
 swh/storage/tests/test_in_memory.py | 62 ++++++++++++++++++++++++-
 swh/storage/tests/test_storage.py   | 66 +++++++++++++++++++++++++--
 6 files changed, 278 insertions(+), 40 deletions(-)
Changes applied before test
commit 20971864bb1638b22bef00c79c7f0cb1be8afeed
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:24:33 2020 +0200

    in_memory: Add class Table, which emulates a Cassandra table.
    
    It will be used to implement the in-memory storage as a backend for the
    cassandra storage.

commit ef0600539bb7716380791e9c49619387c30407f4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:16:44 2020 +0200

    cassandra.cql: Fix return type of stat_counters.

commit 1266b6a7fe5006746e579e1ec28b4fb600b1188e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:16:13 2020 +0200

    cassandra.model: Add PARTITION_KEY and CLUSTERING_KEY to the model classes.
    
    They will be used by the in-mem implementation of CqlRunner.

commit 3dc69aaa42e76e373c4d23c1e8d98af5a804970f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 20:25:23 2020 +0200

    cassandra: Make origin_visit_get_latest filter using any status of a visit, instead of just the last.
    
    This fixes a mismatch in behavior with the pg and the in-mem storages

commit 006eeecaba7523f407fa0e253d6c832ea246d959
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 19:08:17 2020 +0200

    cassandra: Fix wrong algo reported in HashCollision, because of variable shadowing.

commit da287313765da62d34dc0dca7c22dbd61f40504f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 14 15:15:54 2020 +0200

    cassandra: Fix content_missing_per_sha1 when its parameter has length != 1.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/793/ for more details.