Page MenuHomeSoftware Heritage
Differential D3767

Make InMemoryStorage inherit from CassandraStorage.
ClosedPublic
Actions

Authored by vlorentz on Aug 12 2020, 6:01 PM.

Details

Reviewers
anlambert
Group Reviewers
Reviewers
Maniphest Tasks
T2519: Rewrite the in-mem storage as a backend for the cassandra storage
Commits
rDSTObc47283ddc8c: Make InMemoryStorage inherit from CassandraStorage.
Summary

This has no effect for now, other than deduplicating a method
and causing a name clash.

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Aug 12 2020, 6:01 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptAug 12 2020, 6:01 PM
vlorentz added a task: T2519: Rewrite the in-mem storage as a backend for the cassandra storage.Aug 12 2020, 6:06 PM
swh-public-ci added a comment.Aug 12 2020, 6:14 PM

Build is green

Patch application report for D3767 (id=13252)

Could not rebase; Attempt merge onto 6675286c4f...

Updating 6675286c..6a3ef9a9
Fast-forward
 swh/storage/cassandra/cql.py        |  17 +++---
 swh/storage/cassandra/model.py      |  35 +++++++++++-
 swh/storage/in_memory.py            | 111 ++++++++++++++++++++++++++++--------
 swh/storage/tests/test_in_memory.py |  56 +++++++++++++++++-
 swh/storage/writer.py               |   4 +-
 5 files changed, 184 insertions(+), 39 deletions(-)
Changes applied before test
commit 6a3ef9a907c76e8a99c06690af3dffb7d93318a8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:34:30 2020 +0200

    Make InMemoryStorage inherit from CassandraStorage.
    
    This has no effect for now, other than deduplicating a method
    and causing a name clash.

commit d9bfed69178e07f882bade35b50fa99a2130211c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:24:33 2020 +0200

    in_memory: Add class Table, which emulates a Cassandra table.
    
    It will be used to implement the in-memory storage as a backend for the
    cassandra storage.

commit fc80a7f06ae8e1fccbe3ce2ee7576d2e9c5ff93a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:16:44 2020 +0200

    cassandra.cql: Fix return type of stat_counters.

commit 267f48024bfd257e91aa95f6e1ae99320587add7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:16:13 2020 +0200

    cassandra.model: Add PARTITION_KEY and CLUSTERING_KEY to the model classes.
    
    They will be used by the in-mem implementation of CqlRunner.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/737/ for more details.

Harbormaster completed remote builds in B14515: Diff 13252.Aug 12 2020, 6:14 PM
vlorentz added a parent revision: D3766: in_memory: Add class Table, which emulates a Cassandra table..Aug 12 2020, 7:02 PM
vlorentz added a child revision: D3768: in_memory: Add InMemoryCqlRunner, a class that emulates cassandra.cql.CqlRunner without Cassandra..
vlorentz updated this revision to Diff 13275.Aug 12 2020, 8:40 PM

rebase

swh-public-ci added a comment.Aug 12 2020, 9:07 PM

Build is green

Patch application report for D3767 (id=13275)

Could not rebase; Attempt merge onto 6675286c4f...

Updating 6675286c..69cb5573
Fast-forward
 swh/storage/cassandra/cql.py        |  17 +++--
 swh/storage/cassandra/model.py      |  35 ++++++++++-
 swh/storage/cassandra/storage.py    |  45 +++++++-------
 swh/storage/in_memory.py            | 121 ++++++++++++++++++++++++++++--------
 swh/storage/tests/test_in_memory.py |  62 +++++++++++++++++-
 swh/storage/tests/test_storage.py   |  56 ++++++++++++++++-
 swh/storage/writer.py               |   4 +-
 7 files changed, 278 insertions(+), 62 deletions(-)
Changes applied before test
commit 69cb557387a8a5e490501c0f2bd9aec47762ee36
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:34:30 2020 +0200

    Make InMemoryStorage inherit from CassandraStorage.
    
    This has no effect for now, other than deduplicating a method
    and causing a name clash.

commit 30d65e6bd9e290e2eb0b1e029a93be2428fb33fa
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:24:33 2020 +0200

    in_memory: Add class Table, which emulates a Cassandra table.
    
    It will be used to implement the in-memory storage as a backend for the
    cassandra storage.

commit a0037c5d12a5694bfab749896db8546b7176b633
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:16:44 2020 +0200

    cassandra.cql: Fix return type of stat_counters.

commit f53cc6fc8606e9f97af0c3eea0ce0d7ec46d486a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:16:13 2020 +0200

    cassandra.model: Add PARTITION_KEY and CLUSTERING_KEY to the model classes.
    
    They will be used by the in-mem implementation of CqlRunner.

commit dd320a63b32b0e79c6f90a83236d607093219359
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 20:25:23 2020 +0200

    cassandra: Make origin_visit_get_latest filter using any status of a visit, instead of just the last.
    
    This fixes a mismatch in behavior with the pg and the in-mem storages

commit 1d45c5a0debc2e45ceee5effe565aab36905eb1c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 19:08:17 2020 +0200

    cassandra: Fix wrong algo reported in HashCollision, because of variable shadowing.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/760/ for more details.

Harbormaster completed remote builds in B14538: Diff 13275.Aug 12 2020, 9:07 PM
anlambert accepted this revision.Aug 14 2020, 11:49 AM
This revision is now accepted and ready to land.Aug 14 2020, 11:49 AM
vlorentz updated this revision to Diff 13322.Aug 14 2020, 4:07 PM

rebase

vlorentz updated this revision to Diff 13324.Aug 14 2020, 4:07 PM

rebase

swh-public-ci added a comment.Aug 14 2020, 4:39 PM

Build is green

Patch application report for D3767 (id=13322)

Could not rebase; Attempt merge onto 6675286c4f...

Updating 6675286c..bc47283d
Fast-forward
 swh/storage/cassandra/cql.py        |  17 +++--
 swh/storage/cassandra/model.py      |  35 ++++++++++-
 swh/storage/cassandra/storage.py    |  47 +++++++-------
 swh/storage/in_memory.py            | 121 ++++++++++++++++++++++++++++--------
 swh/storage/tests/test_in_memory.py |  62 +++++++++++++++++-
 swh/storage/tests/test_storage.py   |  66 ++++++++++++++++++--
 swh/storage/writer.py               |   4 +-
 7 files changed, 286 insertions(+), 66 deletions(-)
Changes applied before test
commit bc47283ddc8ca5397d8e25f25c670a9c4f21c435
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:34:30 2020 +0200

    Make InMemoryStorage inherit from CassandraStorage.
    
    This has no effect for now, other than deduplicating a method
    and causing a name clash.

commit 20971864bb1638b22bef00c79c7f0cb1be8afeed
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:24:33 2020 +0200

    in_memory: Add class Table, which emulates a Cassandra table.
    
    It will be used to implement the in-memory storage as a backend for the
    cassandra storage.

commit ef0600539bb7716380791e9c49619387c30407f4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:16:44 2020 +0200

    cassandra.cql: Fix return type of stat_counters.

commit 1266b6a7fe5006746e579e1ec28b4fb600b1188e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:16:13 2020 +0200

    cassandra.model: Add PARTITION_KEY and CLUSTERING_KEY to the model classes.
    
    They will be used by the in-mem implementation of CqlRunner.

commit 3dc69aaa42e76e373c4d23c1e8d98af5a804970f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 20:25:23 2020 +0200

    cassandra: Make origin_visit_get_latest filter using any status of a visit, instead of just the last.
    
    This fixes a mismatch in behavior with the pg and the in-mem storages

commit 006eeecaba7523f407fa0e253d6c832ea246d959
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 19:08:17 2020 +0200

    cassandra: Fix wrong algo reported in HashCollision, because of variable shadowing.

commit da287313765da62d34dc0dca7c22dbd61f40504f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 14 15:15:54 2020 +0200

    cassandra: Fix content_missing_per_sha1 when its parameter has length != 1.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/794/ for more details.

Harbormaster completed remote builds in B14585: Diff 13322.Aug 14 2020, 4:39 PM
vlorentz updated this revision to Diff 13340.Aug 14 2020, 4:39 PM

fix wrong commit being pushed

swh-public-ci added a comment.Aug 14 2020, 5:11 PM

Build is green

Patch application report for D3767 (id=13340)

Could not rebase; Attempt merge onto 6675286c4f...

Updating 6675286c..bc47283d
Fast-forward
 swh/storage/cassandra/cql.py        |  17 +++--
 swh/storage/cassandra/model.py      |  35 ++++++++++-
 swh/storage/cassandra/storage.py    |  47 +++++++-------
 swh/storage/in_memory.py            | 121 ++++++++++++++++++++++++++++--------
 swh/storage/tests/test_in_memory.py |  62 +++++++++++++++++-
 swh/storage/tests/test_storage.py   |  66 ++++++++++++++++++--
 swh/storage/writer.py               |   4 +-
 7 files changed, 286 insertions(+), 66 deletions(-)
Changes applied before test
commit bc47283ddc8ca5397d8e25f25c670a9c4f21c435
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:34:30 2020 +0200

    Make InMemoryStorage inherit from CassandraStorage.
    
    This has no effect for now, other than deduplicating a method
    and causing a name clash.

commit 20971864bb1638b22bef00c79c7f0cb1be8afeed
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:24:33 2020 +0200

    in_memory: Add class Table, which emulates a Cassandra table.
    
    It will be used to implement the in-memory storage as a backend for the
    cassandra storage.

commit ef0600539bb7716380791e9c49619387c30407f4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:16:44 2020 +0200

    cassandra.cql: Fix return type of stat_counters.

commit 1266b6a7fe5006746e579e1ec28b4fb600b1188e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 13:16:13 2020 +0200

    cassandra.model: Add PARTITION_KEY and CLUSTERING_KEY to the model classes.
    
    They will be used by the in-mem implementation of CqlRunner.

commit 3dc69aaa42e76e373c4d23c1e8d98af5a804970f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 20:25:23 2020 +0200

    cassandra: Make origin_visit_get_latest filter using any status of a visit, instead of just the last.
    
    This fixes a mismatch in behavior with the pg and the in-mem storages

commit 006eeecaba7523f407fa0e253d6c832ea246d959
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 12 19:08:17 2020 +0200

    cassandra: Fix wrong algo reported in HashCollision, because of variable shadowing.

commit da287313765da62d34dc0dca7c22dbd61f40504f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 14 15:15:54 2020 +0200

    cassandra: Fix content_missing_per_sha1 when its parameter has length != 1.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/801/ for more details.

Harbormaster completed remote builds in B14603: Diff 13340.Aug 14 2020, 5:11 PM
Closed by commit rDSTObc47283ddc8c: Make InMemoryStorage inherit from CassandraStorage. (authored by vlorentz). · Explain WhyAug 14 2020, 6:07 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSTObc47283ddc8c: Make InMemoryStorage inherit from CassandraStorage..

Revision Contents
Changeset List

PathSize
swh/
storage/
30 lines
4 lines

Diff 13361

View Options

swh/storage/in_memory.py

Loading...
View Options

swh/storage/writer.py

Loading...
About · Developer information · Legal