Differential D6263

wip: Optimization for cassandra extid reading to avoid an extra index
AbandonedPublicDraft
Actions

Authored by ardumont on Sep 15 2021, 11:48 AM.

Details

Reviewers

vlorentz

Summary

wip because it does not work for now

Depends on D6249

Test Plan

tox (fails for now hence the wip stuff)

Diff Detail

Repository

rDSTO Storage manager

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 23608
Build 36839: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 36838: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	330 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_cassandra.TestCassandraStorage::test_extid_version_behavior self = <swh.storage.tests.test_cassandra.TestCassandraStorage object at 0x7fa13c466550> swh_storage = <swh.storage.cassandra.storage.CassandraStorage object at 0x7fa13c311c88> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7fa13c2005f8>
	23,131 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_kafka_writer::test_storage_direct_writer kafka_prefix = 'llueygjsrh', kafka_server = '127.0.0.1:52057' consumer = <cimpl.Consumer object at 0x7f9f85a6b5e8>
	4 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_content
	1 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin
	3 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin_visit
		View Full Test Results (2 Failed · 1,091 Passed · 40 Skipped)

Event Timeline

ardumont created this revision.Sep 15 2021, 11:48 AM

Build has FAILED

Patch application report for D6263 (id=22675)

Could not rebase; Attempt merge onto 589d20ed64...

Merge made by the 'recursive' strategy.
 swh/storage/cassandra/cql.py       | 60 ++++++++++++++++++++++++++++++++++++--
 swh/storage/cassandra/storage.py   | 36 ++++++++++++++++++-----
 swh/storage/in_memory.py           | 40 ++++++++++++++++++++++---
 swh/storage/interface.py           | 20 +++++++++++--
 swh/storage/postgresql/db.py       | 33 +++++++++++++++++----
 swh/storage/postgresql/storage.py  | 32 +++++++++++++++++---
 swh/storage/tests/storage_tests.py | 42 ++++++++++++++++++++++++++
 7 files changed, 238 insertions(+), 25 deletions(-)

Changes applied before test

commit 7b0ea794d2c06eda73fd47ee6784f592ecc0418c
Merge: 589d20ed 5c7412fd
Author: Jenkins user <jenkins@localhost>
Date:   Wed Sep 15 09:48:46 2021 +0000

    Merge branch 'diff-target' into HEAD

commit 5c7412fddf7391b25eb493e62007166bb27a0cf1
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Sep 15 11:47:22 2021 +0200

    wip: Optimization for cassandra extid reading to avoid an extra index
    
    wip because it does not work for now

commit fc3d21489bf75058604873fa3629cd78da10c5e2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Sep 13 14:25:34 2021 +0200

    Allow filtering extids per extid_version/extid_type when reading
    
    This impacts both the `extid_get_from_extid` and `extid_get_from_target` endpoints.
    
    Whe extid_version/extid_type are not provided, this keeps the existing behavior of
    returning all extids matching.
    
    Related to T3567

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1408/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1408/console

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 15 2021, 11:56 AM

Harbormaster failed remote builds in B23608: Diff 22675!

ardumont mentioned this in D6249: Allow filtering extids per extid_version/extid_type when reading.Sep 15 2021, 2:25 PM

Discarded as the version that works got integrated into D6249.