Page MenuHomeSoftware Heritage

wip: Optimization for cassandra extid reading to avoid an extra index
AbandonedPublicDraft

Authored by ardumont on Sep 15 2021, 11:48 AM.

Details

Reviewers
vlorentz
Summary

wip because it does not work for now

Depends on D6249

Test Plan

tox (fails for now hence the wip stuff)

Diff Detail

Repository
rDSTO Storage manager
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 23608
Build 36839: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 36838: arc lint + arc unit

Unit TestsFailed

TimeTest
330 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_cassandra.TestCassandraStorage::test_extid_version_behavior
self = <swh.storage.tests.test_cassandra.TestCassandraStorage object at 0x7fa13c466550> swh_storage = <swh.storage.cassandra.storage.CassandraStorage object at 0x7fa13c311c88> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7fa13c2005f8>
23,131 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_kafka_writer::test_storage_direct_writer
kafka_prefix = 'llueygjsrh', kafka_server = '127.0.0.1:52057' consumer = <cimpl.Consumer object at 0x7f9f85a6b5e8>
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_content
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin_visit
View Full Test Results (2 Failed · 1,091 Passed · 40 Skipped)

Event Timeline

Build has FAILED

Patch application report for D6263 (id=22675)

Could not rebase; Attempt merge onto 589d20ed64...

Merge made by the 'recursive' strategy.
 swh/storage/cassandra/cql.py       | 60 ++++++++++++++++++++++++++++++++++++--
 swh/storage/cassandra/storage.py   | 36 ++++++++++++++++++-----
 swh/storage/in_memory.py           | 40 ++++++++++++++++++++++---
 swh/storage/interface.py           | 20 +++++++++++--
 swh/storage/postgresql/db.py       | 33 +++++++++++++++++----
 swh/storage/postgresql/storage.py  | 32 +++++++++++++++++---
 swh/storage/tests/storage_tests.py | 42 ++++++++++++++++++++++++++
 7 files changed, 238 insertions(+), 25 deletions(-)
Changes applied before test
commit 7b0ea794d2c06eda73fd47ee6784f592ecc0418c
Merge: 589d20ed 5c7412fd
Author: Jenkins user <jenkins@localhost>
Date:   Wed Sep 15 09:48:46 2021 +0000

    Merge branch 'diff-target' into HEAD

commit 5c7412fddf7391b25eb493e62007166bb27a0cf1
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Sep 15 11:47:22 2021 +0200

    wip: Optimization for cassandra extid reading to avoid an extra index
    
    wip because it does not work for now

commit fc3d21489bf75058604873fa3629cd78da10c5e2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Sep 13 14:25:34 2021 +0200

    Allow filtering extids per extid_version/extid_type when reading
    
    This impacts both the `extid_get_from_extid` and `extid_get_from_target` endpoints.
    
    Whe extid_version/extid_type are not provided, this keeps the existing behavior of
    returning all extids matching.
    
    Related to T3567

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1408/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1408/console

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 15 2021, 11:56 AM
Harbormaster failed remote builds in B23608: Diff 22675!

Discarded as the version that works got integrated into D6249.