Page MenuHomeSoftware Heritage
Differential D6269

cassandra: Make _content_get_from_hashes run concurrently
ClosedPublic
Actions

Authored by vlorentz on Sep 15 2021, 3:13 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Commits
rDSTO61e9e4a3232f: cassandra: Make _content_get_from_hashes run concurrently
Summary

This is used by directory_ls and content_get.

Diff Detail

Repository
rDSTO Storage manager
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 23634
Build 36886: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 36885: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
23,122 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_kafka_writer::test_storage_direct_writer
kafka_prefix = 'paamwyezla', kafka_server = '127.0.0.1:39785' consumer = <cimpl.Consumer object at 0x7f133e62f5e8>
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_content
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin_visit
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_raw_extrinsic_metadata
View Full Test Results (1 Failed · 1,087 Passed · 40 Skipped)

Event Timeline

vlorentz created this revision.Sep 15 2021, 3:13 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptSep 15 2021, 3:13 PM
swh-public-ci added a comment.Sep 15 2021, 3:20 PM

Build has FAILED

Patch application report for D6269 (id=22698)

Rebasing onto 589d20ed64...

Current branch diff-target is up to date.
Changes applied before test
commit ac93773a14e094341878779eb50cbc4406871a9e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Sep 15 15:13:09 2021 +0200

    cassandra: Make _content_get_from_hashes run concurrently
    
    This is used by directory_ls and content_get.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1410/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1410/console

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 15 2021, 3:20 PM
Harbormaster failed remote builds in B23631: Diff 22698!
vlorentz updated this revision to Diff 22701.Sep 15 2021, 3:46 PM

fix tests

swh-public-ci added a comment.Sep 15 2021, 3:54 PM

Build has FAILED

Patch application report for D6269 (id=22701)

Rebasing onto 589d20ed64...

Current branch diff-target is up to date.
Changes applied before test
commit 69a828f32088392548258636336ed83147a52c60
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Sep 15 15:13:09 2021 +0200

    cassandra: Make _content_get_from_hashes run concurrently
    
    This is used by directory_ls and content_get.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1411/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1411/console

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 15 2021, 3:54 PM
Harbormaster failed remote builds in B23634: Diff 22701!
vsellier mentioned this in T3573: [cassandra] directory and content read benchmarks.Sep 15 2021, 5:12 PM
vlorentz updated this revision to Diff 22755.Sep 16 2021, 3:26 PM

rerun black

swh-public-ci added a comment.Sep 16 2021, 3:47 PM

Build was aborted

Patch application report for D6269 (id=22755)

Rebasing onto 9465054ef0...

First, rewinding head to replay your work on top of it...
Applying: cassandra: Make _content_get_from_hashes run concurrently
Changes applied before test
commit e5d5aacb5297552c47c2473d496dfa9d6b7d107a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Sep 15 15:13:09 2021 +0200

    cassandra: Make _content_get_from_hashes run concurrently
    
    This is used by directory_ls and content_get.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1417/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1417/console

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 16 2021, 3:47 PM
Harbormaster failed remote builds in B23687: Diff 22755!
swh-public-ci added a comment.Sep 16 2021, 4:20 PM

Build is green

Patch application report for D6269 (id=22755)

Rebasing onto 9465054ef0...

First, rewinding head to replay your work on top of it...
Applying: cassandra: Make _content_get_from_hashes run concurrently
Changes applied before test
commit 543f04e18298d38dc09c5574c0dc39294cace44a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Sep 15 15:13:09 2021 +0200

    cassandra: Make _content_get_from_hashes run concurrently
    
    This is used by directory_ls and content_get.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1418/ for more details.

Harbormaster completed remote builds in B23687: Diff 22755.Sep 16 2021, 4:20 PM
vlorentz requested review of this revision.Sep 16 2021, 4:20 PM
vlorentz updated this revision to Diff 22882.Sep 17 2021, 5:38 PM

rebase

swh-public-ci added a comment.Sep 17 2021, 5:45 PM

Build is green

Patch application report for D6269 (id=22882)

Rebasing onto 9465054ef0...

Current branch diff-target is up to date.
Changes applied before test
commit 2fed9491d7feb8ab9c887c6fdb6f16e32fe79aae
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Sep 15 15:13:09 2021 +0200

    cassandra: Make _content_get_from_hashes run concurrently
    
    This is used by directory_ls and content_get.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1419/ for more details.

Harbormaster completed remote builds in B23810: Diff 22882.Sep 17 2021, 5:45 PM
vlorentz updated this revision to Diff 22948.Sep 21 2021, 6:14 PM

rebase

This revision was not accepted when it landed; it landed in state Needs Review.Sep 21 2021, 6:14 PM
This revision was landed with ongoing or failed builds.
Closed by commit rDSTO61e9e4a3232f: cassandra: Make _content_get_from_hashes run concurrently (authored by vlorentz). · Explain Why
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSTO61e9e4a3232f: cassandra: Make _content_get_from_hashes run concurrently.
swh-public-ci added a comment.Sep 21 2021, 6:23 PM

Build is green

Patch application report for D6269 (id=22948)

Rebasing onto 59e63db65c...

Current branch diff-target is up to date.
Changes applied before test
commit 61e9e4a3232f1f88c5cfce353b39c53316949a5e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Sep 15 15:13:09 2021 +0200

    cassandra: Make _content_get_from_hashes run concurrently
    
    This is used by directory_ls and content_get.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1421/ for more details.

Harbormaster completed remote builds in B23879: Diff 22948.Sep 21 2021, 6:23 PM

Revision Contents
Changeset List

PathSize
swh/
storage/
cassandra/
9 lines
20 lines
6 lines
tests/
12 lines
13 lines

Diff 22701

View Options

swh/storage/cassandra/cql.py

Loading...
View Options

swh/storage/cassandra/storage.py

Loading...
View Options

swh/storage/in_memory.py

Loading...
View Options

swh/storage/tests/test_cassandra.py

Loading...
View Options

swh/storage/tests/test_cassandra_migration.py

Loading...
About · Developer information · Legal