Add a keyspace_queue fixture to prepare keyspaces concurrently.
AbandonedPublic
Actions

Authored by vlorentz on Jan 31 2020, 4:06 PM.

Details

Reviewers

Group Reviewers

Summary

Keyspace creation + initialization with a schema takes about 60% of the
total time to run a test.
This new fixture runs keyspace creation concurrently to tests,
so other tests can run while a keyspace is being created.

Depends on D2566.

Diff Detail

Repository

rDSTO Storage manager

Branch

keyspace_queue

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 10412
Build 15499: tox-on-jenkins	Jenkins
Build 15498: arc lint + arc unit

Event Timeline

vlorentz created this revision.Jan 31 2020, 4:06 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJan 31 2020, 4:06 PM

Harbormaster failed remote builds in B10406: Diff 9345!Jan 31 2020, 4:47 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/947/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/947/console

vlorentz added a child revision: D2613: Tune Cassandra test config for lower test latency..Jan 31 2020, 5:10 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/949/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/949/console

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/951/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/951/console

Worth a try i guess.

The diff's job seems to fail though (triggered a rebuild from jenkins).

Trying to check the difference on my box as well.

This revision is now accepted and ready to land.Feb 3 2020, 10:29 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/952/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/952/console

Build has FAILED

i reproduce this here as well...

???
E cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'127.0.0.1:59493': OperationTimedOut('errors=None, last_host=None')})

Seems to be the root of the problem.
Initial tests seem to be fine up to the point where the connection refuses to establish again.

Also, that's not the only problem i had.
On another run which got further, hypothesis found a falsifiable sample...
Relatedly to timing apparently.

Falsifying example: test_add_arbitrary(
    self=<swh.storage.tests.test_cassandra.TestCassandraStorageGeneratedData at 0x7fed490336d8>,
    swh_storage=<swh.storage.cassandra.storage.CassandraStorage at 0x7fed0c2c47b8>,
    objects=[('content',
      Content(sha1=b'\t&\xc1\xd5!f\xd1O\x1879\xb3\xe0\x94\xeb\xcb\xf9d\x1bl', sha1_git=b'\xf6\x93\xe5\x00\xe2\xa4\xc4\xc9<\x12\x1d\x8d\x0eN\xb9\x01O\xca\x11<', sha256=b'\xaf\xbd\xc7\xf4Nt8@u\nQ\xcbN\x96|Q\xd1}\xf9]\xa8\xbf\xb6\x11"\xabHV\x16[\xa2\x85', blake2s256=b'\t\x07\xc2\xec\x8c \xd9`\x11\xc0\xe6\xe3\xf8\xa2\xd6\x95=\xf3\xccE\xff\xfa:\x1d\xbf\x08\xee\xf3q\x0b\xe4J', length=60, status='absent', reason='&\x1b', data=None, ctime=None)),
     ('origin_visit',
      OriginVisit(origin='http://m0w-.o0k-.iq59897v', date=datetime.datetime(593, 3, 28, 1, 1, 58, 553680), status='partial', type='\x0c"\x06\x08\x18!\U00062918\x1f\x19\U0009e77f', snapshot=b'', metadata=None, visit=351))],
)
Unreliable test timings! On an initial run, this test took 6727.64ms, which exceeded the deadline of 5000.00ms, but on a subsequent run it took 11.29 ms, which did not. If you expect this sort of variability in your test timings, consider turning deadlines off for this test by setting deadline=None.

It may have fallen on the case in-between keyspace creation so that took more time?

It just means Cassandra crashed in the middle of the tests.