Page MenuHomeSoftware Heritage

sqlite: Try to mitigate locked database when running cypress tests
ClosedPublic

Authored by anlambert on Aug 23 2022, 7:20 PM.

Details

Summary

Just testing if those errors still popup when building that diff on Jenkins using multiple jobs in parallel.

Diff Detail

Repository
rDWAPPS Web applications
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D8300 (id=29966)

Rebasing onto 8496d6a551...

Current branch diff-target is up to date.
Changes applied before test
commit 4bff6ddb4e341bf391ecb4be1fa2918455d4f4cb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Aug 23 19:18:26 2022 +0200

    sqlite: Try to mitigate locked database when running tests in parallel

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/2002/ for more details.

Build is green

Patch application report for D8300 (id=29966)

Rebasing onto 8496d6a551...

Current branch diff-target is up to date.
Changes applied before test
commit 4bff6ddb4e341bf391ecb4be1fa2918455d4f4cb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Aug 23 19:18:26 2022 +0200

    sqlite: Try to mitigate locked database when running tests in parallel

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/2003/ for more details.

Build is green

Patch application report for D8300 (id=29966)

Rebasing onto 8496d6a551...

Current branch diff-target is up to date.
Changes applied before test
commit 4bff6ddb4e341bf391ecb4be1fa2918455d4f4cb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Aug 23 19:18:26 2022 +0200

    sqlite: Try to mitigate locked database when running tests in parallel

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/2004/ for more details.

Have you tried PRAGMA journal_mode = WAL? It should allow running queries in parallel without locking like this. (and https://stackoverflow.com/a/6843199/539465 to apply it)

Actually, what about changing the DB path, so both tests instances don't use the same? :memory: might even work

Actually, what about changing the DB path, so both tests instances don't use the same? :memory: might even work

Actually, what about changing the DB path, so both tests instances don't use the same? :memory: might even work

Actually, reading tests configuration, Python tests use a postgres db while Cypress ones use a sqlite db so looks like
the locked database is due to the cypress tests execution only (we have some cypress tasks populating db dynamically,
they are supposed to be synchronous but my guess is that issues must come from here).

For Python tests, the postgresdb is created on the fly using pytest-postgresql magics, for cypress ones we cannot
use them unfortunately.

I will try the WAL journal approach then.

Hmm it's probably not good to have multiple tests using the same DB at the same time, but I don't see how to avoid it if it all runs in the same process :/

anlambert retitled this revision from sqlite: Try to mitigate locked database when running tests in parallel to sqlite: Try to mitigate locked database when running cypress tests.Aug 24 2022, 11:44 AM
anlambert edited the summary of this revision. (Show Details)

Yrying sqlite WAL journal approach.

Add comments and remove timeout config

This revision is now accepted and ready to land.Aug 24 2022, 12:06 PM

Build is green

Patch application report for D8300 (id=29972)

Rebasing onto 8496d6a551...

Current branch diff-target is up to date.
Changes applied before test
commit 03cdf5b4e72c6d965c79f8ca028b1dfd189ba17f
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Aug 24 11:44:05 2022 +0200

    sqlite: Try to mitigate locked database when running cypress tests

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/2006/ for more details.

Build is green

Patch application report for D8300 (id=29972)

Rebasing onto 8496d6a551...

Current branch diff-target is up to date.
Changes applied before test
commit 03cdf5b4e72c6d965c79f8ca028b1dfd189ba17f
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Aug 24 11:44:05 2022 +0200

    sqlite: Try to mitigate locked database when running cypress tests

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/2005/ for more details.

Build has FAILED

Patch application report for D8300 (id=29972)

Rebasing onto 8496d6a551...

Current branch diff-target is up to date.
Changes applied before test
commit 03cdf5b4e72c6d965c79f8ca028b1dfd189ba17f
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Aug 24 11:44:05 2022 +0200

    sqlite: Try to mitigate locked database when running cypress tests

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/2007/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/2007/console

Build is green

Patch application report for D8300 (id=29972)

Rebasing onto 8496d6a551...

Current branch diff-target is up to date.
Changes applied before test
commit 03cdf5b4e72c6d965c79f8ca028b1dfd189ba17f
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Aug 24 11:44:05 2022 +0200

    sqlite: Try to mitigate locked database when running cypress tests

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/2008/ for more details.

This revision was landed with ongoing or failed builds.Aug 24 2022, 12:11 PM
This revision was automatically updated to reflect the committed changes.

Build is green

Patch application report for D8300 (id=29974)

Rebasing onto 8496d6a551...

Current branch diff-target is up to date.
Changes applied before test
commit ddcfa6ce48c1d5c4e80044197fac576ba66ad7b1
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Aug 24 11:44:05 2022 +0200

    sqlite: Prevent locked database errors when running cypress tests

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/2009/ for more details.