Page MenuHomeSoftware Heritage

Use swh-storage PostgreSQL backend in tests
Closed, MigratedEdits Locked

Description

swh-web tests historically use the memory backend for swh-storage as no pytest fixture was available at the time to easily use the PostgreSQL backend.

While using memory backend is convenient, it does not allow to catch PostgreSQL related issues that could occur in production.
For instance that sentry reported issue was due to an inconsistency between the memory and PostreSQL backend implementation.

Nowadays, there is a swh_storage fixture enabling to use PostgreSQL backend in tests. We should use it to align with production.

This would require to adapt the tests_data fixture and tests/data.py module implementation.

Event Timeline

anlambert triaged this task as Normal priority.Sep 23 2021, 11:25 AM
anlambert created this task.

Instead of running tests only with the pg backend, what about both the in-mem and pg, using a parametrized fixture?
This would allow catching discrepancies between backends, instead of accidentally relying on pg's.

And, as the in-mem backend is much faster, we could exclude the pg backend while iterating (eg. pytest -k 'not postgresql').

Instead of running tests only with the pg backend, what about both the in-mem and pg, using a parametrized fixture?
This would allow catching discrepancies between backends, instead of accidentally relying on pg's.

And, as the in-mem backend is much faster, we could exclude the pg backend while iterating (eg. pytest -k 'not postgresql').

Good idea, we could use the memory backend while developing as we currently do and run the tests with both backends on Jenkins.