I guess it is time to migrate the webapp database from SQLite to PostgreSQL.

Also the Save code now requests table is constantly growing and now contains more than 50000 lines, making the select requests quite slow.

I will check locally if the migration is straightforward or not.

Also the Save code now requests table is constantly growing and now contains more than 50000 lines, making the select requests quite slow.

Are you sure that's an SQLite issue? Looking at the models declared in swh/web/common/models.py, there doesn't seem to be any index at all.

-> D4815

Thanks for the indexes ! Regarding the OperationalError, maybe we could mitigate it with a retry process as in the storage ?

Other solution, increase timeout value for waiting on the database lock.

This really feels like the wrong solution to the problem... postgres is the right one

Also the Save code now requests table is constantly growing and now contains more than 50000 lines, making the select requests quite slow.

Turns out the slow requests was not due to the database but rather an implementation flaw while retrieving a save requests page. The fix is short but I realized that code is not covered so more work to do before submitting a diff.

In T2891#56061, @anlambert wrote:

Also the Save code now requests table is constantly growing and now contains more than 50000 lines, making the select requests quite slow.

Turns out the slow requests was not due to the database but rather an implementation flaw while retrieving a save requests page. The fix is short but I realized that code is not covered so more work to do before submitting a diff.

--> D4816

In T2891#56047, @vlorentz wrote:

This really feels like the wrong solution to the problem... postgres is the right one

I have created T2945 on the subject.