Just saw this search page on Zenodo: https://help.zenodo.org/guides/search/

The fix is now deployed and proxied graph responses are now properly streamed \o/

In T2891#56047, @vlorentz wrote:

This really feels like the wrong solution to the problem... postgres is the right one

In T2926#56128, @rdicosmo wrote:

Thanks Antoine, any way to have this kind of errors also reported in the admin dashboard for save code now.

Thanks Antoine, any way to have this kind of errors also reported in the admin dashboard for save code now.

For the record, the load failure on 2021-01-04T17:05:11Z was due to a network error (found via Kibana):

In T2891#56061, @anlambert wrote:

Also the Save code now requests table is constantly growing and now contains more than 50000 lines, making the select requests quite slow.

Turns out the slow requests was not due to the database but rather an implementation flaw while retrieving a save requests page. The fix is short but I realized that code is not covered so more work to do before submitting a diff.

Also the Save code now requests table is constantly growing and now contains more than 50000 lines, making the select requests quite slow.

This really feels like the wrong solution to the problem... postgres is the right one

Other solution, increase timeout value for waiting on the database lock.

Thanks for the indexes ! Regarding the OperationalError, maybe we could mitigate it with a retry process as in the storage ?

-> D4815

Also the Save code now requests table is constantly growing and now contains more than 50000 lines, making the select requests quite slow.

I guess it is time to migrate the webapp database from SQLite to PostgreSQL.

The repository has been correctly ingested on 05 January 2021, 11:56 UTC .

It looks like you already agree, but FWIW I'd also would like to have a dedicated (micro)service that keeps an up-to-date bloom filter for the entire archive, with a REST API.
It might be useful for other use cases (swh-scanner comes to mind, but I'm sure we'll find others as time passes).

In T2912#55860, @rdicosmo wrote:

In T2912#55849, @olasd wrote:

I think we should be able to decouple these counters completely from the loaders, and have them directly updated/handled by a client of the swh-journal. This would be a "centralized" component, but which we can parallelize quite heavily thanks to basic kafka design. We can also leverage the way kafka clients do parallelism to sidestep the locking issues arising in a potentially distributed filter.

Maybe my writing was not all that clear: I also had in mind a single centralised component (the ArchiveCounter) per Bloom filter, receiving the lists newcontents of ids from the loaders.
Getting the feed of ids from swh-journal instead of from the loaders is really neat: we avoid touching the loader code, and we gain a better capability of monitoring the load on the ArchiveCounter, so I'm all for it :-)

In T2912#55849, @olasd wrote:

Thanks for sketching out this proposal! It looks quite promising (and neat!).

I'm also having the "full journal" approach in mind after a quick reading of this neat proposal :-)

Thanks for sketching out this proposal! It looks quite promising (and neat!).

Updated the proposal with your suggestions, thanks!

In T2912#55487, @vlorentz wrote:

A Python library may be an issue, as it requires a central process with a global lock. Sharding by hash may fix the issue, though.

A Python library may be an issue, as it requires a central process with a global lock. Sharding by hash may fix the issue, though.

Nice, let me know when it's deployed and I will test if the problem is indeed fixed