well, it will even be simpler than that: let's get rid of those "dumps" and fix the db test fixture to build the db on the fly from the sql files found in the swh-<name>/sql directory.

well, it will even be simpler than that: let's get rid of those "dumps" and fix the db test fixture to build the db on the fly from the sql files found in the swh-<name>/sql directory.

In T1265#23549, @douardda wrote:

yes I'm aware of this logic in the makefile. Not sure yet what's the best to do with this. Move it up in swh-development?

In T1265#23548, @ardumont wrote:

Its main use is the sql generation for our multiple modules using db in their internals (storage, indexer, archiver, scheduler, scheduler-updater)...
See the root Makefile of that repository.

Its main use is the sql generation for our multiple modules using db in their internals (storage, indexer, archiver, scheduler, scheduler-updater)...
See the root Makefile of that repository.

For your information, we recently moved out of this repository the loader-svn and loader-tar's testdata.
So yes, this sounds reasonable.

in fact, let's kill this blob repo; in there we have:

potentially useful related tooling: http://carsonip.me/posts/debugging-memory-usage-python-tracemalloc/

agreed, they should be removed (I've updated the task title accordingly)

All the old visits have now been migrated to snapshots.

There's some work in progress by @zack in branch feature/schema-revamp of rDSTO, I'll pick that up to drive it to completion.

This is now done.

Thanks for your analysis.

The main issue with the current query is that the id sort happens after we had to fetch all the indexed results, even if we're only presenting a few of these results to the user. When searching gitbla, the bitmap index scan will find all entries containing the trigram git (so, almost everything), then recheck them for exact match, then finally sort the results.

db-init has been updated a while ago to force UTF8 encoding, and it is now the documented way to initialize the DB, so there is no need to further specify DB encoding requirements elsewhere

In T1194#22202, @olasd wrote:

However I'd be happy for the static default landing pages to tell people that, and to point to the (static) API documentation of the respective components.

The RPC client/server components have been made as trivial as possible and considering their presence in the critical path of everything we do, I think we should keep them that way. They're not meant to conform to REST or any other standards, and are not supposed to support browsing.

transforming the db.py function call into a call to a query server side.

How about just *not* sort by origin ID then?

How about just *not* sort by origin ID then?

(pinging this issue, because it's 2018, and it really looks bad that we're apparently not capable of quickly returning results in our main search :-))

So, that was a beast of a puppet refactoring, but the end result is: all our RPC servers are now nicely tucked behind an instance of nginx.

Here is a pcap of the issue observed while a Vault cooker was calling set_progress on the Vault backend. The BadStatusLine packet that only contains "\r\n" is packet 306.

The BadStatusLine error reared back its ugly head on the vault backend. It was reproducible. Putting a nginx in front of the vault backend fixed it. I guess it's time to puppetize the configuration of a http frontend to all our RPC servers.

To use the full features of jsonb, we indeed need the database encoding to be UTF8.

You can add swh.indexer.storage.db.Db to your list.

I'll do this tomorrow when i'm fresh.

I'll do this tomorrow when i'm fresh.

Well, strike POC, i had something that worked at that time.