In D5427#137970, @vlorentz wrote:

The new way of doing things is a lot more natural thing to do since we already have the MPH and the .order file

Newcomers aren't aware of this. I had no idea we had those before reading this diff.

I'm not saying the current state of the docs is good enough, I'm saying this commit message doesn't explain the design but why we're moving away from the old binary search solution. The new way of doing things is a lot more natural thing to do since we already have the MPH and the .order file, so there's no need to document why the old solution was bad in the main docs.

Where are the .order and MPH computed?

Thanks for the review. I don't think this needs to be documented elsewhere, it just describes why we're doing the change. What should be documented instead is why we're using these data structures in the first place. Right now this is done sparsely in the different source files, and this commit updates the already existing documentation.

There's a problem with this diff, it's on an old java-only backend that isn't the one we use when we run swh graph rpc-serve. The one that is currently used is in python, at swh/graph/server/app.py

Rebase + fix phabricator incorrect ID

Merged in https://forge.softwareheritage.org/rDMOD9523be0552d822be617da77bf0d2ca2f479da572

kSJFGSLHDFSKJGHDKFJHGDKFJHG

Remove phabricator garbage

The ORC exporter is done, and it's likely that we won't provide CSV exports in the future, or we'll generate them from the ORC format.

I added unit tests and reworked the logic, and also addressed @olasd 's comment. Could you please rereview? :-)

typo

ORC exporter: avoid fromtimestamp(), use datetime() from epoch instead

• Add unit tests
• Refactor export paths in the base Exporter class

Rebase on master, include webgraph files

I'm realizing that this is missing the "simplified" step and needs more changes.

rebase

Add task name to commit message

Normalize .timestamp()

Landed, but phabricator doesn't seem to see it.

• journalprocessor: remove comment about deserialize_message overload being a 'hack'

• journalprocessor: also partition sqlite files by first byte
• SQLite on-disk set: disable journalling and synchronous mode
• tests: fix test_export_origin

• Exporter documentation fixes
• Journal processor: fetch offsets in parallel

Fix various coding errors and minor improvements