I concur with @tenma here, that function made me raise eyebrows in the past.
I agree that making sure we are not / sensitive is a good usability feature, but that could be a oneliner somewhere that doesn't need a function.

the CI is now failing due to the lack of fuse kernel module in the test environment, I've filed T2654 about this

use case for is client code that just want to access the raw JSON answers, as it is the case for the FUSE implementation

I forgot to mention: sharding archive/ and meta/ as archive/XX/ and meta/XX/ is also an important one, but probably not a blocker for an initial merge into master.
However, it might be more convenient to implement it right away rather than having to add it later. Up to you.

This is amazing, I didn't expect it to reach this level of functionality so quickly, kudos! :-)

As @olasd pointed out, async is a keyword so let's not use it as module name :-)

In D4046#100083, @tenma wrote:

on another note, do I write tests for this cli?

trigger CI build

In D4041#99964, @vlorentz wrote:

Why do we need markdown support in the docs?

In T2640#49740, @moranegg wrote:

I agree on linking Gitorious to the wiki page, but we should find a consistency that can generate automatically the coverage page.

For whoever who want to work on this (and on doc restructuring in general), I recommend to watch this talk: https://www.youtube.com/watch?v=t4vKPhjcMZg , which provides a good taxonomy of the kind of docs that exist. Our GSoD 2019 application page also contains a lot of useful related work.

In D4017#99716, @tenma wrote:

The fact that is not so common a pattern does not make it bad.

we have rewritten this from scratch and we will commit it separately

Caveat: as this results in an API change (I think), it should be announced on the swh-devel mailing list before deployment.

In T2373#49214, @ardumont wrote:

fwiw, loader-core v0.11.0 deployed in production.

I can confirm that with the current master HEAD of swh-loader-core (452fa224f9ca635a979cf1a8e98c88bb560ca98a), loading of the Linux kernel repo no longer OOM.
(It failed after ~24 hours, but apparently for unrelated reasons.)

This is fine, and better than the status quo.
(We should in theory also mention the other applicable copyright notices at the beginning of the file, but we weren't doing that before either, so this change LGTM.)

this has been fixed a while ago by D2669

In D3983#98386, @anlambert wrote:

The repository is missing configuration for using black. Could you add the pre-commit hook and the black configuration in pyproject.toml as in other swh repos ?

• uniform tox.ini and setup.cfg with python module template