As @olasd pointed out, async is a keyword so let's not use it as module name :-)

In D4046#100083, @tenma wrote:

on another note, do I write tests for this cli?

trigger CI build

In D4041#99964, @vlorentz wrote:

Why do we need markdown support in the docs?

In T2640#49740, @moranegg wrote:

I agree on linking Gitorious to the wiki page, but we should find a consistency that can generate automatically the coverage page.

For whoever who want to work on this (and on doc restructuring in general), I recommend to watch this talk: https://www.youtube.com/watch?v=t4vKPhjcMZg , which provides a good taxonomy of the kind of docs that exist. Our GSoD 2019 application page also contains a lot of useful related work.

In D4017#99716, @tenma wrote:

The fact that is not so common a pattern does not make it bad.

we have rewritten this from scratch and we will commit it separately

Caveat: as this results in an API change (I think), it should be announced on the swh-devel mailing list before deployment.

In T2373#49214, @ardumont wrote:

fwiw, loader-core v0.11.0 deployed in production.

I can confirm that with the current master HEAD of swh-loader-core (452fa224f9ca635a979cf1a8e98c88bb560ca98a), loading of the Linux kernel repo no longer OOM.
(It failed after ~24 hours, but apparently for unrelated reasons.)

This is fine, and better than the status quo.
(We should in theory also mention the other applicable copyright notices at the beginning of the file, but we weren't doing that before either, so this change LGTM.)

this has been fixed a while ago by D2669

In D3983#98386, @anlambert wrote:

The repository is missing configuration for using black. Could you add the pre-commit hook and the black configuration in pyproject.toml as in other swh repos ?

• uniform tox.ini and setup.cfg with python module template

blackify all module, rather than just test_vault.py

In T2589#49073, @anlambert wrote:

You are right, they are not stored in database but there is a storage.origin_get_by_sha1 method.

LGTM

In T2589#48566, @anlambert wrote:

• We can process swh-graph responses to enrich the data (notably get origin urls from their sha1 or turn swhids into dicts) and returns them in JSON format

(I was initially surprised by the mixing together of "import" lines with "from ... import" ones, but upon reflection it makes a lot of sense, because one might have to switch between the two forms, and it's silly to have to move the line back and forth between import blocks when that happens.)

this has been addressed, and in a more general way that works for any SWHID, in D2582

Very likely the same issue, thanks @ardumont !
Given what @olasd said in that issue (the ingestion logic having remained pretty much the same since ever), and that I can confirm linux.git was loading just fine on my laptop no more than a year ago, the increased memory usage probably comes from elsewhere.
Anyway, it looks like a potentially important issue, so I'm raising priority and also removing the association with the docker env (as you could also reproduce this on staging).

I think this is (reasonably) done now, please check and close it.

We have now a newer version of the compressed graph (2020-05-20), but it's not yet running on granet (I *think*, and, lacking T2579, I haven't checked).
Please make granet run that version of this task and close this task. (Or just close this task if it's already done.)

looks great in general!
just a few nits here and there (and possibly a separate issue to file for the sane default part)

In T2577#48486, @vsellier wrote:

An email was sent on the swh-devel mailing list to ask for reviews.
The deployment in production will be performed in the middle of week 38 is no problems are raised.