blackify all module, rather than just test_vault.py

In T2589#49073, @anlambert wrote:

You are right, they are not stored in database but there is a storage.origin_get_by_sha1 method.

LGTM

In T2589#48566, @anlambert wrote:

• We can process swh-graph responses to enrich the data (notably get origin urls from their sha1 or turn swhids into dicts) and returns them in JSON format

(I was initially surprised by the mixing together of "import" lines with "from ... import" ones, but upon reflection it makes a lot of sense, because one might have to switch between the two forms, and it's silly to have to move the line back and forth between import blocks when that happens.)

this has been addressed, and in a more general way that works for any SWHID, in D2582 by @DanSeraf

Very likely the same issue, thanks @ardumont !
Given what @olasd said in that issue (the ingestion logic having remained pretty much the same since ever), and that I can confirm linux.git was loading just fine on my laptop no more than a year ago, the increased memory usage probably comes from elsewhere.
Anyway, it looks like a potentially important issue, so I'm raising priority and also removing the association with the docker env (as you could also reproduce this on staging).

I think this is (reasonably) done now, please check and close it.

We have now a newer version of the compressed graph (2020-05-20), but it's not yet running on granet (I *think*, and, lacking T2579, I haven't checked).
Please make granet run that version of this task and close this task. (Or just close this task if it's already done.)

looks great in general!
just a few nits here and there (and possibly a separate issue to file for the sane default part)

In T2577#48486, @vsellier wrote:

An email was sent on the swh-devel mailing list to ask for reviews.
The deployment in production will be performed in the middle of week 38 is no problems are raised.

Re supported VCSs: sure, but I'd start with git that is a low-hanging fruit.

This seems to have been addressed with the path qualifier in SWHIDs.
Closing.
(Please reopen if I'm missing something.)

As an update: a feature equivalent to this one has been implemented in swh-scanner by @DanSeraf.
I guess it would still be useful to have (as it seems like a natural need) also in swh-identify, but of course the code should not be duplicated.

Noting down that I had a tentative very preliminary implementation in the feature/fuse branch of swh-graph; see in particular fuse.py there.
It's probably no worth picking up and we should restart from scratch at this point, but might still contain useful material.
(The webclient in there has since become a proper thing, see T2279. So that part is definitely obsolete.)

I've no specific advice here, as I wasn't aware that shorturl/SLUG existed, so I don't know where it might be used.
As a general comment, using such a "memorable" shorturl for a blog post doesn't seem like a good idea, as blog posts age pretty quickly.
So I'm in favor of removing it, but don't know how to evaluate the impact of doing so.
(For what is worth: I don't think that using it as a URI in the context you need it in is incompatible with having it being a valid URL pointing to something else, but I agree it would be weird, because some people will load it in their browsers for trying it out.)

(tag removed by mistake, adding it back)

Thanks for the pointers @ardumont !

nice refactoring work !

I think it's related to the shm trick.

the java/ part needs porting too