Rebase and rename api_info to get_url_body

Rebase and update commit message

In D8495#220442, @vlorentz wrote:

api_info sounds like it returns information about an API. Could you rename it to get_url_body?

All swh-web features have been isolated in standalone django applications. This is now deployed so closing this.

Update years in license header

In D8489#220281, @vlorentz wrote:

I hope the memory usage won't be an issue :/

In D8489#220272, @ardumont wrote:

with a more costly

i think you meant *without* ;)

Nope, when using the docker-compose.search.yml, an elasticsearch service is created, which is clearly more costly than a memory backend.

ah yeah, ok got it.

By default, we are not using that file, hence why you integrated that search rpc service part inside the main docker-compose.
With this ^, We are using the default memory backend.

But if we want to use a real elasticsearch backend, we still can by using -f docker-compose.search.yml which will use a real es container (and backend).

In D8489#220269, @ardumont wrote:

ok

I guess the memory implementation is enough to reproduce the same behavior as the actual search service.

In D8489#220267, @ardumont wrote:

with a more costly

i think you meant *without* ;)

Update commit message

Rebase and fix typo in commit message

Add logger.exception call

Rename global variable

Rebase and update commit message

In D8458#219763, @vlorentz wrote:

Don't we actually want to keep it across runs? It's pretty big

Looks good to me, thanks !

Also I think it can be doable to totally remove the GIT part of the lister. The csv files have everything we need. For the incremental part a metadata.json file at the root of the archive a date and a commit hash that represents the date of the db dump.
In incremental case the lister can compare that date to the last update of the crate.

It seems the issue comes from the use of some before hooks,
turning them to beforeEach ones seems to fix the issue.

Looks good to me, just one missing return type to add.

• Add missing references to Hashbang in arch and aur loader grant
• Remove duplicated link for crates loader source code

Rebase

Looks good to me.

Thanks, before I can accept this, one inline comment to handle and the packaging dependency needs to be added in the requirements.txt file.