Rebase

Rebase

Add commit fixing mypy errors with Python 3.7

The origin does not exist so elasticsearch is returning similar URLs to the search query as any search engine.

Remove last commit as it did not work

Add commit to fix mypy issue

Rebase

LGTM, added a couple of nitpicks as inline comments.

LGTM, been a while since I read java code, so verbose (especially for iterations).

Could you add a test checking luigi parameters are correctly passed to the subprocess.run instruction ?

In D8909#231701, @franckbret wrote:

In D8909#231631, @vlorentz wrote:

@anlambert Shouldn't this be replaced by swh auth generate-token?

@anlambert @vlorentz seems legit that anything related to auth for a cli command should be centralized in swh auth.

I can adapt to :

• make swh scanner depends on swh auth
• alias swh scanner login to one of swh auth
• add a set token command to swh auth

What do you think?

In D8920#231860, @vlorentz wrote:

Does it mean we were silently dropping data until this? Which loaders use this?

In D8914#231759, @olasd wrote:

In D8914#231729, @anlambert wrote:

Thanks, I will try to update the swh/lister/rubygems/data/small_rubygems_dump.sh script to generate a postgres dump that can be loaded when ERROR_ON_STOP is set.

There's not much point in doing that, the actual dump is generated with ownership instructions as well. I've wasted a good chunk of time trying to work around that but filtering their silly double-wrapped plain text SQL export is a PITA.

Thanks, I will try to update the swh/lister/rubygems/data/small_rubygems_dump.sh script to generate a postgres dump that can be loaded when ERROR_ON_STOP is set.

Rebase

Rebase

Fix docstring and doc build.

Rebase

Update apidoc for /origin/save/ endpoint with new webhook related fields

Drop visit_type label in prometheus gauge.

Fix docstring

Update: Add label for webhook origin in prometheus gauge

Update: also add column in save_origin_requests table indicating which forge type sent the webhook

In D8889#231323, @vlorentz wrote:

In D8889#231299, @anlambert wrote:

And would it make sense to store which webhook was used?

It can be easily guessed from the origin URL so I do not think we should add that info.

How? That works for Bitbucket and GitHub, but you can't tell whether "git.example.org" is Gitlab or Gitea

In D8886#231331, @vlorentz wrote:

@anlambert Looks like you should use a separate logger, so it integrates with the standard logging config, eg.

swh -l DEBUG -l swh.loader.svn.whatever:INFO loader run svn <repo_url>

to have all debug logs except from the swh.loader.svn.whatever logger

In D8889#231299, @anlambert wrote:

What is it going to be used for?

To compute some stats about usage of that feature.

And would it make sense to store which webhook was used?

It can be easily guessed from the origin URL so I do not think we should add that info.

What is it going to be used for?

In D8886#231281, @lunar wrote:

Could you explain why using loglevels would not work? It feels to me they are meant to solve the problem of selecting the appropriate amount of debug messages… Having a separate flag introduce quite some extra noise in the process.

Fixed and deployed.

Fix cypress test

Update

Handle all possible cooking statuses.

Rework fix to ensure cooking task will be displayed in vault UI.

In D8878#230891, @anlambert wrote:

In D8878#230887, @vlorentz wrote:

thanks!

Is it visible on the Downloads page even if someone else requested the cooking?

Ah right, need to check.