Update: Prefer to use an extid manifest instead of simply using sha256 checksum as it seems more robust.

Looks good to me, thanks !

Address @vlorentz comments

Update: Simplify implementation using classes (thanks to @vlorentz)

In D8798#228725, @vlorentz wrote:

Thanks, I like it.

Before going into the details, what do you think of D8800? it's a refactoring of this diff, using an abstract OriginSaveWebhookReceiver class, and each forge defines a subclass. This replaces calls to origin_save_webhook_receiver with lots of arguments and decorators. I didn't touch the tests at all.

Rebase

Update: Address @ardumont and @vlorentz comments

Update:

• rename some variables
• fix some typing
• harmonize replay.DirEditor.add_[file|directory] implementation

Rebase

Remove extra line break

Could you also test that raw extrinsic metadata have been correctly stored in the storage and can be retrieved ?
You can inspire from the cpan loader test to implement it.

Rebase

In D8569#228493, @franckbret wrote:

In D8569#228487, @franckbret wrote:

Rubygems: Improve loader to make use of artifacts and rubygems_metadata provided by the lister extra_loader_arguments

Use artifacts and rubygems_metadata to get list of versions, artifacts checksums and extrinsic metadata url
Add an EXTID manifest
Add metadata from extrinsic metadata

@anlambert Please note I used 'rubygems_metadata' instead of 'rubygem_metadata' as in the lister. Maybe I'm wrong but I think the lister should rename to rubygems_metadata?

Rebase

Only quote spaces as svn will also raise an assertion if we quote other special characters.

In D8782#228466, @olasd wrote:

I'm a bit confused about who is copying what in there, but I trust your judgment...

In T4665#97985, @anlambert wrote:

We are already using the https://pub.dev/api/package-names endpoint to list pubdev packages (see code), it is just the lister documentation that is outdated.

I will push a diff to fix the doc.

We are already using the https://pub.dev/api/package-names endpoint to list pubdev packages (see code), it is just the lister documentation that is outdated.

Rebase

Update safe chars list

@vlorentz, you misread the task, protocol should be http not https

@vlorentz, I do not from my side, link is fixed. I just forced browser cache to be invalidated for our WP site assets. Did it change something for you ?

In D8783#228307, @anlambert wrote:

Better like this indeed, could you add a test with a sample credentials config file to check github_session object has loaded credentials ?

LGTM

LGTM

LGTM, added some nitpicks about typing as inline comments.

This is the POST request I received from Bitbucket when pushing commit to a sample repository.

LGTM

Regarding JSON payload sent by SourceForge, we cannot guess the visit type (git, svn, or hg) from it so we will have to query the project endpoint from SourceForge REST API to get it.

In T4548#97884, @vlorentz wrote:

In T4548#97818, @anlambert wrote:

Based on the received HTTP POST request headers, we should be able to determine which type of forge sent us the request and then
extract repository URL and visit type from the JSON payload in order to create a Save Code Now request when new commits are pushed.

We can do that and it certainly works, but once we start allowing a single endpoint for all forges, we cannot go back.

What about a separate path for each forge type, so we have more flexibility later on? eg. to route/"cache" requests from some forges differently directly in Varnish

I am not a big fan either of adding more tarball detection heuristics but after quickly hacking on the code, it seems this is the only way to handle these edge case URLS (plus there is some cases when even analyzing HTTP headers cannot help to detect if the URL targets a tarball P1512). So let's land this and move on in deploying and testing that lister on staging.

This is the POST request I received from SourceForge when pushing commit to a sample hg repository.

This is the POST request I received from SourceForge when adding commit to a sample svn repository.

This is the POST request I received from SourceForge when pushing commit to a sample git repository.

This is the POST request I received from Gitea when pushing commit to a sample repository.

This is the POST request I received from GitLab when pushing commit to a sample repository.