In D8759#227732, @vlorentz wrote:

Hi, thanks for the diffs

This is now done, the objects are fixed in the production DB and kafka.

@vlorentz I'm running the following adaptation to your script:

Last analysis without [1]. That last diff should fix the key entries marked with the key 'only-version-should-be-tarball'.

I think detecting versions is a lost cause. Here is what I had to do for PyPI:

https://forge.softwareheritage.org/source/swh-storage/browse/master/swh/storage/migrate_extrinsic_metadata.py$234-281

Build is green

In D8773#228150, @vlorentz wrote:

plus, Nix and Guix already have the package name and version in their metadata, can't we ask them to provide this data to us?

Build is green

plus, Nix and Guix already have the package name and version in their metadata, can't we ask them to provide this data to us?

Yeah, I think that makes sense. Time for these tests to be written? :)

I think detecting versions is a lost cause. Here is what I had to do for PyPI: https://forge.softwareheritage.org/source/swh-storage/browse/master/swh/storage/migrate_extrinsic_metadata.py$234-281

Rebase

Improve the regexp version detection to be more restrictive.

must... resist... painting the bikeshed (git_manifest? git_object? raw_object?)

Oh yeah, I was thinking of just removing the entire project, but your solution also works.

Great, thanks!

Holes are bad. And I just opened a diff to make the git loader apply the same transformation, as @olasd made the same comment: D8776

Do you actually want to keep these objects? This would be inconsistent with the fixed loader behavior that would just reject those objects, and not load the repository at all.

This is the POST request I received from SourceForge when pushing commit to a sample hg repository.

This is the POST request I received from SourceForge when adding commit to a sample svn repository.

This is the POST request I received from SourceForge when pushing commit to a sample git repository.

I tried to add a workaround in the backfiller, but it is incredibly hard to do properly, especially as entries as disordered, so raw_manifest needs to be fixed in two different ways.

This is the POST request I received from Gitea when pushing commit to a sample repository.

This is the POST request I received from GitLab when pushing commit to a sample repository.

I used ngrok to forward webhook requests to my local machine.

Build is green

Build is green

Rebase on updated D8769

Variabilize task_name

Build is green

Add tests for the new absolute_cooldown

I started looking how to implement that task.

* sigh *

youmight want to add a variable for the value of f"load-{visit_type}", it's used four times now

\o/

Build is green

remove leftover marker

Build is green

Looks good to me, thanks !

Build is green

Build is green

Rebase