I noticed that https://archive.softwareheritage.org/browse/origin/directory/?origin_url=deb://Ubuntu/packages/nginx has duplicate branch names, which is very confusing. In fact, even the default branch is repeated twice and I see two check marks. If we use branch names like 0.3.9-15.fc26, won't the same happen with Fedora listers? It doesn't seem to differentiate between the editions. (or does it?)

Build is green

• Send origins in the last iteration
• Change origin url format
• Use intrinsic version for pkg version key

I wonder whether we should merge this config file and that of the other script, so that we get all settings in one place.

What do you think?

Yes, i think so too. But which script, to be sure, manage_user_groups right?

I just started it that way to simplify the reading.
I think that i'll adapt according to what you proposed here then update the diff then we land.

In D8829#229627, @ardumont wrote:

Overall, that feels like luigi is a kinda "distributed makefile" (with a python dsl).

Add cli.py

one typo inline.

Adapt according to last remark, dump partial json updated project.

And fix some staging issues (after that first run):

@KShivendu , I added some inline comments to improve the loader output.

I'm not too sure about having a single "big" object as json output, as dumping that full json structure will only work out if the process didn't crash in the middle (and when that does, you won't really know what had happened until then). Maybe output one json object per project mutated instead?

• Manage only a list of namespaces and skip the other projects
• Make the script outputs json (for easier parsing)

Looks great, thanks.

In T4632#98216, @vlorentz wrote:

@anlambert What about non-Fedora RPM repositories? (RHEL, SUSE, Rocky Linux, ...)

@KShivendu thanks for the adaptations !

@anlambert What about non-Fedora RPM repositories? (RHEL, SUSE, Rocky Linux, ...)

buuuut you are using a strict inequality, so you need to subtract one day, in order not to miss uploads submitted after the previous run of the lister but on the same day.

Oh, I missed that you actually convert using .date() in the code, my bad.

lastUpload is documented as working on dates rather than datetime, so we are going to need to do the same stuff as with NuGet.

In T4675#98181, @anlambert wrote:

It seems the fix is to only encode the ? in an origin URL when it is provided as URL argument.

Looks great, thanks.

Adapt according to discussion and more

Actually for fedora, I found a better origin URL pattern: https://src.fedoraproject.org/rpms/{pkg_name}

In D8825#229537, @olasd wrote:

Looks nice!

I guess we may want to merge three levels of settings:

• global (default) settings
• per-namespace settings (for instance, to have all projects under infra/puppet/3rdparty have the same config overrides, e.g. disabling the issue tracker, changing the destination of webhooks to #swh-sysadm, etc.)
• and per-repository settings (in the case of visibility, it's quite obvious)

@franckbret, as explained in my inline comment we cannot use the date filtering on the release index of CPAN elasticsearch.

Looks nice!

Build is green

• Improvements suggested by @anlambert

Fixed and deployed.

It seems the fix is to only encode the ? in an origin URL when it is provided as URL argument.
The issue we hit in production might be related to the varnish cache I think.

Hmm, I do not hit the issue locally so that is why tests did not spot the issue. This seems to only happen in production.

Actually this is not related to the replica lag but it is a regression induced by recent commit rDWAPPS4cc9676a54cc368394c05b7f19c92ea072f8041e.

Ah right I noticed that behavior when fixing a recent bug in the webapp (D8820), will fix that asap.

@KShivendu I forgot to mention in my review that we should also get the checksums associated to a rpm archive, the loader will then use it to check download integrity.

@KShivendu , after testing the lister in docker environment there is room for improvements before we can accept that diff.

After reviewing and hacking on the fedora lister, I think we should use origin URL in the form https://packages.fedoraproject.org/pkgs/{src_pkg_name} for a fedora source package.