Does it mean we were silently dropping data until this? Which loaders use this?

So, overall, what I'm uncomfortable with is decoding incoming cursors without validating that we have output and generated them ourselves. They're currently untrusted user input, that we're decoding with a very simple algorithm, and it'd be easy enough for anyone to fiddle with them (even with the prefix/suffix addition).

In D8918#231834, @vlorentz wrote:

Could you add this check?

for prefix in ignored_project_prefixes:
    if not prefix.endswith("/"):
        raise ValueError("Prefixes must end with /")

Otherwise we are going to forget to add slashes at the end, and it will match too many namespaces.

Could you add this check?

ok then!

In D8914#231759, @olasd wrote:

In D8914#231729, @anlambert wrote:

Thanks, I will try to update the swh/lister/rubygems/data/small_rubygems_dump.sh script to generate a postgres dump that can be loaded when ERROR_ON_STOP is set.

There's not much point in doing that, the actual dump is generated with ownership instructions as well. I've wasted a good chunk of time trying to work around that but filtering their silly double-wrapped plain text SQL export is a PITA.

Build is green

Build is green

Rebase

Add explicit test for default behavior

Good idea but you need to update that conditional.

In D8914#231729, @anlambert wrote:

Thanks, I will try to update the swh/lister/rubygems/data/small_rubygems_dump.sh script to generate a postgres dump that can be loaded when ERROR_ON_STOP is set.

just something i don't get in the test, as per my question inline.

Thanks, I will try to update the swh/lister/rubygems/data/small_rubygems_dump.sh script to generate a postgres dump that can be loaded when ERROR_ON_STOP is set.

In D8909#231631, @vlorentz wrote:

@anlambert Shouldn't this be replaced by swh auth generate-token?

Build is green

rebase

Build is green

bug fix

Let's see how this goes: https://github.com/hexpm/hexpm/pull/1168

You need to pass complete_qs to requests_mock to actually test the query string: https://requests-mock.readthedocs.io/en/latest/matching.html#query-strings (and then you'll need to add the sort param, because it's missing from tests).

Build is green

Build was aborted

remove logs

Build is green

fix copyright header

Some screen capture of the differents scenario when login:

@anlambert Shouldn't this be replaced by swh auth generate-token?