Neat simplification indeed, congrats \m/.

In D8944#232519, @vlorentz wrote:

Are you sure the path argument to add_directory cannot start with a / or contain ..?

huh, nice

Are you sure the path argument to add_directory cannot start with a / or contain ..?

For --staging, we just want to create a single oneshot full listing task.

Why bother with a full listing?

Since we now got the means to limit the listing, i'd use it. That's a faster feedback
loop and without stressing twice the upstream forge (one round for staging and another
for production at a relatively small intervals of time).

Maybe I need to configure the default to a more sensible defaults though. Like 3 pages
with 10 results (so that we can see that the pagination works too). Currently it's a tad
small (1 page of 15 or something).

Sure, that's what I meant: for staging, schedule a oneshot full listing task, but with the limiting and "origin disabling" options ("full" as opposed to "incremental")

Could you use a logger instance, and add if logger.isEnabledFor(logging.DEBUG): before logger.debug statements that use hash_to_hex?

In D8940#232501, @ardumont wrote:

For --staging, we just want to create a single oneshot full listing task.

Why bother with a full listing?

Since we now got the means to limit the listing, i'd use it. That's a faster feedback
loop and without stressing twice the upstream forge (one round for staging and another
for production at a relatively small intervals of time).

Maybe I need to configure the default to a more sensible defaults though. Like 3 pages
with 10 results (so that we can see that the pagination works too). Currently it's a tad
small (1 page of 15 or something).

So, I don't think we want to have to remember if there's full or incremental versions
of the lister.

Build is green

extra tests for pagination

So, I don't think we want to have to remember if there's full or incremental versions of the lister.

Build is green

Rebase

@rdicosmo: Can you share access to the myBox so I can add what I have from events?

Build is green

Adapt according to suggestion

d'oh at your suggestions, you're totally right!

(no test? worth it?)

A couple of suggestions inline, but lgtm otherwise, thanks!

In D8539#222941, @olasd wrote:

In D8539#222800, @ardumont wrote:

Fine, i've one comment i'd like others to have a look at though [1] regarding where
that new discovery (interface) code should go. It feels currently a bit off to me that this code
is in loader-core. Loaders are not the sole archive consumers (scanner, webapp, cli, indexer, cooker, ...).

[1] https://forge.softwareheritage.org/D8539?id=30916#inline-60774

@vlorentz @douardda @olasd @anlambert ^

If it's to be used generically, this discovery code should pretty definitely not be in swh.loader.core.

• The generic discovery algorithm, and base abstract classes/protocols, should probably be in swh.model, as they're tied to that structure;
• The swh.storage-based discovery mechanism could live in swh.storage.algorithms, and be used by swh.loader.core;
• The REST API-based discovery mechanism could live in swh.web.client, or stay in swh.scanner.

Build is green

I was sold on the description!

Adapt according to irc sysadm discussion

Build is green

Build has FAILED

Build is green

Build has FAILED

rebase

I'm tired

Build is green

Build has FAILED

rebase

remove useless function

rebase

less awful fix

In D8931#232231, @olasd wrote:

Why not just touch all the files?