Thanks for taking this suggestion in stride!

In D8682#226117, @olasd wrote:

swh.model.from_disk.Directory has a collect method which is supposed to do the change tracking by itself (it only returns the nodes that have changed since the last time .collect() was called). This should allow you to drop the modified_paths tracking altogether.

swh.model.from_disk.Directory has a collect method which is supposed to do the change tracking by itself (it only returns the nodes that have changed since the last time .collect() was called). This should allow you to drop the modified_paths tracking altogether.

does ballooning interact correctly with kubernetes' memory accounting? I assume that'd do a better job if the node only had a static amount of memory.

Looks fine except for the return type which I believe is too narrow?

I'm not too fond of the List[str] -> List[Dict[str, Any]] signature, with the "not found" requested documents just disappearing from the result list.

FWIW, we didn't manage to replicate the timeout issue when manually killing and/or bringing down the network on the node currently responding to the MetalLB IP address... Every time, the failover happened within 10 seconds.

I think this makes sense, thanks!

In T4527#92580, @vlorentz wrote:

I think I'll implement it as a table in the scrubber's DB, this will make it easier to query the current status of scrubbing and add it to the Grafana dashboard

Ah, d'oh, it is (but your test fixture lies :-P)

Generating the issueTracker should probably be gated on the has_issues field (which is false, for this specific repository).

That seems sensible, thanks.

In D8582#223187, @vlorentz wrote:

In D8582#223184, @ardumont wrote:

It was initially to just compute that new hash at the same time as the standard ones we store but that's getting out of hand.

then use MultiHash directly in the Content loader, instead of Content.from_data

In D8582#223184, @ardumont wrote:

Why are the changes to the model object needed, instead of just hashing the file directly?

You mean using MultiHash directly. Yes, that feels more and more like the right way to do it.

It was initially to just compute that new hash at the same time as the standard ones we store but that's getting out of hand.

I don't agree with the idea of adding additional hashes, that will not be stored, to the Content model object. Model objects should map 1:1 with what is stored in the archive.

The logic seems sound, but I'm proposing an (untested) edit to make the code clearer, WDYT?

In D8539#222800, @ardumont wrote:

Fine, i've one comment i'd like others to have a look at though [1] regarding where
that new discovery (interface) code should go. It feels currently a bit off to me that this code
is in loader-core. Loaders are not the sole archive consumers (scanner, webapp, cli, indexer, cooker, ...).

[1] https://forge.softwareheritage.org/D8539?id=30916#inline-60774

@vlorentz @douardda @olasd @anlambert ^

The hang really is in the /java/ process, so that's not going to help me much, but it won't hurt either.

This seems reasonable in terms of ""ontology design"".

At a glance, this seems okay.

Rebase

fun with inconsistent side effects during imports

Thanks.

(indeed, collected resources)