@KShivendu it sounds like you haven't installed the various swh-* modules with pip -e. Did you follow the https://docs.softwareheritage.org/devel/developer-setup.html#developer-setup ? Steps in there should take care of that.

In D5107#128987, @vlorentz wrote:

@zack added to D5081: https://forge.softwareheritage.org/D5081?vs=18237&id=18239#toc

if that's fine with you, I'll do the same here

I've only added a nitpicky comment about docstrings.

Thanks for this summary/status, very useful. Regarding goals, I think we want to have a read goal also about time to first bite, which is a performance metric which is particularly bad in the current filesystem-based object storage. Not sure what would be a reasonable goal though. Poke @olasd: any idea about a good target for this?

I'm not approving (and only resigning as reviewer instead) just to allow others to have a look, given it's something quite foundational in the data model.

followed sequence of:

Size of SHA256, SWHID, Content
SHA256
SWHID
Content

The object storage is a collection of RBD images containing a sequence of objects (SHA256 + SWHID + content).

(I've finally caught up with the backlog in this task, sorry I'm late to the party.)

(thanks for this!)

I'm requesting changes as the idea of hexlifying URLs behind the "swh:1:ori:" prefix is still being discussed (on the swh-devel list) and, for now, I'm against that specific part of the proposal.

Awesome!, thanks.

In T2912#58062, @rdicosmo wrote:

Thanks @vsellier, that seems quite ok indeed. The only question left is to know if the estimator implemented is monotonic (i.e. we will never have negative bumps in the graph :-))

In T3008#58039, @olasd wrote:

I've written a basic documentation page : https://wiki.softwareheritage.org/wiki/HedgeDoc

@zack, do you see anything else you would like to have documented?

for context: this change is only going to be committed in a separate branch that we're using for experiments with swh-scanner, hence the testing requirements are different/lower than for master

ah, yes, of course, sorry I didn't notice this either in the previous diff :-)

Why do we still have tests that check error messages meant for human consumption, again? :-)
If the difference in how they fail is meaningful (which I don't know), they should raise different exceptions and we should just check their types.

looks great, thanks !

Thanks for the bug report. Do you have a log of what happens upon bin/update if pre-commit is missing? (it's sure is needed for committing, but i'm surprised it's needed for bin/update).
Also, is it needed for a raw "bin/update" or only for "bin/update -i" (that also install Python dependencies?).

oh, also, this diff should close T2647 in its commit message

note that this diff should close T2595, so the commit message should be updated

My answer is a very emphatic YES.

In T2912#55860, @rdicosmo wrote:

In T2912#55849, @olasd wrote:

I think we should be able to decouple these counters completely from the loaders, and have them directly updated/handled by a client of the swh-journal. This would be a "centralized" component, but which we can parallelize quite heavily thanks to basic kafka design. We can also leverage the way kafka clients do parallelism to sidestep the locking issues arising in a potentially distributed filter.

Maybe my writing was not all that clear: I also had in mind a single centralised component (the ArchiveCounter) per Bloom filter, receiving the lists newcontents of ids from the loaders.
Getting the feed of ids from swh-journal instead of from the loaders is really neat: we avoid touching the loader code, and we gain a better capability of monitoring the load on the ArchiveCounter, so I'm all for it :-)

I don't like this "asymmetric" approach much. I'd rather have both caches being equal and taking as class constructor input an optional sqlite connection. If it's None, they'll each create a connection (and not share it). If a connection is passed, they use it and do not close it on exit. Then, the init code in FuseCache takes care of initializing the shared sqlite connection, and passing it to both constructors. This will make the classes more reusable and the init code more clean.