In T2332#42825, @ardumont wrote:

Finally, we should make sure that the storage implementations reject objects with hashes of the wrong length. I'm /almost/ sure that's the case, but we should be sure of it.

That's the case.

Finally, we should make sure that the storage implementations reject objects with hashes of the wrong length. I'm /almost/ sure that's the case, but we should be sure of it.

to be more sure of that, I think we should make sure that all hash data in all exception arguments is hex-encoded unicode strings, rather than bytes objects left for python to repr(); this would circumvent a lot of places where encoding or decoding the data in transfer can go wrong.

it looks like there's a few actual collisions; seems that they're the known-colliding Google PDFs

I'll write my remarks down here for tracking purposes

sampled collisions extracted from sentry and storage [1]

thanks!

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/1041/ for more details.

Rebase on latest master

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/1040/ for more details.

Build was aborted

Build was aborted

Improve collision scenario checks

Build was aborted

Could you add more assertions to test_content_add_collision and test_content_add_metadata_collision, to check for the new common behavior?

Could you add more assertions to test_content_add_collision and test_content_add_metadata_collision, to check for the new common behavior?

Add coverage on extra conversion step

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/1033/ for more details.

Build was aborted

Align storages to return the list of colliding hashes

• pgstorage: Return the list of colliding content hashes
• improve regexp extraction

As mentioned inline in the pg storage diff, in general we should return /all/ colliding contents that we can find, rather than a single one. So in the end, the exception argument should be a List[Dict[str, bytes]].

Build is green

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/1031/ for more details.

Build was aborted

Adapt according to review

lgtm, but I'd like someone else to review it as well

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/1029/ for more details.

caf51a044377cf62f73a02cd6c641d94b4e32c95

I think so (latest storage version deployed).

@vlorentz I guess this could be closed?

python-cassandra too has some un-unpicklable errors:

Relatedly, errors raised by tenacity cannot be pickled because they contain a Lock: https://github.com/jd/tenacity/issues/147

storage02.euwest.azure exposes a rpc server using cassandra as storage backend.
webapp0 has been updated to use it.

thx

and swh-storage debian package built [1] (passing the cassandra tests ;)

I'm fine with switching to IRIs in the doc, just please expand what it means on first use (with a mention like "they are like URIs but"), as I don't think the acronym is that well-known yet, especially in the US.

done indeed.

It's deployed btw.

Cool, looks like this is all ready within our code base:

Considering the age of the bug report and how many underlying libraries have been upgraded, we can reopen this when we notice it again.

Is this still "a thing"?