rebase

Forgot one occurrence

Reading the code dealing with snapshot branches in several storage implementations, it really seems to me that storing them as a dict-like structure has no advantage.

But I'd like to use the opportunity of this cleanup to go a bit further than "the minimal amount of work for pedantic correctness", and actually make changes that have a conceptual meaning.

make sure there is no doubt on who the dumb fellow is

simplify the diff by killing map_optional()

Not sure I buy the map_optional here. When I read a line of code with [x if x else f(x)] I understand it right away. With map_optional, I need to go get the definition of that oneliner function.
Plus you have the type annotation headache...

In D3475#85434, @ardumont wrote:

Adapt according to remarks:

• Update docstring

looks fine, as far as I can tell

typo (thx ardumont)

lgtm

small simplification as suggested by anlambert

rebase

fix MANIFEST.in to include the git bundle file

bump dependencies

Thx, looks good. Just a small question about the modification of the swhid() API.

In D3454#84921, @olasd wrote:

Oh, that needs a requirements-swh.txt bump as well, I guess.

fix sql schema and simplify the migration script

typos/fixes in 158.sql

add a test for "true" bw compat, aka old data in pg still works

As said on IRC, I'd rather see this diff depends on D3417, but otherwise I'm fine with it.

knowing this is still WIP, I have a few comments/questions:

use frozen<list<list<blob>>> in cassandra's schema for extra_headers

In D3426#84268, @olasd wrote:

Looks nice.

Few comments:

• I'm not convinced about the Cassandra storage of these. Have you tried a list <frozen list<blob>> or somesuch?

bump deps for swh-journal and swh-core

bump dep swh-model>= 0.4.0

typos

In D3427#84191, @ardumont wrote:

ok

jsyk, noticed 2 typos in the commit message (and diff description):

• "wee" -> "we"
• "whic" -> "which"

split the 2 test functions in parts, and fix the bwcompat hook

In D3389#84078, @douardda wrote:

In D3389#83915, @olasd wrote:

Accepted, with the following caveat: I'm still not sure what the plan is to deploy this now? In the current state of this diff, as soon as we deploy this on workers, data loss will occur (as the new extra_headers field isn't supported by swh.storage).

Now that I think of it, introducing a new field in swh.model needs a coordinated deployment anyway so maybe it doesn't matter (as long as the swh.storage update to support the new field occurs before the next deployment).

Yes this needs a counterpart in storage (wip), and needs a coordinated deployment.

In D3389#83915, @olasd wrote:

Accepted, with the following caveat: I'm still not sure what the plan is to deploy this now? In the current state of this diff, as soon as we deploy this on workers, data loss will occur (as the new extra_headers field isn't supported by swh.storage).

Now that I think of it, introducing a new field in swh.model needs a coordinated deployment anyway so maybe it doesn't matter (as long as the swh.storage update to support the new field occurs before the next deployment).

not sure about the db space as an argument, but the CPU is by itself worth the move IMHO.

rebase

Any reason not to use type_validator() in these new schema entities?

LGTM but I'd prefer the commit message be self-contained, i.e. explicitly explain the unttest->pytest conversion rather then delegating this to a task in the forge.

Thx a lot, LGTM (can't say I've reviewed the tests very carefully though.)

Looks globally fine to me, but I have a few comments/requests.

more mypy vs. attrs-strict fighting

make mypy happy (hopefully)

restrict extra_headers to (bytes, bytes) only

improve bw-compat support, tests and hypothesis strategies

This origin (and others) also fails to be mirrored with errors like:

Rereading this task, I have a few comments/questions.

kill redundant test

rebase and improve a bit the test as well as the cassandra and in_memory backends