simplify the diff by killing map_optional()

Not sure I buy the map_optional here. When I read a line of code with [x if x else f(x)] I understand it right away. With map_optional, I need to go get the definition of that oneliner function.
Plus you have the type annotation headache...

In D3475#85434, @ardumont wrote:

Adapt according to remarks:

• Update docstring

looks fine, as far as I can tell

typo (thx ardumont)

lgtm

small simplification as suggested by anlambert

rebase

fix MANIFEST.in to include the git bundle file

bump dependencies

Thx, looks good. Just a small question about the modification of the swhid() API.

In D3454#84921, @olasd wrote:

Oh, that needs a requirements-swh.txt bump as well, I guess.

fix sql schema and simplify the migration script

typos/fixes in 158.sql

add a test for "true" bw compat, aka old data in pg still works

As said on IRC, I'd rather see this diff depends on D3417, but otherwise I'm fine with it.

knowing this is still WIP, I have a few comments/questions:

use frozen<list<list<blob>>> in cassandra's schema for extra_headers

In D3426#84268, @olasd wrote:

Looks nice.

Few comments:

• I'm not convinced about the Cassandra storage of these. Have you tried a list <frozen list<blob>> or somesuch?

bump deps for swh-journal and swh-core

bump dep swh-model>= 0.4.0

typos

In D3427#84191, @ardumont wrote:

ok

jsyk, noticed 2 typos in the commit message (and diff description):

• "wee" -> "we"
• "whic" -> "which"

split the 2 test functions in parts, and fix the bwcompat hook

In D3389#84078, @douardda wrote:

In D3389#83915, @olasd wrote:

Accepted, with the following caveat: I'm still not sure what the plan is to deploy this now? In the current state of this diff, as soon as we deploy this on workers, data loss will occur (as the new extra_headers field isn't supported by swh.storage).

Now that I think of it, introducing a new field in swh.model needs a coordinated deployment anyway so maybe it doesn't matter (as long as the swh.storage update to support the new field occurs before the next deployment).

Yes this needs a counterpart in storage (wip), and needs a coordinated deployment.

In D3389#83915, @olasd wrote:

Accepted, with the following caveat: I'm still not sure what the plan is to deploy this now? In the current state of this diff, as soon as we deploy this on workers, data loss will occur (as the new extra_headers field isn't supported by swh.storage).

Now that I think of it, introducing a new field in swh.model needs a coordinated deployment anyway so maybe it doesn't matter (as long as the swh.storage update to support the new field occurs before the next deployment).

not sure about the db space as an argument, but the CPU is by itself worth the move IMHO.

rebase

Any reason not to use type_validator() in these new schema entities?

LGTM but I'd prefer the commit message be self-contained, i.e. explicitly explain the unttest->pytest conversion rather then delegating this to a task in the forge.

Thx a lot, LGTM (can't say I've reviewed the tests very carefully though.)

Looks globally fine to me, but I have a few comments/requests.

more mypy vs. attrs-strict fighting

make mypy happy (hopefully)

restrict extra_headers to (bytes, bytes) only

improve bw-compat support, tests and hypothesis strategies

This origin (and others) also fails to be mirrored with errors like:

Rereading this task, I have a few comments/questions.

kill redundant test

rebase and improve a bit the test as well as the cassandra and in_memory backends

lgtm

In D3353#82224, @vlorentz wrote:

do you know where the old hashes came from?

closed by D3152

update copyright

It's a bit sad we cannot generically use sets for this (because some model objects are still unhashable), so fine for me.

due to changes in from_disk, I'd rather have a new review on this.

improve commit message + use '_' in DiskBakedContent.object_type

Update the diff to make it work (see T2422)

Previously proposed "short-term" solution does not work. So the only "short-term" solution is to make DiskBakedContent inherit from BaseModel (or BaseContent).

This task is currently blocked by an implementation "detail":