bump dep on swh.loader.core

Bump dep on swh.loader.core

bump dep on swh-loader-core

In D3501#86262, @ardumont wrote:

I'm very fine with landing a reasonably consistent and stable baseline any time you want.

In that case, i think a plan matching those properties could be:

• review/land first D3503 (so the check-snapshot implementation is complete, it checks up to the contents). It's not per say a blocker but that'd be more consistent.
• Then release new loader-core.
• Trigger back the builds of your mutiple diffs on the dvcs loaders (and land them when green, they should be ;)
• Then getting back here, rebase on latest loader-core and land this one.
• And then we (either you or me) should pick up those tests and replace the snapshots as dict with snapshot model objects (both core and again dvcs). <- that was my initial plan
• Optionally (or not), then we can come back here again and drop the from_dict conversion from the input... (only model object!!! at least in tests)

Does that sound reasonable?

looks ok, but I find it temptating to factorize(!) the missing objects looking pattern (not asked right now!).

rebase

no noqa

In D3501#86144, @ardumont wrote:

fwiw, i was heading towards making the transition you started *after* having deployed

the current implementation of "check-snapshot" (without that diff so with the internal
manipulation) And then commit the current diff ;)

Do you still want to land it first?

rebase

Forgot one occurrence

Reading the code dealing with snapshot branches in several storage implementations, it really seems to me that storing them as a dict-like structure has no advantage.

But I'd like to use the opportunity of this cleanup to go a bit further than "the minimal amount of work for pedantic correctness", and actually make changes that have a conceptual meaning.

make sure there is no doubt on who the dumb fellow is

simplify the diff by killing map_optional()

Not sure I buy the map_optional here. When I read a line of code with [x if x else f(x)] I understand it right away. With map_optional, I need to go get the definition of that oneliner function.
Plus you have the type annotation headache...

In D3475#85434, @ardumont wrote:

Adapt according to remarks:

• Update docstring

looks fine, as far as I can tell

typo (thx ardumont)

lgtm

small simplification as suggested by anlambert

rebase

fix MANIFEST.in to include the git bundle file

bump dependencies

Thx, looks good. Just a small question about the modification of the swhid() API.

In D3454#84921, @olasd wrote:

Oh, that needs a requirements-swh.txt bump as well, I guess.

fix sql schema and simplify the migration script

typos/fixes in 158.sql

add a test for "true" bw compat, aka old data in pg still works

As said on IRC, I'd rather see this diff depends on D3417, but otherwise I'm fine with it.

knowing this is still WIP, I have a few comments/questions:

use frozen<list<list<blob>>> in cassandra's schema for extra_headers

In D3426#84268, @olasd wrote:

Looks nice.

Few comments:

• I'm not convinced about the Cassandra storage of these. Have you tried a list <frozen list<blob>> or somesuch?

bump deps for swh-journal and swh-core

bump dep swh-model>= 0.4.0

typos

In D3427#84191, @ardumont wrote:

ok

jsyk, noticed 2 typos in the commit message (and diff description):

• "wee" -> "we"
• "whic" -> "which"

split the 2 test functions in parts, and fix the bwcompat hook

In D3389#84078, @douardda wrote:

In D3389#83915, @olasd wrote:

Accepted, with the following caveat: I'm still not sure what the plan is to deploy this now? In the current state of this diff, as soon as we deploy this on workers, data loss will occur (as the new extra_headers field isn't supported by swh.storage).

Now that I think of it, introducing a new field in swh.model needs a coordinated deployment anyway so maybe it doesn't matter (as long as the swh.storage update to support the new field occurs before the next deployment).

Yes this needs a counterpart in storage (wip), and needs a coordinated deployment.

In D3389#83915, @olasd wrote:

Accepted, with the following caveat: I'm still not sure what the plan is to deploy this now? In the current state of this diff, as soon as we deploy this on workers, data loss will occur (as the new extra_headers field isn't supported by swh.storage).

Now that I think of it, introducing a new field in swh.model needs a coordinated deployment anyway so maybe it doesn't matter (as long as the swh.storage update to support the new field occurs before the next deployment).

not sure about the db space as an argument, but the CPU is by itself worth the move IMHO.

rebase

Any reason not to use type_validator() in these new schema entities?

LGTM but I'd prefer the commit message be self-contained, i.e. explicitly explain the unttest->pytest conversion rather then delegating this to a task in the forge.

Thx a lot, LGTM (can't say I've reviewed the tests very carefully though.)

Looks globally fine to me, but I have a few comments/requests.

more mypy vs. attrs-strict fighting

make mypy happy (hopefully)