Reading the code dealing with snapshot branches in several storage implementations, it really seems to me that storing them as a dict-like structure has no advantage.

Fri, Jul 10, 12:17 PM · Data Model

douardda added a comment to T2421: Make model objects immutable.

But I'd like to use the opportunity of this cleanup to go a bit further than "the minimal amount of work for pedantic correctness", and actually make changes that have a conceptual meaning.

Fri, Jul 10, 12:13 PM · Data Model

douardda added inline comments to D3456: Make metadata-related endpoints consistent with other endpoints by using Iterables of swh-model objects instead of a dict..

Fri, Jul 10, 11:17 AM

douardda requested changes to D3473: loader.core: check_snapshot: Allow to check using both Snapshot and dict objects.

Fri, Jul 10, 10:19 AM

douardda abandoned D762: tox: add a flake8 environment that generates Phabricator-friendly reports.

Fri, Jul 10, 10:07 AM

Thu, Jul 9

douardda committed rDSTOc3803ef8f797: Fix a typo I introduced in previous revision (authored by douardda).

Fix a typo I introduced in previous revision

Thu, Jul 9, 12:47 PM

douardda closed D3477: Fix a typo in previous revision.

Thu, Jul 9, 12:47 PM

douardda updated the diff for D3477: Fix a typo in previous revision.

make sure there is no doubt on who the dumb fellow is

Thu, Jul 9, 11:58 AM

douardda created D3477: Fix a typo in previous revision.

Thu, Jul 9, 11:56 AM

douardda committed rDSTO8bf3794874fa: Convert ImmutableDict to dict before passing it to json.dumps. (authored by vlorentz).

Convert ImmutableDict to dict before passing it to json.dumps.

Thu, Jul 9, 11:44 AM

douardda closed D3476: Convert ImmutableDict to dict before passing it to json.dumps..

Thu, Jul 9, 11:44 AM

douardda added inline comments to D3476: Convert ImmutableDict to dict before passing it to json.dumps..

Thu, Jul 9, 11:42 AM

douardda updated the diff for D3476: Convert ImmutableDict to dict before passing it to json.dumps..

simplify the diff by killing map_optional()

Thu, Jul 9, 11:33 AM

douardda commandeered D3476: Convert ImmutableDict to dict before passing it to json.dumps..

Thu, Jul 9, 11:09 AM

douardda added a comment to D3476: Convert ImmutableDict to dict before passing it to json.dumps..

Not sure I buy the map_optional here. When I read a line of code with [x if x else f(x)] I understand it right away. With map_optional, I need to go get the definition of that oneliner function.
Plus you have the type annotation headache...

Thu, Jul 9, 10:48 AM

douardda added a comment to D3475: setup: Do no expose the pytest-plugin any longer.

In D3475#85434, @ardumont wrote:

Adapt according to remarks:

Update docstring

Thu, Jul 9, 10:43 AM

Wed, Jul 8

douardda accepted D3469: Move all celery-related fixtures to the swh.scheduler pytest plugin.

looks fine, as far as I can tell

Wed, Jul 8, 6:15 PM

douardda committed rDLDSVNfe5e4d342f30: Update the svn loader to new Revision API (authored by douardda).

Update the svn loader to new Revision API

Wed, Jul 8, 5:11 PM

douardda closed D3464: Update the svn loader to new Revision API.

Wed, Jul 8, 5:11 PM

douardda updated the diff for D3464: Update the svn loader to new Revision API.

typo (thx ardumont)

Wed, Jul 8, 4:58 PM

douardda committed rDLDG8694ec70cda7: Small code simplification (authored by douardda).

Small code simplification

Wed, Jul 8, 4:13 PM

douardda created D3464: Update the svn loader to new Revision API.

Wed, Jul 8, 4:07 PM

douardda accepted D3463: loader: Adapt to swh-model >= 0.4.0.

lgtm

Wed, Jul 8, 3:43 PM

douardda committed rDLDG0394f0f85edf: Adapt the git loader to swh-model >0.4 (authored by douardda).

Adapt the git loader to swh-model >0.4

Wed, Jul 8, 3:13 PM

douardda closed D3454: Adapt the git loader to swh-model >0.4.

Wed, Jul 8, 3:13 PM

douardda updated the diff for D3454: Adapt the git loader to swh-model >0.4.

small simplification as suggested by anlambert

Wed, Jul 8, 3:11 PM

douardda updated the diff for D3454: Adapt the git loader to swh-model >0.4.

rebase

Wed, Jul 8, 2:12 PM

douardda updated the diff for D3454: Adapt the git loader to swh-model >0.4.

fix MANIFEST.in to include the git bundle file

Wed, Jul 8, 2:00 PM

douardda updated the diff for D3454: Adapt the git loader to swh-model >0.4.

bump dependencies

Wed, Jul 8, 12:25 PM

douardda committed rCDFJd502f6eb5a6d: Merge both "pip install" RUN commands in tox/Dockerfile (authored by douardda).

Merge both "pip install" RUN commands in tox/Dockerfile

Wed, Jul 8, 11:56 AM

douardda closed D3457: Merge both "pip install" RUN commands in tox/Dockerfile.

Wed, Jul 8, 11:56 AM

douardda created D3457: Merge both "pip install" RUN commands in tox/Dockerfile.

Wed, Jul 8, 10:18 AM

douardda accepted D3450: Use attr instead of NamedTuple to generate SWHID..

Thx, looks good. Just a small question about the modification of the swhid() API.

Wed, Jul 8, 9:32 AM

douardda added a comment to D3454: Adapt the git loader to swh-model >0.4.

In D3454#84921, @olasd wrote:

Oh, that needs a requirements-swh.txt bump as well, I guess.

Wed, Jul 8, 9:19 AM

Tue, Jul 7

douardda created D3454: Adapt the git loader to swh-model >0.4.

Tue, Jul 7, 7:03 PM

douardda added a comment to P718 (An Untitled Masterwork).

object 9768d0b576dbaaecd80abedad6dfd0d72f1476da
type commit
tag v0.0.1
tagger David Douard <david.douard@sdfa3.org> 1594138133 +0200

Tue, Jul 7, 6:42 PM

douardda added a comment to P718 (An Untitled Masterwork).

git cat-file -p 9768d0b576dbaaecd80abedad6dfd0d72f1476da
tree f0695c2e2fa7ce9d574023c3413761a473e500ca
parent c3c588713233609f5bbbb2d9e7f3fb4a660f3f72
author Stefano Zacchiroli <zack@upsilon.cc> 1443083765 +0200
committer Stefano Zacchiroli <zack@upsilon.cc> 1443083765 +0200

Tue, Jul 7, 6:41 PM

douardda created P718 (An Untitled Masterwork).

Tue, Jul 7, 6:39 PM

douardda committed rDSTO5ab70237d6c3: Extract revision's extra_header as a top level attribute (authored by douardda).

Extract revision's extra_header as a top level attribute

Tue, Jul 7, 5:18 PM

douardda closed D3426: Extract revision's extra_header as a top level attribute.

Tue, Jul 7, 5:18 PM

douardda updated the diff for D3426: Extract revision's extra_header as a top level attribute.

fix sql schema and simplify the migration script

Tue, Jul 7, 4:50 PM

douardda added inline comments to D3426: Extract revision's extra_header as a top level attribute.

Tue, Jul 7, 4:45 PM

douardda updated the diff for D3426: Extract revision's extra_header as a top level attribute.

typos/fixes in 158.sql

Tue, Jul 7, 3:35 PM

douardda updated the diff for D3426: Extract revision's extra_header as a top level attribute.

add a test for "true" bw compat, aka old data in pg still works

Tue, Jul 7, 3:20 PM

douardda created P717 (An Untitled Masterwork).

Tue, Jul 7, 2:21 PM

douardda accepted D3396: Add raw metadata to the model..

As said on IRC, I'd rather see this diff depends on D3417, but otherwise I'm fine with it.

Tue, Jul 7, 11:26 AM

douardda requested changes to D3435: WIP: Add mercurial.from_disk.HgLoaderFromDisk.

knowing this is still WIP, I have a few comments/questions:

Tue, Jul 7, 10:51 AM

Mon, Jul 6

douardda updated the diff for D3426: Extract revision's extra_header as a top level attribute.

use frozen<list<list<blob>>> in cassandra's schema for extra_headers

Mon, Jul 6, 6:55 PM

douardda added a comment to D3426: Extract revision's extra_header as a top level attribute.

In D3426#84268, @olasd wrote:

Looks nice.

Few comments:

I'm not convinced about the Cassandra storage of these. Have you tried a list <frozen list<blob>> or somesuch?

Mon, Jul 6, 4:33 PM

douardda updated the diff for D3426: Extract revision's extra_header as a top level attribute.

bump deps for swh-journal and swh-core

Mon, Jul 6, 2:35 PM

douardda updated the diff for D3426: Extract revision's extra_header as a top level attribute.

bump dep swh-model>= 0.4.0

Mon, Jul 6, 2:21 PM

douardda committed rDMODa7d9aca2b0df: Extract the extra_headers from metadata on the Revision model class (authored by douardda).

Extract the extra_headers from metadata on the Revision model class

Mon, Jul 6, 2:12 PM

douardda closed D3389: Extract the extra_headers from metadata on the Revision model class.

Mon, Jul 6, 2:12 PM

douardda committed rDJNLbf35ea79d3cc: Refactor pytest_plugins.assert_all_objects_consumed() to use model objects (authored by douardda).

Refactor pytest_plugins.assert_all_objects_consumed() to use model objects

Mon, Jul 6, 1:47 PM

douardda closed D3427: Refactor pytest_plugins.assert_all_objects_consumed() to use model objects.

Mon, Jul 6, 1:47 PM

douardda updated the diff for D3427: Refactor pytest_plugins.assert_all_objects_consumed() to use model objects.

typos

Mon, Jul 6, 1:42 PM

douardda added a comment to D3427: Refactor pytest_plugins.assert_all_objects_consumed() to use model objects.

In D3427#84191, @ardumont wrote:

ok

jsyk, noticed 2 typos in the commit message (and diff description):

"wee" -> "we"

"whic" -> "which"

Mon, Jul 6, 1:40 PM

douardda updated the task description for T2423: Extract the `extra_headers` away from `Revision.metadata` into a top-level immutable object.

Mon, Jul 6, 1:09 PM · Data Model

douardda created D3427: Refactor pytest_plugins.assert_all_objects_consumed() to use model objects.

Mon, Jul 6, 1:01 PM

douardda updated the diff for D3389: Extract the extra_headers from metadata on the Revision model class.

split the 2 test functions in parts, and fix the bwcompat hook

Mon, Jul 6, 11:59 AM

douardda added a comment to D3389: Extract the extra_headers from metadata on the Revision model class.

In D3389#84078, @douardda wrote:

In D3389#83915, @olasd wrote:

Accepted, with the following caveat: I'm still not sure what the plan is to deploy this now? In the current state of this diff, as soon as we deploy this on workers, data loss will occur (as the new extra_headers field isn't supported by swh.storage).

Now that I think of it, introducing a new field in swh.model needs a coordinated deployment anyway so maybe it doesn't matter (as long as the swh.storage update to support the new field occurs before the next deployment).

Yes this needs a counterpart in storage (wip), and needs a coordinated deployment.

Mon, Jul 6, 10:49 AM

douardda created D3426: Extract revision's extra_header as a top level attribute.

Mon, Jul 6, 10:44 AM

douardda added a revision to T2423: Extract the `extra_headers` away from `Revision.metadata` into a top-level immutable object: D3426: Extract revision's extra_header as a top level attribute.

Mon, Jul 6, 10:44 AM · Data Model

douardda added a comment to D3389: Extract the extra_headers from metadata on the Revision model class.

In D3389#83915, @olasd wrote:

Accepted, with the following caveat: I'm still not sure what the plan is to deploy this now? In the current state of this diff, as soon as we deploy this on workers, data loss will occur (as the new extra_headers field isn't supported by swh.storage).

Now that I think of it, introducing a new field in swh.model needs a coordinated deployment anyway so maybe it doesn't matter (as long as the swh.storage update to support the new field occurs before the next deployment).

Mon, Jul 6, 10:11 AM

douardda accepted D3421: storage: Send metrics from the origin_add endpoint.

Mon, Jul 6, 9:50 AM

Fri, Jul 3

douardda added a comment to T2474: drop blake2 hashes.

not sure about the db space as an argument, but the CPU is by itself worth the move IMHO.

Fri, Jul 3, 4:30 PM · Data Model, Storage manager

douardda updated the diff for D3389: Extract the extra_headers from metadata on the Revision model class.

rebase

Fri, Jul 3, 11:57 AM

douardda requested changes to D3396: Add raw metadata to the model..

Any reason not to use type_validator() in these new schema entities?

Fri, Jul 3, 11:33 AM

douardda accepted D3403: loader-svn: tests: Refactor svn-eol-style scenario.

LGTM but I'd prefer the commit message be self-contained, i.e. explicitly explain the unttest->pytest conversion rather then delegating this to a task in the forge.

Fri, Jul 3, 10:54 AM

douardda accepted D3394: Improve test coverage and type coverage for copy_to.

Thx a lot, LGTM (can't say I've reviewed the tests very carefully though.)

Fri, Jul 3, 10:41 AM