Page MenuHomeSoftware Heritage

douardda (David Douard)
User

Projects (7)

User Details

User Since
Jul 10 2018, 12:38 PM (89 w, 4 d)

Recent Activity

Fri, Mar 27

douardda created D2919: test: make test data properly typed.
Fri, Mar 27, 5:05 PM
douardda created D2918: GitLoader: instantiate SnapshotBranch object using the from_dict factory.
Fri, Mar 27, 4:36 PM
douardda created D2917: converters: fix types of TimestampWithTimezone arguments.
Fri, Mar 27, 4:36 PM
douardda created D2916: Adapt cassandra backend to validating model types.
Fri, Mar 27, 4:12 PM
douardda created D2915: test: get rid of normalized_xxx in tests.
Fri, Mar 27, 4:12 PM
douardda created D2914: test: ensure timestamp in test data are properly typed.
Fri, Mar 27, 4:11 PM
douardda created D2912: test: add a small test to check for type validation (using release_add).
Fri, Mar 27, 4:11 PM
douardda created D2911: validate: fix type annotation for origin_visit_add.
Fri, Mar 27, 4:10 PM
douardda created D2910: validate: ensure StorageArgumentException is always encodable.
Fri, Mar 27, 4:10 PM
douardda created D2909: writer: fix skipped_content_add type declaration to use SkippedContent.
Fri, Mar 27, 4:09 PM
douardda created D2908: tests: fix types of several test data sets.
Fri, Mar 27, 4:08 PM
douardda created D2906: model: add support for ctime in [Skipped]Content.from_[data,dict]().
Fri, Mar 27, 3:06 PM
douardda updated the diff for D2884: model: fix SkippedContent origin to be a str.

Rebase and add a test, as requested.

Fri, Mar 27, 3:05 PM
douardda created D2905: hypothesis: split hypothesis strategies as a dict + entity instance.
Fri, Mar 27, 3:04 PM
douardda updated the summary of D2819: model: use attrs_static to enforce type validation of model objects.
Fri, Mar 27, 11:07 AM

Thu, Mar 26

douardda created D2884: model: fix SkippedContent origin to be a str.
Thu, Mar 26, 11:32 AM
douardda updated the diff for D2824: model: improve a bit the TimestampWithTimezone model.

rebase

Thu, Mar 26, 11:30 AM
douardda updated the diff for D2823: tests: add low level tests for the Timestamp model entity.

rebase

Thu, Mar 26, 11:29 AM
douardda updated the diff for D2819: model: use attrs_static to enforce type validation of model objects.

get rid of the attrib_typecheck() decorator to prevent the need for a mypy plugin

Thu, Mar 26, 11:29 AM
douardda accepted D2769: Fix crash on None snapshot..
Thu, Mar 26, 10:49 AM

Wed, Mar 25

douardda added a comment to D2769: Fix crash on None snapshot..

The annotation part should be done on the whole module and, most importantly, in a dedicated revision.

The actual fix consist in a single 'if snapshot:' but this not what the diffstat "shows".

I only added annotations on code I changed.

Wed, Mar 25, 11:14 AM
douardda added a comment to D2769: Fix crash on None snapshot..

The annotation part should be done on the whole module and, most importantly, in a dedicated revision.

The actual fix consist in a single 'if snapshot:' but this not what the diffstat "shows".

I only added annotations on code I changed.

Wait, i found those annotation change on the code changed reasonable.
Against spend an insane amount of time typing all the module in one go (which must be a pita).
I prefer the incremental approach.

@douardda So what's wrong here?

The fact that val did not say he added types it in the description or that he did add the types?

I kinda do the same.
I increase the types of the code when i'm incrementally changing it.
I'd like to know if that's wrong.

Wed, Mar 25, 11:10 AM

Fri, Mar 20

douardda edited P621 stylish poll.
Fri, Mar 20, 12:26 PM
douardda created P621 stylish poll.
Fri, Mar 20, 12:18 PM

Tue, Mar 17

douardda added a comment to T2308: Better Validation in swh.model .

Also BaseModel.from_dict is currently pretty inconsistent: sometimes it will take care of instanciating model entities for attributes (e.g. for TimestampWithTimezone.timestamp), sometimes not (e.g. SkippedContent.origin)

Tue, Mar 17, 10:53 AM · Data Model

Fri, Mar 13

douardda updated the diff for D2823: tests: add low level tests for the Timestamp model entity.

rebased

Fri, Mar 13, 10:33 AM

Thu, Mar 12

douardda updated the diff for D2824: model: improve a bit the TimestampWithTimezone model.

apply vlorentz' comments

Thu, Mar 12, 5:04 PM
douardda updated the diff for D2819: model: use attrs_static to enforce type validation of model objects.

add missing mypy deps in requirements-test

Thu, Mar 12, 4:22 PM
douardda added inline comments to D2824: model: improve a bit the TimestampWithTimezone model.
Thu, Mar 12, 4:19 PM
douardda created D2824: model: improve a bit the TimestampWithTimezone model.
Thu, Mar 12, 4:09 PM
douardda created D2823: tests: add low level tests for the Timestamp model entity.
Thu, Mar 12, 4:09 PM
douardda updated the diff for D2819: model: use attrs_static to enforce type validation of model objects.

add support for the 'validator' argument in attrib_typecheck

Thu, Mar 12, 4:07 PM
douardda created P613 (An Untitled Masterwork).
Thu, Mar 12, 3:33 PM
douardda closed D2818: tests/identifiers: fix 'target', 'directory' and 'parents' object types.
Thu, Mar 12, 2:53 PM
douardda committed rDMOD56ae59c5ddbd: test/model: do not test direct instanciation of model objects (authored by douardda).
test/model: do not test direct instanciation of model objects
Thu, Mar 12, 2:53 PM
douardda committed rDMOD97af8866ebaf: tests/identifiers: fix 'target', 'directory' and 'parents' object types (authored by douardda).
tests/identifiers: fix 'target', 'directory' and 'parents' object types
Thu, Mar 12, 2:53 PM
douardda closed D2817: test/model: do not test direct instanciation of model objects.
Thu, Mar 12, 2:53 PM
douardda committed rDMODc74696036e97: tests/models: use d.copy() instead of dict(d) (authored by douardda).
tests/models: use d.copy() instead of dict(d)
Thu, Mar 12, 2:53 PM
douardda closed D2816: tests/models: use d.copy() instead of dict(d).
Thu, Mar 12, 2:53 PM
douardda closed D2815: model: kill Origin.type attribute.
Thu, Mar 12, 2:53 PM
douardda committed rDMODf533f62bbf11: model: kill Origin.type attribute (authored by douardda).
model: kill Origin.type attribute
Thu, Mar 12, 2:53 PM
douardda committed rDMOD0a6d7e050d2c: Extract the dictify() function from BaseModel.to_dict() (authored by douardda).
Extract the dictify() function from BaseModel.to_dict()
Thu, Mar 12, 2:53 PM
douardda closed D2814: Extract the dictify() function from BaseModel.to_dict().
Thu, Mar 12, 2:53 PM
douardda requested review of D2814: Extract the dictify() function from BaseModel.to_dict().
Thu, Mar 12, 2:31 PM
douardda updated the diff for D2819: model: use attrs_static to enforce type validation of model objects.

rebase + add missing plugin file

Thu, Mar 12, 2:30 PM
douardda updated the diff for D2818: tests/identifiers: fix 'target', 'directory' and 'parents' object types.

replace bhex() by _x() and other stuff reported by olasd

Thu, Mar 12, 2:29 PM
douardda added inline comments to D2811: scanner: added test for the model.
Thu, Mar 12, 2:15 PM
douardda requested changes to D2769: Fix crash on None snapshot..

The annotation part should be done on the whole module and, most importantly, in a dedicated revision.

Thu, Mar 12, 1:57 PM
douardda accepted D2771: Make release_add support adding the same object twice in the same call.
Thu, Mar 12, 1:53 PM
douardda added a comment to D2771: Make release_add support adding the same object twice in the same call.

I really think we should either have it for all object types or none at all.

Thu, Mar 12, 1:53 PM
douardda added a comment to D2814: Extract the dictify() function from BaseModel.to_dict().

What about adding tests on this or do you rely on BaseModel's?

Thu, Mar 12, 1:49 PM
douardda triaged T2309: Add support for other hash algo than sha1 in current objstorage implementation as Normal priority.
Thu, Mar 12, 1:43 PM · Object storage
douardda created P612 (An Untitled Masterwork).
Thu, Mar 12, 10:52 AM

Wed, Mar 11

douardda created D2819: model: use attrs_static to enforce type validation of model objects.
Wed, Mar 11, 5:56 PM
douardda created D2818: tests/identifiers: fix 'target', 'directory' and 'parents' object types.
Wed, Mar 11, 5:55 PM
douardda created D2817: test/model: do not test direct instanciation of model objects.
Wed, Mar 11, 5:55 PM
douardda created D2816: tests/models: use d.copy() instead of dict(d).
Wed, Mar 11, 5:54 PM
douardda created D2815: model: kill Origin.type attribute.
Wed, Mar 11, 5:54 PM
douardda created D2814: Extract the dictify() function from BaseModel.to_dict().
Wed, Mar 11, 5:53 PM
douardda updated the task description for T2308: Better Validation in swh.model .
Wed, Mar 11, 4:07 PM · Data Model
douardda triaged T2308: Better Validation in swh.model as Normal priority.
Wed, Mar 11, 4:06 PM · Data Model
douardda created T2308: Better Validation in swh.model .
Wed, Mar 11, 4:06 PM · Data Model
douardda created P611 (An Untitled Masterwork).
Wed, Mar 11, 2:11 PM
douardda committed rDSTOaa39be1b3b77: storage/writer: refactor JournalWriter.content_add to send model objects (authored by douardda).
storage/writer: refactor JournalWriter.content_add to send model objects
Wed, Mar 11, 10:40 AM
douardda closed D2803: storage/writer: refactor JournalWriter.content_add to send model objects.
Wed, Mar 11, 10:39 AM
douardda added a comment to D2803: storage/writer: refactor JournalWriter.content_add to send model objects.
In D2803#67209, @olasd wrote:
In D2803#67024, @olasd wrote:

My main doubt was whether we stopped explicitly converting model objects to dicts altogether (going through the swh.core model serializer instead). But even in that case contents will still be deserializable (as Content.from_dict(d) still works even when d['data'] is None).

What swh.core model serializer do you refer to? The ones in swh.core.api?

Yes. And now that you've pointed it out, I've remembered that it's the swh.storage RPC layer that adds a hook to support model objects.

Wed, Mar 11, 10:38 AM
douardda added a comment to D2803: storage/writer: refactor JournalWriter.content_add to send model objects.
In D2803#67024, @olasd wrote:

My main doubt was whether we stopped explicitly converting model objects to dicts altogether (going through the swh.core model serializer instead). But even in that case contents will still be deserializable (as Content.from_dict(d) still works even when d['data'] is None).

Wed, Mar 11, 10:35 AM

Tue, Mar 10

douardda closed D2801: kafka: normalize KafkaJournalWriter.write_addition[s] API.
Tue, Mar 10, 5:35 PM
douardda committed rDJNL82df6acedbb1: kafka: normalize KafkaJournalWriter.write_addition[s] API (authored by douardda).
kafka: normalize KafkaJournalWriter.write_addition[s] API
Tue, Mar 10, 5:35 PM
douardda updated the diff for D2801: kafka: normalize KafkaJournalWriter.write_addition[s] API.

remove extra parameter 'anon' mistakenly included in the diff

Tue, Mar 10, 5:29 PM
douardda created D2803: storage/writer: refactor JournalWriter.content_add to send model objects.
Tue, Mar 10, 4:46 PM
douardda committed rDSTOa97781d21131: storage/validate: small code formatting (authored by douardda).
storage/validate: small code formatting
Tue, Mar 10, 4:43 PM
douardda created D2801: kafka: normalize KafkaJournalWriter.write_addition[s] API.
Tue, Mar 10, 4:41 PM

Fri, Mar 6

douardda created P605 (An Untitled Masterwork).
Fri, Mar 6, 5:36 PM
douardda added inline comments to D2777: journal.replay: Batch insert contents/skipped_contents in storage backend.
Fri, Mar 6, 1:40 PM
douardda committed rDSTO3b8b718aa0c5: sql: do not attempt to create the plpgsql lang if already exists (authored by douardda).
sql: do not attempt to create the plpgsql lang if already exists
Fri, Mar 6, 1:39 PM
douardda closed D2776: sql: do not attempt to create the plpgsql lang if already exists.
Fri, Mar 6, 1:39 PM
douardda added a comment to D2776: sql: do not attempt to create the plpgsql lang if already exists.
In D2776#66377, @olasd wrote:

This looks sound but the tests are hanging on the initialization of the postgresql database now... (at least on jenkins)

Fri, Mar 6, 1:37 PM
douardda accepted D2778: Add install instructions for Cassandra..
Fri, Mar 6, 1:25 PM
douardda accepted D2777: journal.replay: Batch insert contents/skipped_contents in storage backend.

ok (besides my remark).

Fri, Mar 6, 11:54 AM
douardda added inline comments to D2777: journal.replay: Batch insert contents/skipped_contents in storage backend.
Fri, Mar 6, 11:54 AM
douardda created D2776: sql: do not attempt to create the plpgsql lang if already exists.
Fri, Mar 6, 9:31 AM

Wed, Mar 4

douardda accepted D2767: Add some tenacity to checking whether an object is in the destination.
Wed, Mar 4, 5:35 PM
douardda created P603 (An Untitled Masterwork).
Wed, Mar 4, 3:20 PM
douardda created P602 (An Untitled Masterwork).
Wed, Mar 4, 2:04 PM

Tue, Mar 3

douardda committed rCDFPcc2ae5af9877: images/base: add support for the LOG_LEVEL env var for replayer services (authored by douardda).
images/base: add support for the LOG_LEVEL env var for replayer services
Tue, Mar 3, 10:53 AM
douardda committed rCDFPb730b619299c: Update a bit the README file (authored by douardda).
Update a bit the README file
Tue, Mar 3, 10:53 AM
douardda committed rCDFP05dde3bf616d: grafana: fix the datasource config (authored by douardda).
grafana: fix the datasource config
Tue, Mar 3, 10:53 AM
douardda committed rCDFP6748a2080ca2: grafana: add a backend statistics dashboard, tune a bit the graph replayer one (authored by douardda).
grafana: add a backend statistics dashboard, tune a bit the graph replayer one
Tue, Mar 3, 10:53 AM
douardda committed rCDFPa7d896f05aa2: Move nginx listening port to 5081 (authored by douardda).
Move nginx listening port to 5081
Tue, Mar 3, 10:53 AM
douardda committed rCDFP9ecc8aa09974: update images entrypoint files (authored by douardda).
update images entrypoint files
Tue, Mar 3, 10:53 AM
douardda committed rCDFPefd4b4496e46: images/web: use a better 'shell' CMD support in web's entrypoint (authored by douardda).
images/web: use a better 'shell' CMD support in web's entrypoint
Tue, Mar 3, 10:53 AM
douardda committed rCDFP698e861a7056: example: fix the content-replayer.yml.example file (authored by douardda).
example: fix the content-replayer.yml.example file
Tue, Mar 3, 10:53 AM
douardda committed rCDFPd4c658bf1a6a: mirror: update the mirror deployment compose file (authored by douardda).
mirror: update the mirror deployment compose file
Tue, Mar 3, 10:53 AM
douardda committed rCDFP9fd8cdc38af7: images/web: reduce swh-web image size (authored by douardda).
images/web: reduce swh-web image size
Tue, Mar 3, 10:53 AM
douardda committed rCDFP59ae8d7374b0: postgres: improve a bit the Postgresql configuration (authored by douardda).
postgres: improve a bit the Postgresql configuration
Tue, Mar 3, 10:53 AM
douardda committed rCDFP390be1b78a7a: Dockerfile: update to buster and add the pgsql.sh utils file (authored by douardda).
Dockerfile: update to buster and add the pgsql.sh utils file
Tue, Mar 3, 10:53 AM
douardda committed rCDFPd9078f56c6ed: Add prometheus, statsd and grafana services (authored by douardda).
Add prometheus, statsd and grafana services
Tue, Mar 3, 10:53 AM
douardda committed rCDFPa0021f7e1bbb: web: add missing config entries (authored by douardda).
web: add missing config entries
Tue, Mar 3, 10:53 AM
douardda committed rCDFP524e7ee6d410: Add a pre-commit config file (authored by douardda).
Add a pre-commit config file
Tue, Mar 3, 10:53 AM