Page MenuHomeSoftware Heritage
Feed Advanced Search

Jun 26 2020

douardda accepted D3361: loader*: Drop obsolete origin visit fields.

lgtm

Jun 26 2020, 10:15 AM
douardda added inline comments to D3359: storage*: Open order parameter to origin-visit-get endpoint.
Jun 26 2020, 10:11 AM
douardda committed rDJNLaff31e54e966: Use proper hash ids in tests' journal_data (authored by douardda).
Use proper hash ids in tests' journal_data
Jun 26 2020, 9:23 AM
douardda closed D3353: Use proper hash ids in tests' journal_data.
Jun 26 2020, 9:23 AM
douardda added a comment to D3353: Use proper hash ids in tests' journal_data.

do you know where the old hashes came from?

Jun 26 2020, 9:22 AM

Jun 25 2020

douardda created D3353: Use proper hash ids in tests' journal_data.
Jun 25 2020, 1:03 PM
douardda closed T2422: Add an `object_type` attribute to model classes as Resolved.

closed by D3152

Jun 25 2020, 12:57 PM · Data Model
douardda triaged T2466: Re design the lazy loading feature for Content objects as Normal priority.
Jun 25 2020, 12:57 PM · Core Loader
douardda committed rDMODe632abed41c5: Tag model entities with their "object_type" (authored by douardda).
Tag model entities with their "object_type"
Jun 25 2020, 10:29 AM
douardda closed D3152: Tag model entities with their "type".
Jun 25 2020, 10:29 AM

Jun 24 2020

douardda updated the diff for D3152: Tag model entities with their "type".

update copyright

Jun 24 2020, 5:40 PM
douardda added inline comments to D3152: Tag model entities with their "type".
Jun 24 2020, 5:38 PM
douardda added inline comments to D3152: Tag model entities with their "type".
Jun 24 2020, 5:38 PM
douardda accepted D3346: tests: Relax checks on journal writes regarding origin-visit*/snapshot.

It's a bit sad we cannot generically use sets for this (because some model objects are still unhashable), so fine for me.

Jun 24 2020, 5:07 PM
douardda requested review of D3152: Tag model entities with their "type".

due to changes in from_disk, I'd rather have a new review on this.

Jun 24 2020, 4:37 PM
douardda updated the diff for D3152: Tag model entities with their "type".

improve commit message + use '_' in DiskBakedContent.object_type

Jun 24 2020, 4:33 PM
douardda updated the diff for D3152: Tag model entities with their "type".

Update the diff to make it work (see T2422)

Jun 24 2020, 4:01 PM
douardda added a comment to T2422: Add an `object_type` attribute to model classes.

Previously proposed "short-term" solution does not work. So the only "short-term" solution is to make DiskBakedContent inherit from BaseModel (or BaseContent).

Jun 24 2020, 3:45 PM · Data Model
douardda added a comment to T2422: Add an `object_type` attribute to model classes.

This task is currently blocked by an implementation "detail":

Jun 24 2020, 3:28 PM · Data Model
douardda committed rDWAPPSe8f3588ede5f: Do not use the returned value of storage.origin_add in tests (authored by douardda).
Do not use the returned value of storage.origin_add in tests
Jun 24 2020, 11:51 AM
douardda closed D3326: Do not use the returned value of storage.origin_add in tests.
Jun 24 2020, 11:51 AM
douardda updated the diff for D3326: Do not use the returned value of storage.origin_add in tests.

rebase

Jun 24 2020, 11:33 AM

Jun 23 2020

douardda committed rDSTO621fc8d37795: Deprecate the origin_add_one() endpoint (authored by douardda).
Deprecate the origin_add_one() endpoint
Jun 23 2020, 4:49 PM
douardda closed D3328: Deprecate the origin_add_one() endpoint.
Jun 23 2020, 4:49 PM
douardda committed rDSTOfb603e1b9874: Make Storage.add_origin() return a sumary dict (authored by douardda).
Make Storage.add_origin() return a sumary dict
Jun 23 2020, 4:49 PM
douardda closed D3327: Refactor the Storage.add_origin API.
Jun 23 2020, 4:49 PM
douardda updated the diff for D3334: Add a new TenaciousProxyStorage.

rebas

Jun 23 2020, 4:09 PM
douardda updated the diff for D3328: Deprecate the origin_add_one() endpoint.

rebase

Jun 23 2020, 4:08 PM
douardda updated the diff for D3327: Refactor the Storage.add_origin API.

remove the previously added revision and fix the actual bug instead

Jun 23 2020, 4:08 PM
douardda updated the diff for D3328: Deprecate the origin_add_one() endpoint.

rebase

Jun 23 2020, 2:55 PM
douardda updated the diff for D3327: Refactor the Storage.add_origin API.

add a revision to fix test_origin_visit_status_add_twice

Jun 23 2020, 2:55 PM
douardda accepted D3308: loader: Read snapshot out of the last visit status.
Jun 23 2020, 2:30 PM
douardda accepted D3306: loader: Read snapshot out of the last origin visit status.

Not sure about implications of the branches_count=1 argument, but it may be related to my other comment. Otherwise, LGTM.

Jun 23 2020, 9:42 AM
douardda accepted D3305: package.loader: Retrieve latest snapshot out of the latest visit status.

Much easier to review! Thanks.

Jun 23 2020, 9:34 AM

Jun 22 2020

douardda created D3334: Add a new TenaciousProxyStorage.
Jun 22 2020, 5:10 PM
douardda updated the diff for D3328: Deprecate the origin_add_one() endpoint.

rebase

Jun 22 2020, 5:05 PM
douardda updated the diff for D3327: Refactor the Storage.add_origin API.

reword the commit message and fix cassandra impl

Jun 22 2020, 5:05 PM
douardda added inline comments to D3327: Refactor the Storage.add_origin API.
Jun 22 2020, 4:55 PM
douardda added inline comments to D3327: Refactor the Storage.add_origin API.
Jun 22 2020, 4:44 PM
douardda updated the diff for D3328: Deprecate the origin_add_one() endpoint.

rebase

Jun 22 2020, 2:39 PM
douardda updated the diff for D3327: Refactor the Storage.add_origin API.

Fix for cassandra

Jun 22 2020, 2:39 PM
douardda accepted D3305: package.loader: Retrieve latest snapshot out of the latest visit status.

Looks good to me but I would have preferred to see the "use assert_last_visit_matches" chuncks, which are a "pure" refactoring step, in a dedicated commit (so split this commit in 2 parts).

Jun 22 2020, 11:45 AM
douardda created D3328: Deprecate the origin_add_one() endpoint.
Jun 22 2020, 11:38 AM
douardda created D3327: Refactor the Storage.add_origin API.
Jun 22 2020, 11:37 AM
douardda committed rDSTO53c4392accec: Ensure ids are correct in tests' storage_data (authored by douardda).
Ensure ids are correct in tests' storage_data
Jun 22 2020, 11:02 AM
douardda committed rDSTO46ac997d0efe: Fix tests' storage_data revisions (authored by douardda).
Fix tests' storage_data revisions
Jun 22 2020, 11:02 AM
douardda committed rDSTO19354bcd7e44: SQL: replace the hash(url) index by a unique btree(url) on the origin table (authored by douardda).
SQL: replace the hash(url) index by a unique btree(url) on the origin table
Jun 22 2020, 11:02 AM
douardda closed D3325: Fix tests' storage_data.
Jun 22 2020, 11:02 AM
douardda closed D3324: SQL: replace the hash(url) index by a unique btree(url) on the origin table.
Jun 22 2020, 11:02 AM
douardda updated the diff for D3325: Fix tests' storage_data.

typo in commit msg

Jun 22 2020, 11:02 AM
douardda added inline comments to D3325: Fix tests' storage_data.
Jun 22 2020, 10:58 AM
douardda created D3326: Do not use the returned value of storage.origin_add in tests.
Jun 22 2020, 10:56 AM
douardda created D3325: Fix tests' storage_data.
Jun 22 2020, 10:14 AM
douardda created D3324: SQL: replace the hash(url) index by a unique btree(url) on the origin table.
Jun 22 2020, 10:13 AM

Jun 17 2020

douardda created P700 (An Untitled Masterwork).
Jun 17 2020, 2:16 PM
douardda committed rDSTO692bfa394482: Fix/update the backfiller (authored by douardda).
Fix/update the backfiller
Jun 17 2020, 12:16 PM
douardda closed D3299: Fix/update the backfiller.
Jun 17 2020, 12:16 PM
douardda updated the diff for D3299: Fix/update the backfiller.

rebase

Jun 17 2020, 12:07 PM
douardda updated the diff for D3299: Fix/update the backfiller.

rebase

Jun 17 2020, 11:29 AM
douardda updated the diff for D3299: Fix/update the backfiller.

prevent some timeout-kind-of stuff in test_backfiller

Jun 17 2020, 11:09 AM
douardda updated the diff for D3299: Fix/update the backfiller.

fix a couple of tests in test_backfill

Jun 17 2020, 10:19 AM
douardda added a comment to D3299: Fix/update the backfiller.

\m/

Thanks!

The description does not say it but you also dealt with origin-visit-status so another thanks!

Jun 17 2020, 9:48 AM
douardda created D3299: Fix/update the backfiller.
Jun 17 2020, 9:41 AM
douardda committed rDJNL6ea2d6e4c8ae: Set origin visits' metadata to None in journal_data (authored by douardda).
Set origin visits' metadata to None in journal_data
Jun 17 2020, 9:31 AM
douardda closed D3297: Set origin visits' metadata to None in journal_data.
Jun 17 2020, 9:31 AM
douardda accepted D3294: model.hypothesis_strategies: Make metadata always none on origin_visit.
Jun 17 2020, 9:30 AM
douardda created D3297: Set origin visits' metadata to None in journal_data.
Jun 17 2020, 9:23 AM

Jun 16 2020

douardda committed rDSTOc2b673b49b88: Fix db_to_author() converter to return None is all fields are None (authored by douardda).
Fix db_to_author() converter to return None is all fields are None
Jun 16 2020, 1:59 PM
douardda closed D3290: Fix db_to_author() converter to return None is all fields are None.
Jun 16 2020, 1:59 PM
douardda closed T2455: backfiller: properly support null authors as Resolved by committing rDSTOc2b673b49b88: Fix db_to_author() converter to return None is all fields are None.
Jun 16 2020, 1:59 PM · Storage manager
douardda updated the diff for D3290: Fix db_to_author() converter to return None is all fields are None.

update the docstring and add type annotations

Jun 16 2020, 12:35 PM
douardda added a revision to T2455: backfiller: properly support null authors: D3290: Fix db_to_author() converter to return None is all fields are None.
Jun 16 2020, 12:22 PM · Storage manager
douardda created D3290: Fix db_to_author() converter to return None is all fields are None.
Jun 16 2020, 12:22 PM
douardda committed rDMODb70b281404ff: Fix message generation in hypothesis strategy releases_d() (authored by douardda).
Fix message generation in hypothesis strategy releases_d()
Jun 16 2020, 10:38 AM
douardda committed rDMODf723eb12af67: Fix the model: Revision.message can be None (authored by douardda).
Fix the model: Revision.message can be None
Jun 16 2020, 10:38 AM
douardda committed rDMOD5c5f34fe7233: Use the optional() strategy instead of one_of(none(), ...) when possible (authored by douardda).
Use the optional() strategy instead of one_of(none(), ...) when possible
Jun 16 2020, 10:38 AM
douardda closed D3285: Fix the model: Revision.message can be None.
Jun 16 2020, 10:38 AM
douardda updated the diff for D3285: Fix the model: Revision.message can be None.

typo in a ci msg

Jun 16 2020, 10:37 AM
douardda created D3285: Fix the model: Revision.message can be None.
Jun 16 2020, 10:09 AM

Jun 15 2020

douardda created P698 (An Untitled Masterwork).
Jun 15 2020, 5:37 PM
douardda accepted D3283: Introduce a SchedulerException base class.

not completely sure why this is necessary, but meh

Jun 15 2020, 4:16 PM
douardda accepted D3271: Implement basic storage of lister information.
Jun 15 2020, 4:13 PM
douardda committed rDMODa427e184fb02: Allow negative_utc to be None in normalize_timestamp() (authored by douardda).
Allow negative_utc to be None in normalize_timestamp()
Jun 15 2020, 9:45 AM
douardda closed D3263: Allow negative_utc to be None in normalize_timestamp().
Jun 15 2020, 9:45 AM
douardda updated the diff for D3263: Allow negative_utc to be None in normalize_timestamp().

add a warning in the commit message

Jun 15 2020, 9:44 AM

Jun 12 2020

douardda added a comment to T2450: Fix pagination of the /revision/<rev>/log/ public API.

As discussed on IRC, a possible fix for is to clearly document the "limitations" of the current implementation.

Jun 12 2020, 5:30 PM · Web app
douardda triaged T2450: Fix pagination of the /revision/<rev>/log/ public API as High priority.
Jun 12 2020, 5:07 PM · Web app
douardda added inline comments to D3270: Introduce some scaffolding for an attrs-based BaseSchedulerModel.
Jun 12 2020, 2:56 PM
douardda added inline comments to D3270: Introduce some scaffolding for an attrs-based BaseSchedulerModel.
Jun 12 2020, 2:31 PM
douardda added inline comments to D3271: Implement basic storage of lister information.
Jun 12 2020, 2:24 PM
douardda added inline comments to D3270: Introduce some scaffolding for an attrs-based BaseSchedulerModel.
Jun 12 2020, 2:07 PM
douardda added a comment to D3263: Allow negative_utc to be None in normalize_timestamp().

This is needed to help consuming existing (invalid) messages from kafka.

If this is the only purpose, it should be in swh.storage.fixer

Jun 12 2020, 11:56 AM
douardda added a comment to D3263: Allow negative_utc to be None in normalize_timestamp().
In D3263#79805, @olasd wrote:

As I've pointed out on IRC, I think (offset, negative_utc) == (0, None) should raise an error, as this is undefined behavior.

Jun 12 2020, 11:49 AM

Jun 11 2020

douardda created D3263: Allow negative_utc to be None in normalize_timestamp().
Jun 11 2020, 5:35 PM
douardda committed rDJNL07c629d64b07: Add hidden Content in tests' journal_data (authored by douardda).
Add hidden Content in tests' journal_data
Jun 11 2020, 11:47 AM
douardda closed D3256: Add hidden Content in tests' journal_data.
Jun 11 2020, 11:47 AM
douardda committed rDJNLed1fd693e81b: Remove now unused swh/journal/tests/utils.py (authored by douardda).
Remove now unused swh/journal/tests/utils.py
Jun 11 2020, 11:47 AM
douardda closed D3255: Remove now unused swh/journal/tests/utils.py.
Jun 11 2020, 11:47 AM
douardda accepted D3257: Use the automatic RPC client/server generation.

besides I don't like the _scheduler global var name, LGTM

Jun 11 2020, 11:11 AM

Jun 10 2020

douardda added a comment to D3247: [WIP] Add content_metadata_{add,get}..

I'm a bit puzzled by this diff w.r.t this assertion in T2306 "this metadata store should be outside of the main graph storage" . T2306 is related with this diff, right?

Jun 10 2020, 2:57 PM