Page MenuHomeSoftware Heritage
Feed Advanced Search

Jun 24 2020

douardda updated the diff for D3326: Do not use the returned value of storage.origin_add in tests.

rebase

Jun 24 2020, 11:33 AM

Jun 23 2020

douardda committed rDSTO621fc8d37795: Deprecate the origin_add_one() endpoint (authored by douardda).
Deprecate the origin_add_one() endpoint
Jun 23 2020, 4:49 PM
douardda closed D3328: Deprecate the origin_add_one() endpoint.
Jun 23 2020, 4:49 PM
douardda committed rDSTOfb603e1b9874: Make Storage.add_origin() return a sumary dict (authored by douardda).
Make Storage.add_origin() return a sumary dict
Jun 23 2020, 4:49 PM
douardda closed D3327: Refactor the Storage.add_origin API.
Jun 23 2020, 4:49 PM
douardda updated the diff for D3334: Add a new TenaciousProxyStorage.

rebas

Jun 23 2020, 4:09 PM
douardda updated the diff for D3328: Deprecate the origin_add_one() endpoint.

rebase

Jun 23 2020, 4:08 PM
douardda updated the diff for D3327: Refactor the Storage.add_origin API.

remove the previously added revision and fix the actual bug instead

Jun 23 2020, 4:08 PM
douardda updated the diff for D3328: Deprecate the origin_add_one() endpoint.

rebase

Jun 23 2020, 2:55 PM
douardda updated the diff for D3327: Refactor the Storage.add_origin API.

add a revision to fix test_origin_visit_status_add_twice

Jun 23 2020, 2:55 PM
douardda accepted D3308: loader: Read snapshot out of the last visit status.
Jun 23 2020, 2:30 PM
douardda accepted D3306: loader: Read snapshot out of the last origin visit status.

Not sure about implications of the branches_count=1 argument, but it may be related to my other comment. Otherwise, LGTM.

Jun 23 2020, 9:42 AM
douardda accepted D3305: package.loader: Retrieve latest snapshot out of the latest visit status.

Much easier to review! Thanks.

Jun 23 2020, 9:34 AM

Jun 22 2020

douardda created D3334: Add a new TenaciousProxyStorage.
Jun 22 2020, 5:10 PM
douardda updated the diff for D3328: Deprecate the origin_add_one() endpoint.

rebase

Jun 22 2020, 5:05 PM
douardda updated the diff for D3327: Refactor the Storage.add_origin API.

reword the commit message and fix cassandra impl

Jun 22 2020, 5:05 PM
douardda added inline comments to D3327: Refactor the Storage.add_origin API.
Jun 22 2020, 4:55 PM
douardda added inline comments to D3327: Refactor the Storage.add_origin API.
Jun 22 2020, 4:44 PM
douardda updated the diff for D3328: Deprecate the origin_add_one() endpoint.

rebase

Jun 22 2020, 2:39 PM
douardda updated the diff for D3327: Refactor the Storage.add_origin API.

Fix for cassandra

Jun 22 2020, 2:39 PM
douardda accepted D3305: package.loader: Retrieve latest snapshot out of the latest visit status.

Looks good to me but I would have preferred to see the "use assert_last_visit_matches" chuncks, which are a "pure" refactoring step, in a dedicated commit (so split this commit in 2 parts).

Jun 22 2020, 11:45 AM
douardda created D3328: Deprecate the origin_add_one() endpoint.
Jun 22 2020, 11:38 AM
douardda created D3327: Refactor the Storage.add_origin API.
Jun 22 2020, 11:37 AM
douardda committed rDSTO53c4392accec: Ensure ids are correct in tests' storage_data (authored by douardda).
Ensure ids are correct in tests' storage_data
Jun 22 2020, 11:02 AM
douardda committed rDSTO46ac997d0efe: Fix tests' storage_data revisions (authored by douardda).
Fix tests' storage_data revisions
Jun 22 2020, 11:02 AM
douardda committed rDSTO19354bcd7e44: SQL: replace the hash(url) index by a unique btree(url) on the origin table (authored by douardda).
SQL: replace the hash(url) index by a unique btree(url) on the origin table
Jun 22 2020, 11:02 AM
douardda closed D3325: Fix tests' storage_data.
Jun 22 2020, 11:02 AM
douardda closed D3324: SQL: replace the hash(url) index by a unique btree(url) on the origin table.
Jun 22 2020, 11:02 AM
douardda updated the diff for D3325: Fix tests' storage_data.

typo in commit msg

Jun 22 2020, 11:02 AM
douardda added inline comments to D3325: Fix tests' storage_data.
Jun 22 2020, 10:58 AM
douardda created D3326: Do not use the returned value of storage.origin_add in tests.
Jun 22 2020, 10:56 AM
douardda created D3325: Fix tests' storage_data.
Jun 22 2020, 10:14 AM
douardda created D3324: SQL: replace the hash(url) index by a unique btree(url) on the origin table.
Jun 22 2020, 10:13 AM

Jun 17 2020

douardda created P700 (An Untitled Masterwork).
Jun 17 2020, 2:16 PM
douardda committed rDSTO692bfa394482: Fix/update the backfiller (authored by douardda).
Fix/update the backfiller
Jun 17 2020, 12:16 PM
douardda closed D3299: Fix/update the backfiller.
Jun 17 2020, 12:16 PM
douardda updated the diff for D3299: Fix/update the backfiller.

rebase

Jun 17 2020, 12:07 PM
douardda updated the diff for D3299: Fix/update the backfiller.

rebase

Jun 17 2020, 11:29 AM
douardda updated the diff for D3299: Fix/update the backfiller.

prevent some timeout-kind-of stuff in test_backfiller

Jun 17 2020, 11:09 AM
douardda updated the diff for D3299: Fix/update the backfiller.

fix a couple of tests in test_backfill

Jun 17 2020, 10:19 AM
douardda added a comment to D3299: Fix/update the backfiller.

\m/

Thanks!

The description does not say it but you also dealt with origin-visit-status so another thanks!

Jun 17 2020, 9:48 AM
douardda created D3299: Fix/update the backfiller.
Jun 17 2020, 9:41 AM
douardda committed rDJNL6ea2d6e4c8ae: Set origin visits' metadata to None in journal_data (authored by douardda).
Set origin visits' metadata to None in journal_data
Jun 17 2020, 9:31 AM
douardda closed D3297: Set origin visits' metadata to None in journal_data.
Jun 17 2020, 9:31 AM
douardda accepted D3294: model.hypothesis_strategies: Make metadata always none on origin_visit.
Jun 17 2020, 9:30 AM
douardda created D3297: Set origin visits' metadata to None in journal_data.
Jun 17 2020, 9:23 AM

Jun 16 2020

douardda committed rDSTOc2b673b49b88: Fix db_to_author() converter to return None is all fields are None (authored by douardda).
Fix db_to_author() converter to return None is all fields are None
Jun 16 2020, 1:59 PM
douardda closed D3290: Fix db_to_author() converter to return None is all fields are None.
Jun 16 2020, 1:59 PM
douardda closed T2455: backfiller: properly support null authors as Resolved by committing rDSTOc2b673b49b88: Fix db_to_author() converter to return None is all fields are None.
Jun 16 2020, 1:59 PM · Storage manager
douardda updated the diff for D3290: Fix db_to_author() converter to return None is all fields are None.

update the docstring and add type annotations

Jun 16 2020, 12:35 PM
douardda added a revision to T2455: backfiller: properly support null authors: D3290: Fix db_to_author() converter to return None is all fields are None.
Jun 16 2020, 12:22 PM · Storage manager
douardda created D3290: Fix db_to_author() converter to return None is all fields are None.
Jun 16 2020, 12:22 PM
douardda committed rDMODb70b281404ff: Fix message generation in hypothesis strategy releases_d() (authored by douardda).
Fix message generation in hypothesis strategy releases_d()
Jun 16 2020, 10:38 AM
douardda committed rDMODf723eb12af67: Fix the model: Revision.message can be None (authored by douardda).
Fix the model: Revision.message can be None
Jun 16 2020, 10:38 AM
douardda committed rDMOD5c5f34fe7233: Use the optional() strategy instead of one_of(none(), ...) when possible (authored by douardda).
Use the optional() strategy instead of one_of(none(), ...) when possible
Jun 16 2020, 10:38 AM
douardda closed D3285: Fix the model: Revision.message can be None.
Jun 16 2020, 10:38 AM
douardda updated the diff for D3285: Fix the model: Revision.message can be None.

typo in a ci msg

Jun 16 2020, 10:37 AM
douardda created D3285: Fix the model: Revision.message can be None.
Jun 16 2020, 10:09 AM

Jun 15 2020

douardda created P698 (An Untitled Masterwork).
Jun 15 2020, 5:37 PM
douardda accepted D3283: Introduce a SchedulerException base class.

not completely sure why this is necessary, but meh

Jun 15 2020, 4:16 PM
douardda accepted D3271: Implement basic storage of lister information.
Jun 15 2020, 4:13 PM
douardda committed rDMODa427e184fb02: Allow negative_utc to be None in normalize_timestamp() (authored by douardda).
Allow negative_utc to be None in normalize_timestamp()
Jun 15 2020, 9:45 AM
douardda closed D3263: Allow negative_utc to be None in normalize_timestamp().
Jun 15 2020, 9:45 AM
douardda updated the diff for D3263: Allow negative_utc to be None in normalize_timestamp().

add a warning in the commit message

Jun 15 2020, 9:44 AM

Jun 12 2020

douardda added a comment to T2450: Fix pagination of the /revision/<rev>/log/ public API.

As discussed on IRC, a possible fix for is to clearly document the "limitations" of the current implementation.

Jun 12 2020, 5:30 PM · Web app
douardda triaged T2450: Fix pagination of the /revision/<rev>/log/ public API as High priority.
Jun 12 2020, 5:07 PM · Web app
douardda added inline comments to D3270: Introduce some scaffolding for an attrs-based BaseSchedulerModel.
Jun 12 2020, 2:56 PM
douardda added inline comments to D3270: Introduce some scaffolding for an attrs-based BaseSchedulerModel.
Jun 12 2020, 2:31 PM
douardda added inline comments to D3271: Implement basic storage of lister information.
Jun 12 2020, 2:24 PM
douardda added inline comments to D3270: Introduce some scaffolding for an attrs-based BaseSchedulerModel.
Jun 12 2020, 2:07 PM
douardda added a comment to D3263: Allow negative_utc to be None in normalize_timestamp().

This is needed to help consuming existing (invalid) messages from kafka.

If this is the only purpose, it should be in swh.storage.fixer

Jun 12 2020, 11:56 AM
douardda added a comment to D3263: Allow negative_utc to be None in normalize_timestamp().
In D3263#79805, @olasd wrote:

As I've pointed out on IRC, I think (offset, negative_utc) == (0, None) should raise an error, as this is undefined behavior.

Jun 12 2020, 11:49 AM

Jun 11 2020

douardda created D3263: Allow negative_utc to be None in normalize_timestamp().
Jun 11 2020, 5:35 PM
douardda committed rDJNL07c629d64b07: Add hidden Content in tests' journal_data (authored by douardda).
Add hidden Content in tests' journal_data
Jun 11 2020, 11:47 AM
douardda closed D3256: Add hidden Content in tests' journal_data.
Jun 11 2020, 11:47 AM
douardda committed rDJNLed1fd693e81b: Remove now unused swh/journal/tests/utils.py (authored by douardda).
Remove now unused swh/journal/tests/utils.py
Jun 11 2020, 11:47 AM
douardda closed D3255: Remove now unused swh/journal/tests/utils.py.
Jun 11 2020, 11:47 AM
douardda accepted D3257: Use the automatic RPC client/server generation.

besides I don't like the _scheduler global var name, LGTM

Jun 11 2020, 11:11 AM

Jun 10 2020

douardda added a comment to D3247: [WIP] Add content_metadata_{add,get}..

I'm a bit puzzled by this diff w.r.t this assertion in T2306 "this metadata store should be outside of the main graph storage" . T2306 is related with this diff, right?

Jun 10 2020, 2:57 PM
douardda accepted D3253: loader: Migrate to use origin visit status add storage endpoint.

lgtm but see my comments, in case you agree with them.

Jun 10 2020, 2:25 PM
douardda added inline comments to D3253: loader: Migrate to use origin visit status add storage endpoint.
Jun 10 2020, 2:24 PM
douardda added inline comments to D3253: loader: Migrate to use origin visit status add storage endpoint.
Jun 10 2020, 2:22 PM
douardda created D3256: Add hidden Content in tests' journal_data.
Jun 10 2020, 1:38 PM
douardda created D3255: Remove now unused swh/journal/tests/utils.py.
Jun 10 2020, 1:38 PM
douardda committed rDOBJSRPL599d70a487b9: Rewrite test_replay using the kafka_server fixture (authored by douardda).
Rewrite test_replay using the kafka_server fixture
Jun 10 2020, 1:34 PM
douardda closed D3252: Rewrite test_replay using the kafka_server fixture.
Jun 10 2020, 1:34 PM
douardda accepted D3248: Replace swh-worker-control with a swh scheduler celery-monitor subcommand.
Jun 10 2020, 12:19 PM
douardda created D3252: Rewrite test_replay using the kafka_server fixture.
Jun 10 2020, 12:07 PM
douardda added inline comments to D3248: Replace swh-worker-control with a swh scheduler celery-monitor subcommand.
Jun 10 2020, 10:59 AM

Jun 8 2020

douardda committed rDLDSVN2f08b0dfb7d1: test_loader: assert returned loader status is what we expect (authored by douardda).
test_loader: assert returned loader status is what we expect
Jun 8 2020, 9:52 AM
douardda closed D3227: test_loader: assert returned loader status is what we expect.
Jun 8 2020, 9:52 AM

Jun 5 2020

douardda accepted D3237: Fix race condition in multiplexer_objstorage add() return value.
Jun 5 2020, 6:44 PM
douardda committed rDSTO88271f80dc68: Add a replayer test for anonymized journal topics (authored by douardda).
Add a replayer test for anonymized journal topics
Jun 5 2020, 6:27 PM
douardda closed D3232: Add a replayer test for anonymized journal topics.
Jun 5 2020, 6:27 PM
douardda updated the diff for D3227: test_loader: assert returned loader status is what we expect.

fix the failing test

Jun 5 2020, 6:18 PM
douardda updated the diff for D3232: Add a replayer test for anonymized journal topics.

typo

Jun 5 2020, 6:13 PM
douardda committed rDSTOc75da7a90e1d: Small refactoring of the InMemoryStorage to make it more consistent (authored by douardda).
Small refactoring of the InMemoryStorage to make it more consistent
Jun 5 2020, 5:45 PM
douardda closed D3231: Small refactoring of the InMemoryStorage to make it more consistent.
Jun 5 2020, 5:45 PM
douardda accepted D3212: storage*: Add origin_visit_status_add endpoint.
Jun 5 2020, 5:37 PM · Storage manager
douardda created D3232: Add a replayer test for anonymized journal topics.
Jun 5 2020, 12:33 PM