Page MenuHomeSoftware Heritage
Feed Advanced Search

Jul 6 2022

douardda committed rDSTOa00650ea0c34: Update pytest_plugin for swh.core 2.10 (authored by douardda).
Update pytest_plugin for swh.core 2.10
Jul 6 2022, 10:34 AM

Jul 5 2022

douardda requested review of D8067: Add a Storage.flavor property to the postgresql backend.
Jul 5 2022, 3:05 PM
douardda closed D8066: Add the 'flavor' argument to db_utils.initialize_database_for_module().
Jul 5 2022, 2:19 PM
douardda committed rDCOREb805b3106fb9: Add the 'flavor' argument to db_utils.initialize_database_for_module() (authored by douardda).
Add the 'flavor' argument to db_utils.initialize_database_for_module()
Jul 5 2022, 2:19 PM
douardda added inline comments to D8066: Add the 'flavor' argument to db_utils.initialize_database_for_module().
Jul 5 2022, 2:19 PM
douardda updated the diff for D8066: Add the 'flavor' argument to db_utils.initialize_database_for_module().

fix the test (op. priority...)

Jul 5 2022, 10:36 AM
douardda updated the diff for D8066: Add the 'flavor' argument to db_utils.initialize_database_for_module().

add tests + rebase

Jul 5 2022, 10:26 AM

Jul 4 2022

douardda added inline comments to D8053: Add minimal GitHub metadata mapping.
Jul 4 2022, 12:29 PM
douardda accepted D8060: Add extrinsic metadata indexer.
Jul 4 2022, 12:26 PM
douardda accepted D8058: Add support for origin_extrinsic_metadata to the storage.

Shouldn't there be a migration script as well?

Jul 4 2022, 12:21 PM
douardda accepted D8055: github mapping: Add support for more terms from the Codemeta crosswalk.
Jul 4 2022, 12:11 PM
douardda added inline comments to D8045: Add typing to detect_metadata() and related functions.
Jul 4 2022, 12:09 PM
douardda accepted D8054: github mapping: Add support for terms outside the codemeta context.

ok but the wording of the mark_production decorator

Jul 4 2022, 12:06 PM
douardda accepted D8053: Add minimal GitHub metadata mapping.

I'm not against this, but I have my doubts on the overall "architecture".

Jul 4 2022, 11:59 AM
douardda accepted D8048: Move mapping-specific tests to a new directory.

Don't like the zz-celery hack, but well, that's life. Did not check the new test files actually correspond to old test_metadata.py content. If this only test being moved or are there other (eg new tests) coming along? If so, please specify it in the commit message.

Jul 4 2022, 11:47 AM
douardda added inline comments to D8045: Add typing to detect_metadata() and related functions.
Jul 4 2022, 11:42 AM
douardda added a comment to D8066: Add the 'flavor' argument to db_utils.initialize_database_for_module().

lgtm but it's missing the test case around it. It should not be too hard to add a scenario which lifts the flavor parameter, shouldn't it?

Jul 4 2022, 11:00 AM

Jul 1 2022

douardda triaged T4370: Refactor the origin visit data model (aka get rid of the OriginVisit model object) as High priority.
Jul 1 2022, 4:35 PM · Storage manager, Data Model
douardda requested review of D8066: Add the 'flavor' argument to db_utils.initialize_database_for_module().
Jul 1 2022, 3:03 PM
douardda triaged T4368: Loosen "foreign key" validation in storages used as mirror ingestion endpoint as High priority.
Jul 1 2022, 11:13 AM · Storage manager
douardda created T4368: Loosen "foreign key" validation in storages used as mirror ingestion endpoint.
Jul 1 2022, 11:13 AM · Storage manager
douardda added a comment to T2309: Add support for other hash algo than sha1 in current objstorage implementation.

do you have in mind to make the actual hash used as primary key in an objstorage a configuration of said storage instance? e.g. create a pathslicer or s3 objstorage using sha256 is just a matter of configuration of the objstorage?

Jul 1 2022, 10:38 AM · Object storage
douardda added a comment to T2309: Add support for other hash algo than sha1 in current objstorage implementation.

do you have in mind to make the actual hash used as primary key in an objstorage a configuration of said storage instance? e.g. create a pathslicer or s3 objstorage using sha256 is just a matter of configuration of the objstorage?

Jul 1 2022, 10:34 AM · Object storage
douardda added inline comments to D8029: Start introducing composite ObjId in the interface.
Jul 1 2022, 10:22 AM

Jun 30 2022

douardda created P1395 (An Untitled Masterwork).
Jun 30 2022, 1:57 PM

Jun 29 2022

douardda created P1394 (An Untitled Masterwork).
Jun 29 2022, 5:32 PM

Jun 23 2022

douardda triaged T4349: deploy docker-compose-plugin package instead of docker-compose as Normal priority.
Jun 23 2022, 1:48 PM · System administration
douardda accepted D7987: Refactor snapshot branch functional tests.
Jun 23 2022, 1:30 PM

Jun 22 2022

douardda committed rDPROV346f3a575baa: wip (authored by douardda).
wip
Jun 22 2022, 12:41 PM

Jun 21 2022

douardda created P1389 (An Untitled Masterwork).
Jun 21 2022, 2:31 PM

Jun 20 2022

douardda added a subtask for T4338: mirror: k8s/terraform/hlem deployment test: T4342: Add a jenkins job for the AKS mirror deployment test.
Jun 20 2022, 10:17 AM · Mirror
douardda added a parent task for T4342: Add a jenkins job for the AKS mirror deployment test: T4338: mirror: k8s/terraform/hlem deployment test.
Jun 20 2022, 10:17 AM · Mirror
douardda added a parent task for T4341: Add a jenkins job for the docker E2E deployment mirror stack: T4337: mirror: docker swarm deployment test.
Jun 20 2022, 10:17 AM · Mirror
douardda added a subtask for T4337: mirror: docker swarm deployment test: T4341: Add a jenkins job for the docker E2E deployment mirror stack.
Jun 20 2022, 10:17 AM · Mirror
douardda renamed T4342: Add a jenkins job for the AKS mirror deployment test from Add a jankins job for the AKS mirror deployment test to Add a jenkins job for the AKS mirror deployment test.
Jun 20 2022, 10:16 AM · Mirror
douardda triaged T4342: Add a jenkins job for the AKS mirror deployment test as High priority.
Jun 20 2022, 10:16 AM · Mirror
douardda triaged T4341: Add a jenkins job for the docker E2E deployment mirror stack as High priority.
Jun 20 2022, 10:16 AM · Mirror
douardda added a subtask for T4339: Write a tool to handle the kafka dataset for the mirror E2E tests: T4337: mirror: docker swarm deployment test.
Jun 20 2022, 9:58 AM · Mirror
douardda added a parent task for T4337: mirror: docker swarm deployment test: T4339: Write a tool to handle the kafka dataset for the mirror E2E tests.
Jun 20 2022, 9:58 AM · Mirror
douardda added a subtask for T4339: Write a tool to handle the kafka dataset for the mirror E2E tests: T4338: mirror: k8s/terraform/hlem deployment test.
Jun 20 2022, 9:58 AM · Mirror
douardda added a parent task for T4338: mirror: k8s/terraform/hlem deployment test: T4339: Write a tool to handle the kafka dataset for the mirror E2E tests.
Jun 20 2022, 9:58 AM · Mirror
douardda triaged T4339: Write a tool to handle the kafka dataset for the mirror E2E tests as High priority.
Jun 20 2022, 9:57 AM · Mirror
douardda triaged T4338: mirror: k8s/terraform/hlem deployment test as High priority.
Jun 20 2022, 9:53 AM · Mirror
douardda renamed T4336: Add deployment tests for the mirror from Add a deployment test for the mirror to Add deployment tests for the mirror.
Jun 20 2022, 9:52 AM · Mirror
douardda triaged T4337: mirror: docker swarm deployment test as High priority.
Jun 20 2022, 9:51 AM · Mirror
douardda triaged T4336: Add deployment tests for the mirror as High priority.
Jun 20 2022, 9:50 AM · Mirror

Jun 17 2022

douardda added inline comments to D7987: Refactor snapshot branch functional tests.
Jun 17 2022, 2:29 PM
douardda added inline comments to D7987: Refactor snapshot branch functional tests.
Jun 17 2022, 2:27 PM

Jun 15 2022

douardda added inline comments to D7989: Refactor origin connection functional tests.
Jun 15 2022, 3:20 PM
douardda requested changes to D7987: Refactor snapshot branch functional tests.

please explain a bit what this refactor is about in the commit message.

Jun 15 2022, 9:59 AM
douardda added a comment to D7988: api/origin: Do not attempt to lookup similar origin URLs.

Also, this will "break the API". I am not opposed to this change, but some care should be taken when changing public's API behavior like this. (attempt to check if there are users of the behavior to be modified, document the new behavior, display/return a warning or something, etc)

Jun 15 2022, 9:50 AM
douardda added inline comments to D7988: api/origin: Do not attempt to lookup similar origin URLs.
Jun 15 2022, 9:46 AM
douardda closed D7982: docker: update the storage.yml config file a bit.
Jun 15 2022, 9:41 AM
douardda closed D7981: docker: Update and normalize conf/web.yaml and conf/web-search.yml.
Jun 15 2022, 9:41 AM
douardda committed rDENVa20102186be2: docker: update the storage.yml config file a bit (authored by douardda).
docker: update the storage.yml config file a bit
Jun 15 2022, 9:41 AM
douardda committed rDENVc00ee6f08890: docker: Update and normalize conf/web.yaml and conf/web-search.yml (authored by douardda).
docker: Update and normalize conf/web.yaml and conf/web-search.yml
Jun 15 2022, 9:41 AM

Jun 14 2022

douardda added inline comments to D7980: Return origin object for analogues URLs.
Jun 14 2022, 2:52 PM
douardda added a comment to D7980: Return origin object for analogues URLs.

Ack, I will make the change to the /api/1/origin/ endpoint to remove the URL processing then.

Jun 14 2022, 2:47 PM
douardda added a comment to D7980: Return origin object for analogues URLs.

Why should this be implemented by the API (as opposed to UIs)?

This is part of our current REST APIs.

Jun 14 2022, 12:39 PM

Jun 13 2022

douardda added inline comments to D7981: docker: Update and normalize conf/web.yaml and conf/web-search.yml.
Jun 13 2022, 3:26 PM
douardda updated the diff for D7982: docker: update the storage.yml config file a bit.

rebase

Jun 13 2022, 3:26 PM
douardda updated the diff for D7981: docker: Update and normalize conf/web.yaml and conf/web-search.yml.

set timeout to 5s on both configs, as requested by anlambert

Jun 13 2022, 3:25 PM
douardda requested review of D7982: docker: update the storage.yml config file a bit.
Jun 13 2022, 3:08 PM
douardda requested review of D7981: docker: Update and normalize conf/web.yaml and conf/web-search.yml.
Jun 13 2022, 3:07 PM

Jun 10 2022

douardda closed T4265: Create test topics on staging's kafka for the mirror tests as Resolved.

I've added the DescribeConfigs and AlterConfigs permissions for the mirror-test-rw principal, now I can do all I need.

Jun 10 2022, 3:41 PM · System administration
douardda triaged T4325: Remove (useless) metadata_authority and metadata_fetcher from the journal as Normal priority.
Jun 10 2022, 2:51 PM · Storage manager

Jun 8 2022

douardda closed D7970: Fix InMemoryJournalWriter: do not anonymize objects by default.
Jun 8 2022, 12:30 PM
douardda committed rDJNL6b05a6c949b4: Fix InMemoryJournalWriter: do not anonymize objects by default (authored by douardda).
Fix InMemoryJournalWriter: do not anonymize objects by default
Jun 8 2022, 12:30 PM
douardda updated the diff for D7970: Fix InMemoryJournalWriter: do not anonymize objects by default.

typos in the ci msg

Jun 8 2022, 12:29 PM
douardda requested review of D7970: Fix InMemoryJournalWriter: do not anonymize objects by default.
Jun 8 2022, 12:24 PM
douardda accepted D7913: db: Grant read access to guest user on all tables of the schema.

maybe only show a warning if the grant query fails (rather than crashing)?

Jun 8 2022, 10:02 AM

Jun 7 2022

douardda created P1379 (An Untitled Masterwork).
Jun 7 2022, 1:29 PM
douardda committed rDJNLa208f882a7e5: Make the stream journal client accept a string as output_stream config entry (authored by douardda).
Make the stream journal client accept a string as output_stream config entry
Jun 7 2022, 11:56 AM
douardda closed D7933: add a kafka_stream_to_value helper function in serializers.py.
Jun 7 2022, 11:55 AM
douardda committed rDJNL903f8e601aac: Add type annotation for JournalWriter related code (authored by douardda).
Add type annotation for JournalWriter related code
Jun 7 2022, 11:55 AM
douardda committed rDJNL0e90328a2260: Add a kafka_stream_to_value helper function in serializers.py (authored by douardda).
Add a kafka_stream_to_value helper function in serializers.py
Jun 7 2022, 11:55 AM
douardda closed D7952: Add type annotation for JournalWriter related code.
Jun 7 2022, 11:55 AM
douardda committed rDJNL908f015465a3: Remove unused 'privileged' argument from journal writer's write_addition(s) (authored by douardda).
Remove unused 'privileged' argument from journal writer's write_addition(s)
Jun 7 2022, 11:55 AM

Jun 3 2022

douardda accepted D7954: Set current_version attribute to postgresql datastore.
Jun 3 2022, 3:30 PM
douardda accepted D7958: Remove unused get_current_version method.
Jun 3 2022, 3:29 PM
douardda accepted D7953: Set current_version attribute to postgresql datastore.

lgtm (probably needs tedious hand-managed-db-migration-in-docker test)

Jun 3 2022, 3:29 PM
douardda updated the diff for D7933: add a kafka_stream_to_value helper function in serializers.py.

rebase

Jun 3 2022, 12:23 PM
douardda updated the diff for D7952: Add type annotation for JournalWriter related code.

actual fix

Jun 3 2022, 12:23 PM
douardda updated the diff for D7933: add a kafka_stream_to_value helper function in serializers.py.

rebase

Jun 3 2022, 12:20 PM
douardda updated the diff for D7952: Add type annotation for JournalWriter related code.

fix

Jun 3 2022, 12:19 PM
douardda updated the diff for D7933: add a kafka_stream_to_value helper function in serializers.py.

rebase

Jun 3 2022, 12:04 PM
douardda updated the diff for D7952: Add type annotation for JournalWriter related code.

remove TValue typevar as suggested by vlorentz

Jun 3 2022, 12:03 PM
douardda added inline comments to D7933: add a kafka_stream_to_value helper function in serializers.py.
Jun 3 2022, 10:49 AM
douardda added inline comments to D7933: add a kafka_stream_to_value helper function in serializers.py.
Jun 3 2022, 10:48 AM
douardda requested review of D7952: Add type annotation for JournalWriter related code.
Jun 3 2022, 10:33 AM
douardda accepted D7943: Revert "cli.db: Use attribute current_version instead of undeclared getter".
Jun 3 2022, 10:31 AM
douardda updated the diff for D7933: add a kafka_stream_to_value helper function in serializers.py.

rebase and update copyrights

Jun 3 2022, 10:28 AM
douardda updated the diff for D7933: add a kafka_stream_to_value helper function in serializers.py.

Extract the type annotation part of this diff in a dedicated one

Jun 3 2022, 10:15 AM

Jun 2 2022

douardda updated the diff for D7933: add a kafka_stream_to_value helper function in serializers.py.

Add type annotation in swh.journal.writer and rename the 'stream' arg as 'file_like'

Jun 2 2022, 5:42 PM
douardda committed rDENV510f41098258: Include counters in the the main docker-compose file (authored by douardda).
Include counters in the the main docker-compose file
Jun 2 2022, 3:54 PM
douardda closed D7851: Include counters in the the main docker-compose file.
Jun 2 2022, 3:54 PM
douardda committed rDENVa87ff2e88774: Add missing object types in conf/counters_journal_client.yml (authored by douardda).
Add missing object types in conf/counters_journal_client.yml
Jun 2 2022, 3:54 PM
douardda updated the diff for D7851: Include counters in the the main docker-compose file.

rebase

Jun 2 2022, 3:53 PM
douardda added inline comments to D7933: add a kafka_stream_to_value helper function in serializers.py.
Jun 2 2022, 3:48 PM
douardda closed T4285: Deprecate swh.core's postgresql_fact as Resolved.
Jun 2 2022, 9:53 AM · Core & foundations