Page MenuHomeSoftware Heritage
Feed Advanced Search

May 31 2022

douardda added a revision to T4289: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function : D7929: use an in-memory storage for tests.
May 31 2022, 6:39 PM · Vault
douardda closed T4288: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function as Resolved.
May 31 2022, 4:52 PM · Datastore Scrubber
douardda closed T4288: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function , a subtask of T4285: Deprecate swh.core's postgresql_fact, as Resolved.
May 31 2022, 4:52 PM · Core & foundations
douardda closed D7926: tests: use stock pytest_postgresql factory function.
May 31 2022, 4:51 PM
douardda committed rDSCRUB3d999c512948: tests: use stock pytest_postgresql factory function (authored by douardda).
tests: use stock pytest_postgresql factory function
May 31 2022, 4:51 PM
douardda updated the diff for D7919: tests: use stock pytest_postgresql postgresql factory function.

update copyright

May 31 2022, 4:50 PM
douardda updated the diff for D7920: tests: use stock pytest_postgresql factory function.

update copyright

May 31 2022, 4:47 PM
douardda added a revision to T4286: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function : D7918: pytest_plugin: use the stock pytest_postgresql postgresql factory.
May 31 2022, 4:43 PM · Storage manager
douardda added a task to D7918: pytest_plugin: use the stock pytest_postgresql postgresql factory: T4286: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function .
May 31 2022, 4:43 PM
douardda added a revision to T4288: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function : D7926: tests: use stock pytest_postgresql factory function.
May 31 2022, 4:43 PM · Datastore Scrubber
douardda added a task to D7926: tests: use stock pytest_postgresql factory function: T4288: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function .
May 31 2022, 4:43 PM
douardda added a revision to T4287: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function : D7920: tests: use stock pytest_postgresql factory function.
May 31 2022, 4:42 PM · Scheduling utilities
douardda added a task to D7920: tests: use stock pytest_postgresql factory function: T4287: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function .
May 31 2022, 4:42 PM
douardda added a revision to T4290: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function : D7919: tests: use stock pytest_postgresql postgresql factory function.
May 31 2022, 4:42 PM · Indexer
douardda added a task to D7919: tests: use stock pytest_postgresql postgresql factory function: T4290: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function .
May 31 2022, 4:42 PM
douardda triaged T4290: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function as Normal priority.
May 31 2022, 4:41 PM · Indexer
douardda triaged T4289: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function as Normal priority.
May 31 2022, 4:41 PM · Vault
douardda triaged T4288: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function as Normal priority.
May 31 2022, 4:40 PM · Datastore Scrubber
douardda edited Description on Datastore Scrubber.
May 31 2022, 4:39 PM
douardda edited Description on Datastore Scrubber.
May 31 2022, 4:39 PM
douardda edited Description on Datastore Scrubber.
May 31 2022, 4:38 PM
douardda created Datastore Scrubber.
May 31 2022, 4:38 PM
douardda triaged T4287: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function as Normal priority.
May 31 2022, 4:24 PM · Scheduling utilities
douardda triaged T4286: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function as Normal priority.
May 31 2022, 4:23 PM · Storage manager
douardda triaged T4285: Deprecate swh.core's postgresql_fact as Normal priority.
May 31 2022, 4:22 PM · Core & foundations
douardda added a comment to D7920: tests: use stock pytest_postgresql factory function.

lgtm

one remark, I'd be neat you add the related task id on those diffs (if there is one).

May 31 2022, 4:12 PM
douardda requested review of D7926: tests: use stock pytest_postgresql factory function.
May 31 2022, 4:00 PM
douardda updated the diff for D7919: tests: use stock pytest_postgresql postgresql factory function.

rebase

May 31 2022, 3:58 PM
douardda accepted D7922: Workaround mypy false positive.
May 31 2022, 2:50 PM
douardda requested review of D7920: tests: use stock pytest_postgresql factory function.
May 31 2022, 2:29 PM
douardda requested review of D7919: tests: use stock pytest_postgresql postgresql factory function.
May 31 2022, 2:23 PM
douardda updated the diff for D7918: pytest_plugin: use the stock pytest_postgresql postgresql factory.

rebase

May 31 2022, 2:14 PM
douardda added a comment to D7918: pytest_plugin: use the stock pytest_postgresql postgresql factory.

instead of swh-core's postgresql_fact one, since we actually do not
use its custom features any more in swh-storage.

Well, initially the feature used was the reset between tables to make
it faster than it was (and it worked 'cause it was hellishly slow to drop
and create tables again).

So, now storage uses the template to make the tests go as fast, is that it?

May 31 2022, 2:11 PM
douardda closed D7917: Add missing __init__.py in proxies/.
May 31 2022, 2:03 PM
douardda committed rDSTOa936cfda1f8d: Add missing __init__.py in proxies/ (authored by douardda).
Add missing __init__.py in proxies/
May 31 2022, 2:03 PM
douardda requested review of D7918: pytest_plugin: use the stock pytest_postgresql postgresql factory.
May 31 2022, 1:49 PM
douardda requested review of D7917: Add missing __init__.py in proxies/.
May 31 2022, 1:47 PM
douardda accepted D7914: Wrap query in transaction.

makes sense to me

May 31 2022, 11:06 AM
douardda added a comment to D7913: db: Grant read access to guest user on all tables of the schema.

Tests for the cli are using again the pytest-plugin db tool (i think it got removed at some point for some reason).

May 31 2022, 11:03 AM

May 25 2022

douardda accepted D7888: Represent filtered_objects_percent as a histogram rather than average.
May 25 2022, 1:53 PM

May 24 2022

douardda accepted D7859: fix mailto encoding for inbound email.
May 24 2022, 11:51 AM

May 23 2022

douardda added a comment to T4265: Create test topics on staging's kafka for the mirror tests.

topics to be created under the `swh.test' prefix (or similar).

May 23 2022, 3:26 PM · System administration

May 19 2022

douardda created P1366 (An Untitled Masterwork).
May 19 2022, 11:18 AM

May 18 2022

douardda requested review of D7851: Include counters in the the main docker-compose file.
May 18 2022, 2:58 PM
douardda closed D7845: Fix yaml syntax in conf/prometheus-statsd-exporter.yml.
May 18 2022, 2:35 PM
douardda committed rDENV492e05ce6a58: Fix yaml syntax in conf/prometheus-statsd-exporter.yml (authored by douardda).
Fix yaml syntax in conf/prometheus-statsd-exporter.yml
May 18 2022, 2:35 PM
douardda requested review of D7845: Fix yaml syntax in conf/prometheus-statsd-exporter.yml.
May 18 2022, 11:25 AM

May 16 2022

douardda added a comment to T3957: Clean sentry reports.

seems we still have a lot of NotFound error from the swh-loader-git:

May 16 2022, 3:03 PM · meta-task, Roadmap 2022

May 12 2022

douardda triaged T4235: [As a temporary solution] deploy the statsd-exporter binary published by prometheus as High priority.
May 12 2022, 11:00 AM · System administration

May 11 2022

douardda closed D7813: Make 'python -m swh' work as cli entry point.
May 11 2022, 5:47 PM
douardda committed rDCOREdae963440be5: Make 'python -m swh' work as cli entry point (authored by douardda).
Make 'python -m swh' work as cli entry point
May 11 2022, 5:47 PM
douardda requested review of D7813: Make 'python -m swh' work as cli entry point.
May 11 2022, 4:01 PM

May 10 2022

douardda committed rCDFPeeb79bed22f8: Fix a typo and add missing topics in graph-replayer.yml example file (authored by douardda).
Fix a typo and add missing topics in graph-replayer.yml example file
May 10 2022, 2:08 PM

May 9 2022

douardda committed rCDFP79db3d6ad320: Replace tabs with 4-ws in entrypoint.sh (authored by douardda).
Replace tabs with 4-ws in entrypoint.sh
May 9 2022, 3:51 PM
douardda committed rCDFPd5d075f317a8: Update base image entrypoint (authored by douardda).
Update base image entrypoint
May 9 2022, 3:48 PM
douardda committed rCDFPacdf130e2b72: Reduce replicas of the objstorage and storage services to 1 (authored by douardda).
Reduce replicas of the objstorage and storage services to 1
May 9 2022, 3:48 PM
douardda committed rCDFP6f24ff481c96: Make pgsql.sh script a bit more consistent (authored by douardda).
Make pgsql.sh script a bit more consistent
May 9 2022, 3:48 PM
douardda committed rCDFPbb5a6602ea92: Remove deprecated 'args' in config files (authored by douardda).
Remove deprecated 'args' in config files
May 9 2022, 3:48 PM
douardda committed rCDFPcdf96cda1153: Fix the nginx config for grafana 8.3.5+ (authored by douardda).
Fix the nginx config for grafana 8.3.5+
May 9 2022, 3:48 PM

Apr 26 2022

douardda closed D7647: Update the the listing and loading scheduling architecture doc.
Apr 26 2022, 6:14 PM
douardda committed rDDOCb6c59c460142: Update the the listing and loading scheduling architecture doc (authored by douardda).
Update the the listing and loading scheduling architecture doc
Apr 26 2022, 6:13 PM
douardda closed D7692: Make swh.objstorage an optional dependency, as a 'with-content' extra.
Apr 26 2022, 6:09 PM
douardda committed rDDATASET157331b9ae66: Make swh.objstorage an optional dependency, as a 'with-content' extra (authored by douardda).
Make swh.objstorage an optional dependency, as a 'with-content' extra
Apr 26 2022, 6:08 PM
douardda added inline comments to D7692: Make swh.objstorage an optional dependency, as a 'with-content' extra.
Apr 26 2022, 6:01 PM
douardda updated the diff for D7692: Make swh.objstorage an optional dependency, as a 'with-content' extra.

typo + forgot the requirement file

Apr 26 2022, 6:01 PM
douardda requested review of D7692: Make swh.objstorage an optional dependency, as a 'with-content' extra.
Apr 26 2022, 5:55 PM
douardda updated the diff for D7647: Update the the listing and loading scheduling architecture doc.

rephrase (journal usage by the scheduler)

Apr 26 2022, 5:16 PM
douardda added a comment to D7647: Update the the listing and loading scheduling architecture doc.
In D7647#200113, @olasd wrote:

Thanks. On the diagram, it looks like 9' is causing 12 (via 10 and 11), which is not true (yet! D7332 isn't agreed upon) and could be very confusing.

This is because 9' writes to the origin_visit table, but 10/11/12 are based on the origin table/topic.

Could you split swh-storage in two, to avoid this misunderstanding?

To clear the misunderstanding maybe the label for 10 could be Consume final "origin visit status" entries?

Apr 26 2022, 5:10 PM
douardda updated the diff for D7647: Update the the listing and loading scheduling architecture doc.

typos and better rst syntax as suggested by vlorentz

Apr 26 2022, 2:05 PM
douardda added a comment to D7647: Update the the listing and loading scheduling architecture doc.

Fichier archi scheduler:

Apr 26 2022, 1:51 PM
douardda requested review of D7647: Update the the listing and loading scheduling architecture doc.
Apr 26 2022, 12:36 PM
douardda closed D7652: Update a bit the documentation for the new origin visit scheduler.
Apr 26 2022, 10:46 AM
douardda committed rDSCH3687931f78b5: Update a bit the documentation for the new origin visit scheduler (authored by douardda).
Update a bit the documentation for the new origin visit scheduler
Apr 26 2022, 10:46 AM
douardda updated the diff for D7652: Update a bit the documentation for the new origin visit scheduler.

typos

Apr 26 2022, 10:38 AM
douardda added a comment to D7652: Update a bit the documentation for the new origin visit scheduler.

lgtm

couple of typos to fix.

Apr 26 2022, 10:37 AM

Apr 25 2022

douardda requested review of D7652: Update a bit the documentation for the new origin visit scheduler.
Apr 25 2022, 6:20 PM

Apr 21 2022

douardda closed D7589: Add a --margin option to the `swh dataset graph export` command.
Apr 21 2022, 10:10 AM
douardda committed rDDATASET07bcf1674e0a: Add a --margin option to the `swh dataset graph export` command (authored by douardda).
Add a --margin option to the `swh dataset graph export` command
Apr 21 2022, 10:10 AM
douardda closed D7588: Add a --type option to the `swh dataset graph export` command.
Apr 21 2022, 10:10 AM
douardda committed rDDATASET4154d43a4c55: Add a --types option to the `swh dataset graph export` command (authored by douardda).
Add a --types option to the `swh dataset graph export` command
Apr 21 2022, 10:10 AM
douardda added inline comments to D7588: Add a --type option to the `swh dataset graph export` command.
Apr 21 2022, 10:07 AM
douardda closed D7531: Regularly check for EOF in Client.process() while waiting for messages.
Apr 21 2022, 10:03 AM
douardda committed rDJNLf98248dd6938: Regularly check for EOF in Client.process() while waiting for messages (authored by douardda).
Regularly check for EOF in Client.process() while waiting for messages
Apr 21 2022, 10:03 AM
douardda updated the diff for D7531: Regularly check for EOF in Client.process() while waiting for messages.

typo

Apr 21 2022, 9:56 AM
douardda updated the diff for D7589: Add a --margin option to the `swh dataset graph export` command.

rebase

Apr 21 2022, 9:52 AM
douardda updated the diff for D7588: Add a --type option to the `swh dataset graph export` command.

thx vlorentz

Apr 21 2022, 9:51 AM
douardda updated the diff for D7589: Add a --margin option to the `swh dataset graph export` command.

thx vlorentz

Apr 21 2022, 9:46 AM
douardda added inline comments to D7589: Add a --margin option to the `swh dataset graph export` command.
Apr 21 2022, 9:44 AM

Apr 20 2022

douardda closed D7591: Make scheduling policy used in schedule_recurrent configurable.
Apr 20 2022, 6:27 PM
douardda committed rDSCHa76bb02f0e94: Make scheduling policy used in schedule_recurrent configurable (authored by douardda).
Make scheduling policy used in schedule_recurrent configurable
Apr 20 2022, 6:27 PM
douardda added inline comments to D7591: Make scheduling policy used in schedule_recurrent configurable.
Apr 20 2022, 5:15 PM
douardda updated the diff for D7591: Make scheduling policy used in schedule_recurrent configurable.

Improve docstrings, better config validation, use lists in grab_next_visits_policy_weights()

Apr 20 2022, 4:36 PM
douardda updated the summary of D7591: Make scheduling policy used in schedule_recurrent configurable.
Apr 20 2022, 9:34 AM
douardda updated the summary of D7591: Make scheduling policy used in schedule_recurrent configurable.
Apr 20 2022, 9:33 AM
douardda retitled D7591: Make scheduling policy used in schedule_recurrent configurable from [wip] Make scheduling policy used in schedule_recurrent configurable to Make scheduling policy used in schedule_recurrent configurable.
Apr 20 2022, 9:33 AM
douardda updated the diff for D7591: Make scheduling policy used in schedule_recurrent configurable.

Use a flatter config structure

Apr 20 2022, 9:33 AM

Apr 15 2022

douardda requested review of D7591: Make scheduling policy used in schedule_recurrent configurable.
Apr 15 2022, 6:24 PM
douardda updated the diff for D7589: Add a --margin option to the `swh dataset graph export` command.

Improve and simplify a bit the code

Apr 15 2022, 1:27 PM
douardda updated the summary of D7589: Add a --margin option to the `swh dataset graph export` command.
Apr 15 2022, 12:55 PM
douardda accepted D7580: sentry: always override init settings with the environment variables.
Apr 15 2022, 12:42 PM