Page MenuHomeSoftware Heritage
Feed Advanced Search

Today

douardda closed D6727: Fix compatibility with tenacity 6.2.
Thu, Dec 2, 2:37 PM
douardda committed rDOBJSRPL86b8509fc2c7: Fix compatibility with tenacity 6.2 (authored by douardda).
Fix compatibility with tenacity 6.2
Thu, Dec 2, 2:37 PM
douardda committed rDOBJSRPL273675865266: Fix dependencies in d/control (authored by douardda).
Fix dependencies in d/control
Thu, Dec 2, 2:36 PM
douardda requested review of D6727: Fix compatibility with tenacity 6.2.
Thu, Dec 2, 2:27 PM
douardda abandoned D6492: Add support for pathslicing in seaweedfs backend.

probably won't be used

Thu, Dec 2, 1:16 PM
douardda closed D6693: Add support for a redis-based reporter for failed replayed objects.
Thu, Dec 2, 1:12 PM
douardda committed rDOBJSRPLa7bd6bc4427b: Add doctrings and comments in test_cli.py (authored by douardda).
Add doctrings and comments in test_cli.py
Thu, Dec 2, 1:12 PM
douardda committed rDOBJSRPLf5051ce1cbf4: Add support for a redis-based reporter for failed replayed objects (authored by douardda).
Add support for a redis-based reporter for failed replayed objects
Thu, Dec 2, 1:12 PM
douardda closed D6692: Rework the retry and reporting system in replay.py.
Thu, Dec 2, 1:11 PM
douardda committed rDOBJSRPL1d8ea80c7d01: Add tests for expected statsd reports during a content replay session (authored by douardda).
Add tests for expected statsd reports during a content replay session
Thu, Dec 2, 1:11 PM
douardda committed rDOBJSRPL8098798820bb: Rework the retry and reporting system in replay.py (authored by douardda).
Rework the retry and reporting system in replay.py
Thu, Dec 2, 1:11 PM
douardda closed D6724: Add tests for expected statsd reports during a content replay session.
Thu, Dec 2, 1:11 PM
douardda committed rDOBJSRPL002443567791: Small code refactoring in test_cli (authored by douardda).
Small code refactoring in test_cli
Thu, Dec 2, 1:11 PM
douardda updated the diff for D6693: Add support for a redis-based reporter for failed replayed objects.

rebase

Thu, Dec 2, 11:34 AM
douardda updated the diff for D6692: Rework the retry and reporting system in replay.py.

rebase

Thu, Dec 2, 11:34 AM
douardda updated the diff for D6724: Add tests for expected statsd reports during a content replay session.

typo and small improvement in the test

Thu, Dec 2, 11:32 AM
douardda added a comment to D6692: Rework the retry and reporting system in replay.py.
In D6692#174126, @olasd wrote:

You seem to have dropped the CONTENT_BYTES_METRIC ?

Thu, Dec 2, 11:10 AM
douardda updated the diff for D6692: Rework the retry and reporting system in replay.py.

Rebase on D6724 and adapt statsd tests to match new statsd probes

Thu, Dec 2, 11:09 AM
douardda requested review of D6724: Add tests for expected statsd reports during a content replay session.
Thu, Dec 2, 11:08 AM
douardda updated the diff for D6692: Rework the retry and reporting system in replay.py.

Add tests for expected statsd reports during a content replay session

Thu, Dec 2, 11:01 AM
douardda committed rDCORE862e1e51a67f: Port test_statsd.py to pytest and add a statsd test fixture (authored by douardda).
Port test_statsd.py to pytest and add a statsd test fixture
Thu, Dec 2, 10:48 AM
douardda closed D6709: Port test_statsd.py to pytest and add a statsd test fixture.
Thu, Dec 2, 10:48 AM
douardda committed rDCORE58ff77995512: Add magic to mypy ignored packages (authored by douardda).
Add magic to mypy ignored packages
Thu, Dec 2, 10:48 AM
douardda updated the diff for D6709: Port test_statsd.py to pytest and add a statsd test fixture.

Add a docstring in statsd fixture and update copyright dates in pytest_plugin.py

Thu, Dec 2, 10:45 AM

Yesterday

douardda updated the diff for D6709: Port test_statsd.py to pytest and add a statsd test fixture.

fix and swap args of 'assert' statements

Wed, Dec 1, 5:47 PM
douardda added inline comments to D6709: Port test_statsd.py to pytest and add a statsd test fixture.
Wed, Dec 1, 5:39 PM
douardda requested review of D6709: Port test_statsd.py to pytest and add a statsd test fixture.
Wed, Dec 1, 2:02 PM
douardda added a comment to D6692: Rework the retry and reporting system in replay.py.
In D6692#174129, @olasd wrote:

Could you make sure that this dashboard https://grafana.softwareheritage.org/d/d3l2oqXWz/s3-object-copy?orgId=1 is not affected (or that its functionality can be replaced easily?)

Wed, Dec 1, 11:57 AM
douardda added a comment to D6692: Rework the retry and reporting system in replay.py.
In D6692#174126, @olasd wrote:

You seem to have dropped the CONTENT_BYTES_METRIC ?

Wed, Dec 1, 11:47 AM
douardda accepted D6701: objectstorage: http backend: test cannonicalization of URL.

sorry I thought I had accepted it already :-)

Wed, Dec 1, 9:54 AM

Mon, Nov 29

douardda added a comment to D6697: Add tests for conflict resolution functions.

When I did so with other tests your claim was that they were too difficult to follow and that tests should be as explicit as possible. These ones have a declarative function name for each case. What's the criteria after all?

Mon, Nov 29, 12:08 PM
douardda accepted D6697: Add tests for conflict resolution functions.

Fine for me; these tests could be written as parametrized tests (https://docs.pytest.org/en/6.2.x/parametrize.html#parametrize-basics) but not a big deal.

Mon, Nov 29, 10:04 AM
douardda accepted D6699: Stop writing swhid2node.bin maps.

Looks overall ok to me, but I miss a few "why?" here:

  • the "why stop writing this file" should be in the commit message,
  • why the need for SortOutputHandler attributes to become final?
Mon, Nov 29, 10:00 AM

Fri, Nov 26

douardda requested review of D6693: Add support for a redis-based reporter for failed replayed objects.
Fri, Nov 26, 1:35 PM
douardda requested review of D6692: Rework the retry and reporting system in replay.py.
Fri, Nov 26, 1:34 PM
douardda added a revision to T3693: Provide a mecanism to report (with persistence) objects that fails to get replayed (mirror): D6693: Add support for a redis-based reporter for failed replayed objects.
Fri, Nov 26, 1:33 PM · Storage manager

Wed, Nov 24

douardda committed rMSLDab780bbec731: Try to improve a bit the swh-dataflow-merkle image (authored by douardda).
Try to improve a bit the swh-dataflow-merkle image
Wed, Nov 24, 12:08 PM

Wed, Nov 17

douardda committed rCDFP0526bf35c087: Use dnsrr as endpoint_mode for storage and objstorage and document this (authored by douardda).
Use dnsrr as endpoint_mode for storage and objstorage and document this
Wed, Nov 17, 11:02 AM
douardda committed rCDFP62767c6937ab: Improve entrypoint files (authored by douardda).
Improve entrypoint files
Wed, Nov 17, 11:02 AM
douardda committed rCDFPc4c400dd69f5: Make nginx always resolve upstream server names (authored by douardda).
Make nginx always resolve upstream server names
Wed, Nov 17, 11:02 AM
douardda committed rCDFPd131fe26e0e5: Fix the content-replayer grafana dashboard (authored by douardda).
Fix the content-replayer grafana dashboard
Wed, Nov 17, 11:02 AM
douardda committed rDSTOba105df3e5a0: d/changelog: version 0.40.0-2~swh1 (authored by douardda).
d/changelog: version 0.40.0-2~swh1
Wed, Nov 17, 9:47 AM
douardda committed rDSTO9af9b2641cfc: Update dependencies in d/control (authored by douardda).
Update dependencies in d/control
Wed, Nov 17, 9:47 AM

Tue, Nov 16

douardda closed D6645: Replace usage of 'stop_after_objects' by 'stop_on_eof' in tests.
Tue, Nov 16, 4:28 PM
douardda committed rDJNLa38c6e7e94a5: Replace usage of 'stop_after_objects' by 'stop_on_eof' in tests (authored by douardda).
Replace usage of 'stop_after_objects' by 'stop_on_eof' in tests
Tue, Nov 16, 4:28 PM
douardda closed D6644: Fix flakyness in test_client_with_deserializer.
Tue, Nov 16, 4:28 PM
douardda committed rDJNL808a4efcbd9a: Fix flakyness in test_client_with_deserializer (authored by douardda).
Fix flakyness in test_client_with_deserializer
Tue, Nov 16, 4:28 PM
douardda updated the diff for D6645: Replace usage of 'stop_after_objects' by 'stop_on_eof' in tests.

typo

Tue, Nov 16, 4:23 PM
douardda added a comment to D6644: Fix flakyness in test_client_with_deserializer.

since there is no guarantee the "expected" (considered invalid) revision will be consumed first

Really? Shouldn't the order be guaranteed by having only one partition per topic?

Tue, Nov 16, 4:22 PM
douardda requested review of D6645: Replace usage of 'stop_after_objects' by 'stop_on_eof' in tests.
Tue, Nov 16, 4:08 PM
douardda requested review of D6644: Fix flakyness in test_client_with_deserializer.
Tue, Nov 16, 2:06 PM

Mon, Nov 15

douardda renamed T3730: Add a "user support" mechanism in the archive from Add a to Add a "user support" mechanism in the archive.
Mon, Nov 15, 11:51 AM
douardda created T3730: Add a "user support" mechanism in the archive.
Mon, Nov 15, 11:40 AM

Tue, Nov 9

douardda closed D6571: Add support for a redis-based reporting for invalid mirrorred objects.
Tue, Nov 9, 5:28 PM
douardda committed rDSTO850a7553b6d5: Add support for a redis-based reporting for invalid mirrorred objects (authored by douardda).
Add support for a redis-based reporting for invalid mirrorred objects
Tue, Nov 9, 5:28 PM
douardda committed rDSTO04bd15a0bca8: Refactor fixer.fix_objects() to extract the inner object_fixers dict (authored by douardda).
Refactor fixer.fix_objects() to extract the inner object_fixers dict
Tue, Nov 9, 5:28 PM
douardda closed D6570: Remove now useless revision date checker in fixer.
Tue, Nov 9, 5:28 PM
douardda committed rDSTOd655c8581d3c: Remove now useless fixers (authored by douardda).
Remove now useless fixers
Tue, Nov 9, 5:28 PM
douardda closed D6569: Add a --type option to 'swh storage replay'.
Tue, Nov 9, 5:28 PM
douardda committed rDSTO55eed77b519c: Add a --type option to 'swh storage replay' (authored by douardda).
Add a --type option to 'swh storage replay'
Tue, Nov 9, 5:28 PM
douardda accepted D6625: Refactor package loaders to make the version part of BasePackageInfo.

makes sense

Tue, Nov 9, 4:44 PM
douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

rebase

Tue, Nov 9, 4:38 PM
douardda updated the diff for D6570: Remove now useless revision date checker in fixer.

rebase

Tue, Nov 9, 4:38 PM
douardda updated the diff for D6569: Add a --type option to 'swh storage replay'.

rebase

Tue, Nov 9, 4:37 PM

Mon, Nov 8

douardda added a watcher for Object storage: douardda.
Mon, Nov 8, 5:51 PM
douardda closed D6538: Remove the RADOS backend.
Mon, Nov 8, 4:07 PM
douardda committed rDOBJS646aa11297cc: Remove the RADOS backend (authored by douardda).
Remove the RADOS backend
Mon, Nov 8, 4:07 PM
douardda updated the diff for D6538: Remove the RADOS backend.

rebase

Mon, Nov 8, 4:04 PM
douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

fix docstring rst syntax

Mon, Nov 8, 3:44 PM
douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

apply vlorentz' comments

Mon, Nov 8, 3:29 PM
douardda added inline comments to D6571: Add support for a redis-based reporting for invalid mirrorred objects.
Mon, Nov 8, 3:29 PM
douardda updated the diff for D6570: Remove now useless revision date checker in fixer.

rebase

Mon, Nov 8, 3:27 PM
douardda updated the diff for D6569: Add a --type option to 'swh storage replay'.

Use a hardwritten list of accepted objects

Mon, Nov 8, 3:26 PM

Oct 29 2021

douardda abandoned D6554: [WIP] Add a (redis-based) validation error reporting facility.

used another approach, see D6571

Oct 29 2021, 2:14 PM
douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

apply vlorentz' comments/suggestions

Oct 29 2021, 1:45 PM
douardda added inline comments to D6571: Add support for a redis-based reporting for invalid mirrorred objects.
Oct 29 2021, 1:36 PM
douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

fix test_cli

Oct 29 2021, 1:28 PM
douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

rebase and a few fixes

Oct 29 2021, 12:30 PM
douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

"fix" mypy

Oct 29 2021, 10:26 AM
douardda added a comment to D6570: Remove now useless revision date checker in fixer.

You checked on the whole journal, right?

not yet. The shouldn't be any left, but... I'll check as soon as I have credentials for the production kafka cluster.

Oct 29 2021, 10:12 AM

Oct 28 2021

douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

Bump the dependency on swh-journal to 0.9

Oct 28 2021, 6:40 PM
douardda closed D6584: Make the test_client_with_deserializer reliable.
Oct 28 2021, 6:33 PM
douardda committed rDJNL79d28b51ae08: Make the test_client_with_deserializer reliable (authored by douardda).
Make the test_client_with_deserializer reliable
Oct 28 2021, 6:33 PM
douardda requested review of D6584: Make the test_client_with_deserializer reliable.
Oct 28 2021, 4:50 PM
douardda added a comment to D6570: Remove now useless revision date checker in fixer.
In D6570#170941, @olasd wrote:

While we're here, could you check whether the other fixers are really still needed too?

Oct 28 2021, 4:18 PM
douardda added a comment to D6570: Remove now useless revision date checker in fixer.

You checked on the whole journal, right?

Oct 28 2021, 4:18 PM
douardda closed D6565: Pass the object_type to JournalClient.value_serializer().
Oct 28 2021, 2:34 PM
douardda committed rDJNLf92d4acf30e6: Pass the object_type to JournalClient.value_serializer() (authored by douardda).
Pass the object_type to JournalClient.value_serializer()
Oct 28 2021, 2:34 PM
douardda closed D6564: Do call consumer.commit() even if not objects have been received.
Oct 28 2021, 2:34 PM
douardda committed rDJNL88054da89653: Do call consumer.commit() even if not objects have been received (authored by douardda).
Do call consumer.commit() even if not objects have been received
Oct 28 2021, 2:34 PM
douardda updated the diff for D6565: Pass the object_type to JournalClient.value_serializer().

^x^s...

Oct 28 2021, 2:08 PM
douardda updated the diff for D6565: Pass the object_type to JournalClient.value_serializer().

typo

Oct 28 2021, 2:03 PM
douardda accepted D6520: Add a new cli endpoint to schedule recurrent visits in Celery.

Overall it looks very good to me. There is room for improvement, for sure, but let's put this to work and see how it performs.

Oct 28 2021, 12:36 PM
douardda updated the diff for D6565: Pass the object_type to JournalClient.value_serializer().

Document a bit more the value_deserializer and add a test for it

Oct 28 2021, 12:11 PM
douardda added inline comments to D6565: Pass the object_type to JournalClient.value_serializer().
Oct 28 2021, 11:42 AM
douardda requested review of D6571: Add support for a redis-based reporting for invalid mirrorred objects.

asking for review even if tests are expected to fail because it depends on D6565 (in swh-journal)

Oct 28 2021, 11:26 AM
douardda added a comment to D6564: Do call consumer.commit() even if not objects have been received.

why if i may ask?

Oct 28 2021, 11:21 AM
douardda requested review of D6570: Remove now useless revision date checker in fixer.
Oct 28 2021, 10:23 AM
douardda requested review of D6569: Add a --type option to 'swh storage replay'.
Oct 28 2021, 10:22 AM

Oct 27 2021

douardda added a revision to T3693: Provide a mecanism to report (with persistence) objects that fails to get replayed (mirror): D6571: Add support for a redis-based reporting for invalid mirrorred objects.
Oct 27 2021, 6:24 PM · Storage manager