Page MenuHomeSoftware Heritage
Feed Advanced Search

Nov 8 2021

douardda added a watcher for Object storage: douardda.
Nov 8 2021, 5:51 PM
douardda closed D6538: Remove the RADOS backend.
Nov 8 2021, 4:07 PM
douardda committed rDOBJS646aa11297cc: Remove the RADOS backend (authored by douardda).
Remove the RADOS backend
Nov 8 2021, 4:07 PM
douardda updated the diff for D6538: Remove the RADOS backend.

rebase

Nov 8 2021, 4:04 PM
douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

fix docstring rst syntax

Nov 8 2021, 3:44 PM
douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

apply vlorentz' comments

Nov 8 2021, 3:29 PM
douardda added inline comments to D6571: Add support for a redis-based reporting for invalid mirrorred objects.
Nov 8 2021, 3:29 PM
douardda updated the diff for D6570: Remove now useless revision date checker in fixer.

rebase

Nov 8 2021, 3:27 PM
douardda updated the diff for D6569: Add a --type option to 'swh storage replay'.

Use a hardwritten list of accepted objects

Nov 8 2021, 3:26 PM

Oct 29 2021

douardda abandoned D6554: [WIP] Add a (redis-based) validation error reporting facility.

used another approach, see D6571

Oct 29 2021, 2:14 PM
douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

apply vlorentz' comments/suggestions

Oct 29 2021, 1:45 PM
douardda added inline comments to D6571: Add support for a redis-based reporting for invalid mirrorred objects.
Oct 29 2021, 1:36 PM
douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

fix test_cli

Oct 29 2021, 1:28 PM
douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

rebase and a few fixes

Oct 29 2021, 12:30 PM
douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

"fix" mypy

Oct 29 2021, 10:26 AM
douardda added a comment to D6570: Remove now useless revision date checker in fixer.

You checked on the whole journal, right?

not yet. The shouldn't be any left, but... I'll check as soon as I have credentials for the production kafka cluster.

Oct 29 2021, 10:12 AM

Oct 28 2021

douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

Bump the dependency on swh-journal to 0.9

Oct 28 2021, 6:40 PM
douardda closed D6584: Make the test_client_with_deserializer reliable.
Oct 28 2021, 6:33 PM
douardda committed rDJNL79d28b51ae08: Make the test_client_with_deserializer reliable (authored by douardda).
Make the test_client_with_deserializer reliable
Oct 28 2021, 6:33 PM
douardda requested review of D6584: Make the test_client_with_deserializer reliable.
Oct 28 2021, 4:50 PM
douardda added a comment to D6570: Remove now useless revision date checker in fixer.
In D6570#170941, @olasd wrote:

While we're here, could you check whether the other fixers are really still needed too?

Oct 28 2021, 4:18 PM
douardda added a comment to D6570: Remove now useless revision date checker in fixer.

You checked on the whole journal, right?

Oct 28 2021, 4:18 PM
douardda closed D6565: Pass the object_type to JournalClient.value_serializer().
Oct 28 2021, 2:34 PM
douardda committed rDJNLf92d4acf30e6: Pass the object_type to JournalClient.value_serializer() (authored by douardda).
Pass the object_type to JournalClient.value_serializer()
Oct 28 2021, 2:34 PM
douardda closed D6564: Do call consumer.commit() even if not objects have been received.
Oct 28 2021, 2:34 PM
douardda committed rDJNL88054da89653: Do call consumer.commit() even if not objects have been received (authored by douardda).
Do call consumer.commit() even if not objects have been received
Oct 28 2021, 2:34 PM
douardda updated the diff for D6565: Pass the object_type to JournalClient.value_serializer().

^x^s...

Oct 28 2021, 2:08 PM
douardda updated the diff for D6565: Pass the object_type to JournalClient.value_serializer().

typo

Oct 28 2021, 2:03 PM
douardda accepted D6520: Add a new cli endpoint to schedule recurrent visits in Celery.

Overall it looks very good to me. There is room for improvement, for sure, but let's put this to work and see how it performs.

Oct 28 2021, 12:36 PM
douardda updated the diff for D6565: Pass the object_type to JournalClient.value_serializer().

Document a bit more the value_deserializer and add a test for it

Oct 28 2021, 12:11 PM
douardda added inline comments to D6565: Pass the object_type to JournalClient.value_serializer().
Oct 28 2021, 11:42 AM
douardda requested review of D6571: Add support for a redis-based reporting for invalid mirrorred objects.

asking for review even if tests are expected to fail because it depends on D6565 (in swh-journal)

Oct 28 2021, 11:26 AM
douardda added a comment to D6564: Do call consumer.commit() even if not objects have been received.

why if i may ask?

Oct 28 2021, 11:21 AM
douardda requested review of D6570: Remove now useless revision date checker in fixer.
Oct 28 2021, 10:23 AM
douardda requested review of D6569: Add a --type option to 'swh storage replay'.
Oct 28 2021, 10:22 AM

Oct 27 2021

douardda added a revision to T3693: Provide a mecanism to report (with persistence) objects that fails to get replayed (mirror): D6571: Add support for a redis-based reporting for invalid mirrorred objects.
Oct 27 2021, 6:24 PM · Storage manager
douardda requested review of D6565: Pass the object_type to JournalClient.value_serializer().
Oct 27 2021, 4:23 PM
douardda requested review of D6564: Do call consumer.commit() even if not objects have been received.
Oct 27 2021, 4:22 PM
douardda added a revision to T3693: Provide a mecanism to report (with persistence) objects that fails to get replayed (mirror): D6565: Pass the object_type to JournalClient.value_serializer().
Oct 27 2021, 4:19 PM · Storage manager
douardda accepted D6563: grab_next_visits: avoid time interval calculations in PostgreSQL.

A bit sad, but hey, thanks!

Oct 27 2021, 4:03 PM
douardda closed T3668: Improve the seaweedfs backend as Resolved.

Let's consider this is done since 2 of 3 bullets have been canceled (for now)

Oct 27 2021, 10:20 AM · Object storage
douardda updated the task description for T3668: Improve the seaweedfs backend.
Oct 27 2021, 10:18 AM · Object storage

Oct 26 2021

douardda requested review of D6554: [WIP] Add a (redis-based) validation error reporting facility.
Oct 26 2021, 5:56 PM
douardda added a revision to T3693: Provide a mecanism to report (with persistence) objects that fails to get replayed (mirror): D6554: [WIP] Add a (redis-based) validation error reporting facility.
Oct 26 2021, 5:48 PM · Storage manager
douardda triaged T3693: Provide a mecanism to report (with persistence) objects that fails to get replayed (mirror) as High priority.
Oct 26 2021, 5:41 PM · Storage manager

Oct 22 2021

douardda requested review of D6538: Remove the RADOS backend.
Oct 22 2021, 12:26 PM

Oct 21 2021

douardda closed D6521: Add a simple read-only HTTP backend.
Oct 21 2021, 2:23 PM
douardda committed rDOBJS8ed5f4ebc915: Add a simple read-only HTTP backend (authored by douardda).
Add a simple read-only HTTP backend
Oct 21 2021, 2:23 PM
douardda updated the diff for D6521: Add a simple read-only HTTP backend.

rebase

Oct 21 2021, 2:09 PM
douardda closed D6526: Reorganise the seaweedfs backend in a subpackage.
Oct 21 2021, 2:06 PM
douardda committed rDOBJSbcbbfd466987: Reorganise the seaweedfs backend in a subpackage (authored by douardda).
Reorganise the seaweedfs backend in a subpackage
Oct 21 2021, 2:06 PM
douardda closed D6525: Use get_objstorage in seaweedfs tests instead of direct class instanciation.
Oct 21 2021, 2:06 PM
douardda closed D6524: Add support for deprecation of objstorage cls in factory.
Oct 21 2021, 2:06 PM
douardda committed rDOBJS38c02dcfae2b: Add support for deprecation of objstorage cls in factory (authored by douardda).
Add support for deprecation of objstorage cls in factory
Oct 21 2021, 2:06 PM
douardda committed rDOBJS82d9714b0ae5: Use get_objstorage in seaweedfs tests instead of direct class (authored by douardda).
Use get_objstorage in seaweedfs tests instead of direct class
Oct 21 2021, 2:06 PM
douardda updated the diff for D6526: Reorganise the seaweedfs backend in a subpackage.

and fix the LOGGER.error usage

Oct 21 2021, 2:03 PM
douardda updated the diff for D6526: Reorganise the seaweedfs backend in a subpackage.

remove hardcoded log levels

Oct 21 2021, 1:53 PM
douardda added inline comments to D6526: Reorganise the seaweedfs backend in a subpackage.
Oct 21 2021, 1:49 PM
douardda requested review of D6526: Reorganise the seaweedfs backend in a subpackage.
Oct 21 2021, 1:09 PM
douardda requested review of D6525: Use get_objstorage in seaweedfs tests instead of direct class instanciation.
Oct 21 2021, 1:09 PM
douardda requested review of D6524: Add support for deprecation of objstorage cls in factory.
Oct 21 2021, 1:08 PM
douardda updated the diff for D6521: Add a simple read-only HTTP backend.

Use ReadOnlyObjStorage and NonIterableObjStorage instead of NotImplementedError

Oct 21 2021, 12:50 PM
douardda added inline comments to D6521: Add a simple read-only HTTP backend.
Oct 21 2021, 12:24 PM

Oct 20 2021

douardda added inline comments to D6424: Perfect hashmap C implementation.
Oct 20 2021, 5:17 PM
douardda updated the diff for D6521: Add a simple read-only HTTP backend.

document the build_objstorage() test helper function

Oct 20 2021, 5:00 PM
douardda updated the diff for D6521: Add a simple read-only HTTP backend.

remove useless statement

Oct 20 2021, 4:55 PM
douardda updated the diff for D6521: Add a simple read-only HTTP backend.

remove mistakenly commited mypy.ini file

Oct 20 2021, 4:53 PM
douardda requested review of D6521: Add a simple read-only HTTP backend.
Oct 20 2021, 4:53 PM
douardda committed rDOBJS6269067ca7b3: Improve a bit the seaweedfs backend (authored by douardda).
Improve a bit the seaweedfs backend
Oct 20 2021, 2:40 PM
douardda closed D6517: Improve tests of the seaweedfs backend.
Oct 20 2021, 2:40 PM
douardda committed rDOBJS55ff4b95d306: Improve tests of the seaweedfs backend (authored by douardda).
Improve tests of the seaweedfs backend
Oct 20 2021, 2:40 PM
douardda updated the diff for D6492: Add support for pathslicing in seaweedfs backend.

rebase

Oct 20 2021, 2:36 PM
douardda updated the diff for D6517: Improve tests of the seaweedfs backend.

remove a (comment) garbage line

Oct 20 2021, 2:35 PM
douardda updated the diff for D6492: Add support for pathslicing in seaweedfs backend.

rebase

Oct 20 2021, 2:31 PM
douardda updated the diff for D6517: Improve tests of the seaweedfs backend.

slight simplification as suggested by vlorentz

Oct 20 2021, 2:30 PM
douardda added inline comments to D6517: Improve tests of the seaweedfs backend.
Oct 20 2021, 2:17 PM
douardda updated the diff for D6492: Add support for pathslicing in seaweedfs backend.

respawn jenkins

Oct 20 2021, 2:15 PM
douardda updated the diff for D6517: Improve tests of the seaweedfs backend.

respawn jenkins

Oct 20 2021, 2:13 PM
douardda requested review of D6517: Improve tests of the seaweedfs backend.
Oct 20 2021, 2:08 PM
douardda updated the diff for D6492: Add support for pathslicing in seaweedfs backend.

rebase

Oct 20 2021, 12:34 PM
douardda added inline comments to D6492: Add support for pathslicing in seaweedfs backend.
Oct 20 2021, 12:23 PM
douardda updated the diff for D6492: Add support for pathslicing in seaweedfs backend.

split the diff in 2

Oct 20 2021, 12:14 PM

Oct 19 2021

douardda created P1203 (An Untitled Masterwork).
Oct 19 2021, 11:43 AM

Oct 18 2021

douardda triaged T3668: Improve the seaweedfs backend as Normal priority.
Oct 18 2021, 3:34 PM · Object storage
douardda created T3668: Improve the seaweedfs backend.
Oct 18 2021, 3:33 PM · Object storage
douardda created P1202 (An Untitled Masterwork).
Oct 18 2021, 2:47 PM
douardda added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

B3 I am not convinced a "synthetic" flag on the Snapshot branch makes sense, or at least I find this name confusing, especially considering we already have a synthetic flag on Revision: it's not synthetic in the sense of it's not object crafted by SWH, it comes from the origin.

Oct 18 2021, 11:59 AM · Git loader

Oct 15 2021

douardda requested review of D6492: Add support for pathslicing in seaweedfs backend.
Oct 15 2021, 6:35 PM
douardda triaged T3663: Make the swh-environment jenkins job green and activate notifications as High priority.
Oct 15 2021, 10:45 AM · System administration

Oct 14 2021

douardda added a comment to T3635: git loader: enable "partial" global deduplication of revisions via the extid mapping table.

Ok I think what puzzle me in this description is the fact the 2 first bullets of the "git loader adaptations" are actually only one point: at the end of a successful loading, store a mapping in the extid table.

Oct 14 2021, 11:23 AM · Git loader

Oct 13 2021

douardda closed D6442: Extract the path slicing logic in a dedicated PathSlicer class.
Oct 13 2021, 3:19 PM
douardda committed rDOBJS23b7f81c1483: Extract the path slicing logic in a dedicated PathSlicer class (authored by douardda).
Extract the path slicing logic in a dedicated PathSlicer class
Oct 13 2021, 3:19 PM

Oct 12 2021

douardda updated the diff for D6442: Extract the path slicing logic in a dedicated PathSlicer class.

forgotten print statement...

Oct 12 2021, 5:52 PM
douardda added inline comments to D6442: Extract the path slicing logic in a dedicated PathSlicer class.
Oct 12 2021, 5:50 PM
douardda committed rDDOC807d63991a8e: sysadm: fill the mirror deployment section (authored by douardda).
sysadm: fill the mirror deployment section
Oct 12 2021, 4:53 PM
douardda committed rDDOCfefabca8e6d3: conf: add swh-sysadm intershpinx mapping entry (authored by douardda).
conf: add swh-sysadm intershpinx mapping entry
Oct 12 2021, 2:27 PM
douardda committed rDDOCe6ebb39c4b6d: sysadm: add mirror-operations without content (authored by douardda).
sysadm: add mirror-operations without content
Oct 12 2021, 2:07 PM

Oct 11 2021

douardda added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

An alternative to annotating synthetic refs: add a "type" or "forge_type" attribute to snapshots.

Oct 11 2021, 12:33 PM · Git loader
douardda added a comment to T3632: Investigate the ContentDisallowed exception.

What's the difference in deployed dependencies versions (staging vs. prod)?

Oct 11 2021, 12:15 PM · Scheduling utilities
douardda added a comment to T3621: Create a production read-only objstorage.

For ENEA I'd llike to test different scenarios for the source objstorage:

Oct 11 2021, 12:12 PM · System administration