Page MenuHomeSoftware Heritage
Feed Advanced Search

Apr 26 2022

douardda updated the diff for D7692: Make swh.objstorage an optional dependency, as a 'with-content' extra.

typo + forgot the requirement file

Apr 26 2022, 6:01 PM
douardda requested review of D7692: Make swh.objstorage an optional dependency, as a 'with-content' extra.
Apr 26 2022, 5:55 PM
douardda updated the diff for D7647: Update the the listing and loading scheduling architecture doc.

rephrase (journal usage by the scheduler)

Apr 26 2022, 5:16 PM
douardda added a comment to D7647: Update the the listing and loading scheduling architecture doc.
In D7647#200113, @olasd wrote:

Thanks. On the diagram, it looks like 9' is causing 12 (via 10 and 11), which is not true (yet! D7332 isn't agreed upon) and could be very confusing.

This is because 9' writes to the origin_visit table, but 10/11/12 are based on the origin table/topic.

Could you split swh-storage in two, to avoid this misunderstanding?

To clear the misunderstanding maybe the label for 10 could be Consume final "origin visit status" entries?

Apr 26 2022, 5:10 PM
douardda updated the diff for D7647: Update the the listing and loading scheduling architecture doc.

typos and better rst syntax as suggested by vlorentz

Apr 26 2022, 2:05 PM
douardda added a comment to D7647: Update the the listing and loading scheduling architecture doc.

Fichier archi scheduler:

Apr 26 2022, 1:51 PM
douardda requested review of D7647: Update the the listing and loading scheduling architecture doc.
Apr 26 2022, 12:36 PM
douardda closed D7652: Update a bit the documentation for the new origin visit scheduler.
Apr 26 2022, 10:46 AM
douardda committed rDSCH3687931f78b5: Update a bit the documentation for the new origin visit scheduler (authored by douardda).
Update a bit the documentation for the new origin visit scheduler
Apr 26 2022, 10:46 AM
douardda updated the diff for D7652: Update a bit the documentation for the new origin visit scheduler.

typos

Apr 26 2022, 10:38 AM
douardda added a comment to D7652: Update a bit the documentation for the new origin visit scheduler.

lgtm

couple of typos to fix.

Apr 26 2022, 10:37 AM

Apr 25 2022

douardda requested review of D7652: Update a bit the documentation for the new origin visit scheduler.
Apr 25 2022, 6:20 PM

Apr 21 2022

douardda closed D7589: Add a --margin option to the `swh dataset graph export` command.
Apr 21 2022, 10:10 AM
douardda committed rDDATASET07bcf1674e0a: Add a --margin option to the `swh dataset graph export` command (authored by douardda).
Add a --margin option to the `swh dataset graph export` command
Apr 21 2022, 10:10 AM
douardda closed D7588: Add a --type option to the `swh dataset graph export` command.
Apr 21 2022, 10:10 AM
douardda committed rDDATASET4154d43a4c55: Add a --types option to the `swh dataset graph export` command (authored by douardda).
Add a --types option to the `swh dataset graph export` command
Apr 21 2022, 10:10 AM
douardda added inline comments to D7588: Add a --type option to the `swh dataset graph export` command.
Apr 21 2022, 10:07 AM
douardda closed D7531: Regularly check for EOF in Client.process() while waiting for messages.
Apr 21 2022, 10:03 AM
douardda committed rDJNLf98248dd6938: Regularly check for EOF in Client.process() while waiting for messages (authored by douardda).
Regularly check for EOF in Client.process() while waiting for messages
Apr 21 2022, 10:03 AM
douardda updated the diff for D7531: Regularly check for EOF in Client.process() while waiting for messages.

typo

Apr 21 2022, 9:56 AM
douardda updated the diff for D7589: Add a --margin option to the `swh dataset graph export` command.

rebase

Apr 21 2022, 9:52 AM
douardda updated the diff for D7588: Add a --type option to the `swh dataset graph export` command.

thx vlorentz

Apr 21 2022, 9:51 AM
douardda updated the diff for D7589: Add a --margin option to the `swh dataset graph export` command.

thx vlorentz

Apr 21 2022, 9:46 AM
douardda added inline comments to D7589: Add a --margin option to the `swh dataset graph export` command.
Apr 21 2022, 9:44 AM

Apr 20 2022

douardda closed D7591: Make scheduling policy used in schedule_recurrent configurable.
Apr 20 2022, 6:27 PM
douardda committed rDSCHa76bb02f0e94: Make scheduling policy used in schedule_recurrent configurable (authored by douardda).
Make scheduling policy used in schedule_recurrent configurable
Apr 20 2022, 6:27 PM
douardda added inline comments to D7591: Make scheduling policy used in schedule_recurrent configurable.
Apr 20 2022, 5:15 PM
douardda updated the diff for D7591: Make scheduling policy used in schedule_recurrent configurable.

Improve docstrings, better config validation, use lists in grab_next_visits_policy_weights()

Apr 20 2022, 4:36 PM
douardda updated the summary of D7591: Make scheduling policy used in schedule_recurrent configurable.
Apr 20 2022, 9:34 AM
douardda updated the summary of D7591: Make scheduling policy used in schedule_recurrent configurable.
Apr 20 2022, 9:33 AM
douardda retitled D7591: Make scheduling policy used in schedule_recurrent configurable from [wip] Make scheduling policy used in schedule_recurrent configurable to Make scheduling policy used in schedule_recurrent configurable.
Apr 20 2022, 9:33 AM
douardda updated the diff for D7591: Make scheduling policy used in schedule_recurrent configurable.

Use a flatter config structure

Apr 20 2022, 9:33 AM

Apr 15 2022

douardda requested review of D7591: Make scheduling policy used in schedule_recurrent configurable.
Apr 15 2022, 6:24 PM
douardda updated the diff for D7589: Add a --margin option to the `swh dataset graph export` command.

Improve and simplify a bit the code

Apr 15 2022, 1:27 PM
douardda updated the summary of D7589: Add a --margin option to the `swh dataset graph export` command.
Apr 15 2022, 12:55 PM
douardda accepted D7580: sentry: always override init settings with the environment variables.
Apr 15 2022, 12:42 PM
douardda updated the diff for D7531: Regularly check for EOF in Client.process() while waiting for messages.

typo (thx ardumont)

Apr 15 2022, 12:41 PM
douardda added inline comments to D7531: Regularly check for EOF in Client.process() while waiting for messages.
Apr 15 2022, 12:38 PM
douardda requested review of D7589: Add a --margin option to the `swh dataset graph export` command.
Apr 15 2022, 12:31 PM
douardda requested review of D7588: Add a --type option to the `swh dataset graph export` command.
Apr 15 2022, 12:31 PM

Apr 14 2022

douardda added a comment to D7580: sentry: always override init settings with the environment variables.

lgtm, maybe add a comment in test_cli.py to justify the presence of forked over there

Apr 14 2022, 3:35 PM
douardda accepted D7570: Add support for disabling logging integration in sentry.
Apr 14 2022, 12:19 PM
douardda committed rDSTOf136559bbdd7: User logger everywhere in tenacious.py (authored by douardda).
User logger everywhere in tenacious.py
Apr 14 2022, 12:14 PM

Apr 13 2022

douardda requested review of D7570: Add support for disabling logging integration in sentry.
Apr 13 2022, 4:23 PM

Apr 12 2022

douardda accepted D7558: journalprocessor: save final offsets to a text file.
Apr 12 2022, 5:04 PM
douardda claimed T3087: Implement support for takedown notices (infra, admin tools, workflow).
Apr 12 2022, 12:34 PM · Roadmap 2022, meta-task, Roadmap 2021, Web app
douardda accepted D7552: origin_get_with_statuses: Fix case when fetched visits list is empty.
Apr 12 2022, 12:26 PM
douardda accepted D7517: Add support for multipart/mixed + better fallback for multipart/*.

lgtm

Apr 12 2022, 12:24 PM

Apr 11 2022

douardda closed D7536: Add a SWH_MODEL_OBJECT_TYPES map {cls.objet_type: cls} in model.py.
Apr 11 2022, 12:32 PM
douardda committed rDMOD85f36f84347a: Add a SWH_MODEL_OBJECT_TYPES map {cls.objet_type: cls} in model.py (authored by douardda).
Add a SWH_MODEL_OBJECT_TYPES map {cls.objet_type: cls} in model.py
Apr 11 2022, 12:32 PM
douardda updated the diff for D7536: Add a SWH_MODEL_OBJECT_TYPES map {cls.objet_type: cls} in model.py.

rebase

Apr 11 2022, 12:29 PM
douardda updated the diff for D7536: Add a SWH_MODEL_OBJECT_TYPES map {cls.objet_type: cls} in model.py.

Add type annotation for SWH_MODEL_OBJECT_TYPES

Apr 11 2022, 10:25 AM
douardda added inline comments to D7536: Add a SWH_MODEL_OBJECT_TYPES map {cls.objet_type: cls} in model.py.
Apr 11 2022, 10:17 AM

Apr 8 2022

douardda added inline comments to D7536: Add a SWH_MODEL_OBJECT_TYPES map {cls.objet_type: cls} in model.py.
Apr 8 2022, 5:11 PM
douardda added inline comments to D7536: Add a SWH_MODEL_OBJECT_TYPES map {cls.objet_type: cls} in model.py.
Apr 8 2022, 5:08 PM
douardda added inline comments to D7536: Add a SWH_MODEL_OBJECT_TYPES map {cls.objet_type: cls} in model.py.
Apr 8 2022, 5:06 PM
douardda requested review of D7531: Regularly check for EOF in Client.process() while waiting for messages.
Apr 8 2022, 4:57 PM
douardda requested review of D7536: Add a SWH_MODEL_OBJECT_TYPES map {cls.objet_type: cls} in model.py.
Apr 8 2022, 12:48 PM
douardda requested review of D7520: Add a ORC file loading function.
Apr 8 2022, 12:15 PM
douardda closed D7519: Reduce cli's loading time by moving import statements in commands.
Apr 8 2022, 11:48 AM
douardda committed rDDATASET18325cc8e78e: Reduce cli's loading time by moving import statements in commands (authored by douardda).
Reduce cli's loading time by moving import statements in commands
Apr 8 2022, 11:48 AM
douardda added inline comments to D7519: Reduce cli's loading time by moving import statements in commands.
Apr 8 2022, 11:36 AM
douardda accepted D7518: Add support for recursive multipart messages.

LGTM

Apr 8 2022, 11:34 AM

Apr 7 2022

douardda requested review of D7519: Reduce cli's loading time by moving import statements in commands.
Apr 7 2022, 12:05 PM
douardda closed D7465: Add support for blob in content export.
Apr 7 2022, 12:02 PM
douardda committed rDDATASET9d97f0c0826d: Add support for blob in content export (authored by douardda).
Add support for blob in content export
Apr 7 2022, 12:02 PM
douardda updated the diff for D7465: Add support for blob in content export.

rebase

Apr 7 2022, 11:24 AM

Apr 6 2022

douardda accepted D7506: Add support for HAL-ID as identifier.
Apr 6 2022, 3:23 PM
douardda accepted D7433: Add Fixer class, which re-loads corrupt objects from origins.

As I said, I'm not completely enthusiastic by this code (especially the lack of flexibility and versatility to provision for other origin sources than git) but it's a nice start at least.
Thanks

Apr 6 2022, 3:02 PM
douardda closed D7493: docker: make yarn run in the local copy of the swh-web source code.
Apr 6 2022, 2:33 PM
douardda committed rDENVb4ddd00b694a: docker: make yarn run in the local copy of the swh-web source code (authored by douardda).
docker: make yarn run in the local copy of the swh-web source code
Apr 6 2022, 2:33 PM
douardda closed D7039: Update the debian local package building section.
Apr 6 2022, 2:24 PM
douardda committed rDDOC94533948188e: Update the debian local package building section (authored by douardda).
Update the debian local package building section
Apr 6 2022, 2:24 PM
douardda updated the diff for D7039: Update the debian local package building section.

rebase

Apr 6 2022, 2:24 PM
douardda updated the diff for D7493: docker: make yarn run in the local copy of the swh-web source code.

adapt code according to anlambert's suggestion

Apr 6 2022, 1:43 PM
douardda added a comment to D7433: Add Fixer class, which re-loads corrupt objects from origins.

I think I'd prefer not to have 3 versions of almost the same function (corrupt_object_(get|grab_xxx) ).

Apr 6 2022, 10:51 AM

Apr 5 2022

douardda added a comment to D7506: Add support for HAL-ID as identifier.

lgtm but I'm not sure to understand what's at stake here.

Apr 5 2022, 5:01 PM
douardda updated the diff for D7039: Update the debian local package building section.

specify aptitude is to be used only when needed (sic)

Apr 5 2022, 4:57 PM
douardda added a comment to D7039: Update the debian local package building section.

Ping, what do we do with this diff?

Apr 5 2022, 4:55 PM
douardda added a comment to D7039: Update the debian local package building section.
Apr 5 2022, 4:54 PM
douardda updated the diff for D7258: Update the mirror operation docker manual.

typos (thx ardumont) + add a warning about db migration

Apr 5 2022, 4:25 PM
douardda added a comment to D7465: Add support for blob in content export.

LGTM, but we should eventually replace self.objstorage.get() with self.objstorage.get_batch(), it is much faster with the Azure backend (and hopefully more backends in the future, inc. Winery!)

Apr 5 2022, 4:11 PM

Apr 4 2022

douardda closed D7492: docker: add librdkakfa-dev so confluent-kafka in pip installable on non-x86 platforms.
Apr 4 2022, 5:06 PM
douardda committed rDENV995e90739c3d: docker: add librdkakfa-dev so confluent-kafka in pip installable on non-x86… (authored by douardda).
docker: add librdkakfa-dev so confluent-kafka in pip installable on non-x86…
Apr 4 2022, 5:06 PM
douardda updated the diff for D7493: docker: make yarn run in the local copy of the swh-web source code.

fix indentation and do only call 'yarn build-dev'

Apr 4 2022, 5:06 PM
douardda closed D7473: Make postgresql's Storage client options configurable from config.
Apr 4 2022, 4:59 PM
douardda committed rDSTOc6dc5cd3b58a: Make postgresql's Storage client options configurable from config (authored by douardda).
Make postgresql's Storage client options configurable from config
Apr 4 2022, 4:59 PM
douardda updated the diff for D7473: Make postgresql's Storage client options configurable from config.

better docstring (thx olasd)

Apr 4 2022, 4:44 PM
douardda updated the diff for D7473: Make postgresql's Storage client options configurable from config.

docstring

Apr 4 2022, 4:08 PM
douardda added a comment to D7473: Make postgresql's Storage client options configurable from config.

Please add a docstring; it's unclear this is used by swh-core otherwise

Apr 4 2022, 3:41 PM
douardda closed D7472: Make db_transaction's client_options configurable at run time.
Apr 4 2022, 3:38 PM
douardda committed rDCORE2a2615357b07: Make db_transaction's client_options configurable at run time (authored by douardda).
Make db_transaction's client_options configurable at run time
Apr 4 2022, 3:38 PM
douardda added a comment to D7472: Make db_transaction's client_options configurable at run time.
In D7472#195801, @olasd wrote:

The duplication between db_transaction and db_transaction_generator looks a bit silly, but I'm not sure there's much to do about it, so this lgtm, thanks.

Apr 4 2022, 3:37 PM
douardda added inline comments to D7493: docker: make yarn run in the local copy of the swh-web source code.
Apr 4 2022, 3:33 PM
douardda requested review of D7493: docker: make yarn run in the local copy of the swh-web source code.
Apr 4 2022, 2:06 PM
douardda updated the diff for D7492: docker: add librdkakfa-dev so confluent-kafka in pip installable on non-x86 platforms.

wrong head...

Apr 4 2022, 2:06 PM
douardda requested review of D7492: docker: add librdkakfa-dev so confluent-kafka in pip installable on non-x86 platforms.
Apr 4 2022, 2:05 PM

Apr 1 2022

douardda accepted D7483: Ensure that tests run with the C.UTF-8 locale.
Apr 1 2022, 11:36 AM
douardda published D7483: Ensure that tests run with the C.UTF-8 locale for review.
Apr 1 2022, 11:36 AM

Mar 31 2022

douardda accepted D7478: Clean up `model` removing old unused logic.
Mar 31 2022, 3:54 PM