Page MenuHomeSoftware Heritage

ardumont (Antoine R. Dumont)
UserAdministrator

User Details

User Since
Sep 7 2015, 3:42 PM (222 w, 4 d)
Roles
Administrator

Recent Activity

Yesterday

ardumont committed rSPSITE9c32d3f396b3: data/defaults: Fix expected configuration for the archive cli (authored by ardumont).
data/defaults: Fix expected configuration for the archive cli
Fri, Dec 13, 3:37 PM
ardumont closed T2110: archive-loader: add support for .tar.Z and .tar.lz tarball types as Resolved.

Deployed.

Fri, Dec 13, 1:05 PM · Core Loader, Origin-GNU
ardumont changed the status of T1931: scheduler's cron cleanup error when filtering tasks to archive from Work in Progress to Open.
Fri, Dec 13, 11:55 AM · Scheduling utilities
ardumont added a comment to T1931: scheduler's cron cleanup error when filtering tasks to archive.

(That cron is currently stopped to stop harassing the scheduler for nothing)

Fri, Dec 13, 11:55 AM · Scheduling utilities
ardumont accepted D2447: origin_save: Properly handle server errors on requests submission.
Fri, Dec 13, 11:49 AM
ardumont accepted D2448: Use a btree of (task_type, md5(arguments)) to match task arguments.

Awesome.

Fri, Dec 13, 11:36 AM

Thu, Dec 12

ardumont created D2446: deposit.tests: Cleanup some more dead code.
Thu, Dec 12, 7:23 PM
ardumont committed rDLDHGf167e0a36032: mercurial.loader: Fix configuration key duplication (authored by ardumont).
mercurial.loader: Fix configuration key duplication
Thu, Dec 12, 6:13 PM
ardumont accepted D2441: Remove the creation of the 'load-deposit' task type.
Thu, Dec 12, 6:01 PM
ardumont committed rDLDBASE2d65c127ad67: loader.core: Fix correctly loader initialization (authored by ardumont).
loader.core: Fix correctly loader initialization
Thu, Dec 12, 4:21 PM
ardumont committed rDCORE73a055e6038f: tarball: Improve docstring (authored by ardumont).
tarball: Improve docstring
Thu, Dec 12, 4:21 PM
ardumont committed rDCOREa8dd002073b1: tarball: Improve behavior, docstrings and tests (authored by ardumont).
tarball: Improve behavior, docstrings and tests
Thu, Dec 12, 4:21 PM
ardumont committed rDCORE221ac3a6c6d4: tarball: Add support for tar.x files (authored by ardumont).
tarball: Add support for tar.x files
Thu, Dec 12, 4:21 PM
ardumont committed rDCORE5e3d9bcf0290: tarball: Add support for tar.lz files (authored by ardumont).
tarball: Add support for tar.lz files
Thu, Dec 12, 4:21 PM
ardumont committed rDCOREa808dae19096: tarball.tests: Add more supported tarballs in test (authored by ardumont).
tarball.tests: Add more supported tarballs in test
Thu, Dec 12, 4:21 PM
ardumont committed rDCORE10d9b743210a: tarball: Remove dead code (authored by ardumont).
tarball: Remove dead code
Thu, Dec 12, 4:21 PM
ardumont committed rDCORE36a33bff06d9: tarball: Add support for tar.Z file (authored by ardumont).
tarball: Add support for tar.Z file
Thu, Dec 12, 4:21 PM
ardumont closed D2371: tarball: Add support for tar.Z, tar.x, tar.lz files.
Thu, Dec 12, 4:21 PM
ardumont updated the diff for D2371: tarball: Add support for tar.Z, tar.x, tar.lz files.

Rebase and plug to master

Thu, Dec 12, 4:05 PM
ardumont committed rDLDBASEaffd74b0ccf4: loader.core: Fix initialization issue (authored by ardumont).
loader.core: Fix initialization issue
Thu, Dec 12, 3:45 PM
ardumont closed D2440: loader.core: Fix initialization issue.
Thu, Dec 12, 3:45 PM
ardumont created D2440: loader.core: Fix initialization issue.
Thu, Dec 12, 3:41 PM
ardumont updated the task description for T2120: Loaders: standalone task tests.
Thu, Dec 12, 3:22 PM · Sprint 2019/12 (Monitor and Conquer)
ardumont closed T2134: loader: Implement uniform loading CLI as Resolved.
Thu, Dec 12, 3:21 PM · Sprint 2019/12 (Monitor and Conquer)
ardumont closed T2134: loader: Implement uniform loading CLI, a subtask of T2120: Loaders: standalone task tests, as Resolved.
Thu, Dec 12, 3:21 PM · Sprint 2019/12 (Monitor and Conquer)
ardumont committed rDCORE2ba1714f4689: core.config: Rename configuration key (authored by ardumont).
core.config: Rename configuration key
Thu, Dec 12, 3:17 PM
ardumont committed rSPSITEa2ba7e96a964: loaders: Unify configuration (authored by ardumont).
loaders: Unify configuration
Thu, Dec 12, 2:53 PM
ardumont committed rDLDGd2f62ba96f0d: loader: Bump dependency on loader-core (authored by ardumont).
loader: Bump dependency on loader-core
Thu, Dec 12, 2:40 PM
ardumont closed D2439: loader-git: Bump dependency on loader-core.
Thu, Dec 12, 2:40 PM
ardumont created D2439: loader-git: Bump dependency on loader-core.
Thu, Dec 12, 2:37 PM
ardumont accepted D2435: Add a caveats section in docker/README.md.

Aside one typo, lgtm.

Thu, Dec 12, 2:25 PM
ardumont added inline comments to D2433: Attempt to rationalize a bit workers config in docker.
Thu, Dec 12, 2:23 PM
ardumont accepted D2434: Remove task_modules declarations for lister and loader conf files in docker.

lgtm

Thu, Dec 12, 2:21 PM
ardumont closed D2438: loader-hg: Bump dependency on loader-core.
Thu, Dec 12, 2:14 PM
ardumont committed rDLDHG97d94834a025: loader: Bump dependency on loader-core (authored by ardumont).
loader: Bump dependency on loader-core
Thu, Dec 12, 2:14 PM
ardumont committed rDLDSVN4d036c1a0a3d: svn.loader: Migrate to new buffer/filter behavior (authored by ardumont).
svn.loader: Migrate to new buffer/filter behavior
Thu, Dec 12, 2:13 PM
ardumont closed D2405: svn.loader: Migrate to new buffer/filter behavior.
Thu, Dec 12, 2:13 PM
ardumont created D2438: loader-hg: Bump dependency on loader-core.
Thu, Dec 12, 2:11 PM
ardumont updated the diff for D2405: svn.loader: Migrate to new buffer/filter behavior.

Update to the latest unbugged version

Thu, Dec 12, 2:10 PM
ardumont updated the diff for D2405: svn.loader: Migrate to new buffer/filter behavior.

Update requirements

Thu, Dec 12, 2:09 PM
ardumont committed rDLDBASE2b240a76f08c: loader.core: Type methods (authored by ardumont).
loader.core: Type methods
Thu, Dec 12, 2:00 PM
ardumont committed rDLDBASEec6c412236e8: loader.core: Transform data input into list (authored by ardumont).
loader.core: Transform data input into list
Thu, Dec 12, 2:00 PM
ardumont committed rDLDBASE4a26f9fe68bd: loader.core: Add missing conversion step on content (authored by ardumont).
loader.core: Add missing conversion step on content
Thu, Dec 12, 2:00 PM
ardumont closed D2437: loader.core: Add missing conversion steps on merkle dag data.
Thu, Dec 12, 2:00 PM
ardumont created D2437: loader.core: Add missing conversion steps on merkle dag data.
Thu, Dec 12, 1:56 PM
ardumont updated the diff for D2405: svn.loader: Migrate to new buffer/filter behavior.

Rebase and plug to master

Thu, Dec 12, 12:20 PM
ardumont closed D2403: core.loader: Drop deprecated behavior.
Thu, Dec 12, 12:01 PM
ardumont committed rDLDBASE6d28068548d2: core.loader: Drop deprecated behavior (authored by ardumont).
core.loader: Drop deprecated behavior
Thu, Dec 12, 12:01 PM
ardumont updated the diff for D2403: core.loader: Drop deprecated behavior.

Rebase and plug to master

Thu, Dec 12, 11:58 AM
ardumont committed rDLDBASEaf185a4dfb39: loaders: Add missing class loader docstring (authored by ardumont).
loaders: Add missing class loader docstring
Thu, Dec 12, 11:42 AM
ardumont committed rDLDBASE2dc77128ec5d: test_cli: Test the cli list's standard use (authored by ardumont).
test_cli: Test the cli list's standard use
Thu, Dec 12, 11:42 AM
ardumont committed rDLDBASE6be3534ad1ac: loader.cli: Improve current implementation (authored by ardumont).
loader.cli: Improve current implementation
Thu, Dec 12, 11:42 AM
ardumont closed D2430: loader.cli: Add 2 subcommands, one to load, one to introspect.
Thu, Dec 12, 11:42 AM
ardumont added a comment to D2427: Make load-deposit and check-deposit URL argument absolute.

For the sake of my comment [1]

Thu, Dec 12, 11:38 AM
ardumont requested changes to D2427: Make load-deposit and check-deposit URL argument absolute.
Thu, Dec 12, 11:38 AM
ardumont committed rDLDGd3ce9b5a9c20: tasks: Enforce kwargs use in task message (authored by ardumont).
tasks: Enforce kwargs use in task message
Thu, Dec 12, 11:19 AM
ardumont updated the diff for D2430: loader.cli: Add 2 subcommands, one to load, one to introspect.
  • loaders: Add missing class loader docstrings
  • test_cli: Test the cli list's standard use
Thu, Dec 12, 11:14 AM
ardumont updated the summary of D2430: loader.cli: Add 2 subcommands, one to load, one to introspect.
Thu, Dec 12, 11:13 AM
ardumont updated the diff for D2430: loader.cli: Add 2 subcommands, one to load, one to introspect.

Move import statement at the top of the file

Thu, Dec 12, 11:02 AM
ardumont added inline comments to D2430: loader.cli: Add 2 subcommands, one to load, one to introspect.
Thu, Dec 12, 11:00 AM
ardumont added inline comments to D2429: Add sentry integration to swh-deposit.
Thu, Dec 12, 10:59 AM

Wed, Dec 11

ardumont updated the diff for D2403: core.loader: Drop deprecated behavior.

Rebase on latest origin/master

Wed, Dec 11, 6:27 PM
ardumont created D2430: loader.cli: Add 2 subcommands, one to load, one to introspect.
Wed, Dec 11, 4:31 PM
ardumont updated the task description for T2134: loader: Implement uniform loading CLI.
Wed, Dec 11, 4:30 PM · Sprint 2019/12 (Monitor and Conquer)
ardumont updated the task description for T2120: Loaders: standalone task tests.
Wed, Dec 11, 3:18 PM · Sprint 2019/12 (Monitor and Conquer)
ardumont edited P571 migrate load-* task arguments to use kwargs.
Wed, Dec 11, 3:17 PM · Scheduling utilities
ardumont added a comment to P571 migrate load-* task arguments to use kwargs.

Other load-git migration tryouts (to keep a reference on):

Wed, Dec 11, 3:16 PM · Scheduling utilities
ardumont edited P571 migrate load-* task arguments to use kwargs.
Wed, Dec 11, 3:15 PM · Scheduling utilities
ardumont updated the task description for T2120: Loaders: standalone task tests.
Wed, Dec 11, 2:45 PM · Sprint 2019/12 (Monitor and Conquer)
ardumont committed rDLDBASE82ad3e1079e7: tasks: Enforce kwargs use in task message (authored by ardumont).
tasks: Enforce kwargs use in task message
Wed, Dec 11, 1:54 PM
ardumont committed rDLDHG7238bd53bd5d: tasks: Enforce kwargs use in task message (authored by ardumont).
tasks: Enforce kwargs use in task message
Wed, Dec 11, 1:31 PM
ardumont committed rDLDSVN36739d14b16e: tasks: Enforce kwargs use in task message (authored by ardumont).
tasks: Enforce kwargs use in task message
Wed, Dec 11, 1:31 PM
ardumont committed rSPSITE6f94548b2dce: loader: Add buffer release setup (authored by ardumont).
loader: Add buffer release setup
Wed, Dec 11, 11:23 AM
ardumont committed rSPSITE9c1bb521ed85: lister: Let worker01 deal with other listers than github (authored by ardumont).
lister: Let worker01 deal with other listers than github
Wed, Dec 11, 11:23 AM
ardumont edited P571 migrate load-* task arguments to use kwargs.
Wed, Dec 11, 11:20 AM · Scheduling utilities
ardumont updated the title for P571 migrate load-* task arguments to use kwargs from migrate load-npm task arguments to migrate load-* task arguments to use kwargs.
Wed, Dec 11, 11:16 AM · Scheduling utilities
ardumont added a comment to D2427: Make load-deposit and check-deposit URL argument absolute.

tests should pass ok however no test exists for the interaction with the

scheduler, so...

Wed, Dec 11, 11:03 AM
ardumont added a comment to T2134: loader: Implement uniform loading CLI.

The good news is that it's not the big part of the task.
So, yes, i'll improve on the current implementation that after i'm done deploying and migrating all the things ;)
The big part is the implied refactoring pulled by this.
Which i'm almost done (that's the deploy and migrate part i'm doing).

Wed, Dec 11, 10:48 AM · Sprint 2019/12 (Monitor and Conquer)

Tue, Dec 10

ardumont updated the task description for T2120: Loaders: standalone task tests.
Tue, Dec 10, 6:46 PM · Sprint 2019/12 (Monitor and Conquer)
ardumont created P571 migrate load-* task arguments to use kwargs.
Tue, Dec 10, 6:40 PM · Scheduling utilities
ardumont accepted D2424: Enable Sentry for CLIs, and Gunicorn and Celery workers..
Tue, Dec 10, 3:06 PM
ardumont added inline comments to D2424: Enable Sentry for CLIs, and Gunicorn and Celery workers..
Tue, Dec 10, 3:05 PM
ardumont accepted D2421: docker/deposit: configure nginx so the WSGI mountpoint for /deposit is ok-ish.
Tue, Dec 10, 2:59 PM
ardumont added inline comments to D2424: Enable Sentry for CLIs, and Gunicorn and Celery workers..
Tue, Dec 10, 2:58 PM
ardumont added inline comments to D2424: Enable Sentry for CLIs, and Gunicorn and Celery workers..
Tue, Dec 10, 2:57 PM
ardumont committed rDSTObd2a1963930c: storage: Make origin_get_random simpler and faster (authored by ardumont).
storage: Make origin_get_random simpler and faster
Tue, Dec 10, 2:48 PM
ardumont committed rDSTOb440d3a45e29: storage: Prefer sample query on origin_visit (authored by ardumont).
storage: Prefer sample query on origin_visit
Tue, Dec 10, 2:48 PM
ardumont committed rDSTO3855b6ef1a97: tox.ini: Add a py3-dev environment (authored by ardumont).
tox.ini: Add a py3-dev environment
Tue, Dec 10, 2:48 PM
ardumont committed rDSTOa2401b553b5e: storage: Add endpoint to randomly pick an origin (authored by ardumont).
storage: Add endpoint to randomly pick an origin
Tue, Dec 10, 2:48 PM
ardumont closed D2404: storage: Add endpoint to randomly pick an origin.
Tue, Dec 10, 2:48 PM
ardumont updated the summary of D2404: storage: Add endpoint to randomly pick an origin.
Tue, Dec 10, 2:45 PM
ardumont updated the diff for D2404: storage: Add endpoint to randomly pick an origin.

Use mix of CTE (with index) and random() to make query faster

Tue, Dec 10, 2:41 PM
ardumont accepted D2422: Save code now: Fix scheduler task parameters.

yeah, that simplifies things!

Tue, Dec 10, 2:02 PM
ardumont added a comment to D2404: storage: Add endpoint to randomly pick an origin.

materialized views.

We could implement a materialized view on a sliding window of the last 3 months visits.
And take a sample out of it.
pros:

  • simpler? (i'm not that convinced ;)

cons:

  • "more" maintenance (when and how to refresh it ~> periodic cron every night?)
  • less parametric (current code is not, but we could adapt the current code to parametrize the current 3 months interval)
Tue, Dec 10, 1:25 PM
ardumont added a comment to D2404: storage: Add endpoint to randomly pick an origin.

materialized views.

Tue, Dec 10, 12:30 PM
ardumont added a comment to D2404: storage: Add endpoint to randomly pick an origin.
Tue, Dec 10, 12:14 PM
ardumont added a comment to D2404: storage: Add endpoint to randomly pick an origin.

Did we really do any svn loads in the last three months? Maybe there's really nothing to show. In that case, the query _will_ do a sequential scan of the whole table.

Tue, Dec 10, 12:00 PM
ardumont committed rDVAU7bebb14a6ff1: Fix tests data initialization after git loader api changes (authored by ardumont).
Fix tests data initialization after git loader api changes
Tue, Dec 10, 11:38 AM
ardumont closed D2412: vault: Fix tests data initialization after git loader api changes.
Tue, Dec 10, 11:38 AM
ardumont added inline comments to D2404: storage: Add endpoint to randomly pick an origin.
Tue, Dec 10, 11:34 AM