Page MenuHomeSoftware Heritage

douardda (David Douard)
User

User Details

User Since
Jul 10 2018, 12:38 PM (32 w, 3 d)

Recent Activity

Today

Herald added a reviewer for D1181: Move bin/ scripts in cli.py and document a bit how to use the objstorage.: Reviewers.
Fri, Feb 22, 5:39 PM
douardda added a comment to D1176: Fix nginx revert proxy for rabbitmq.

What does this do?

Fri, Feb 22, 3:23 PM
douardda accepted D1165: Add a CLI tool to reindex origins based on mapping used..
Fri, Feb 22, 11:27 AM

Yesterday

Herald added a reviewer for D1177: Fix celery workers conf files: explicit list of queues is now required: Reviewers.
Thu, Feb 21, 6:32 PM
Herald added a reviewer for D1176: Fix nginx revert proxy for rabbitmq: Reviewers.
Thu, Feb 21, 6:31 PM
Herald added a reviewer for D1175: Upgrade to pg 11: Reviewers.
Thu, Feb 21, 6:31 PM
Herald added a reviewer for D1174: Activate a 10s heartbeat for celery workers: Reviewers.
Thu, Feb 21, 6:31 PM
douardda accepted D1166: Double the timeout of snapshot_get_latest..
Thu, Feb 21, 11:21 AM
douardda added a comment to T1442: Replace Munin graphs with Grafana/Prometheus dashboards.

Since the current status of munin's pg monitoring is inconsistent (eg. awure's dbreplica0 has not pg curve), let's get rid of all munin's pg monitors so there is less confusion, and is one step towards T1356.

Thu, Feb 21, 11:06 AM · Sprint 2018 12, System administration
douardda requested changes to D1165: Add a CLI tool to reindex origins based on mapping used..

Please add an entry_point in the setup.py for this cli.

Thu, Feb 21, 10:42 AM
douardda committed rDCIDX76e9796fa475: config: use already loaded swh config, if any, when instantiating an Indexer (authored by douardda).
config: use already loaded swh config, if any, when instantiating an Indexer
Thu, Feb 21, 10:17 AM
douardda closed D1148: config: use already loaded swh config, if any, when instantiating an Indexer.
Thu, Feb 21, 10:17 AM
douardda updated the diff for D1148: config: use already loaded swh config, if any, when instantiating an Indexer.

rebased

Thu, Feb 21, 10:17 AM

Wed, Feb 20

douardda accepted D1161: deposit: Use explicit configuration.
Wed, Feb 20, 6:02 PM · SWORD deposit, Puppet recipes
douardda added a comment to D398: [WIP] "packing" object storage design documentation.

No idea whether if it's of some interest for our subject, but we may also have a look at openio

Wed, Feb 20, 1:09 PM
douardda committed rDWAPPS7aa941d7b2dd: Update vault service config for vault 0.0.23 (authored by douardda).
Update vault service config for vault 0.0.23
Wed, Feb 20, 12:02 PM
douardda closed D1114: Update vault service config for vault 0.0.22.
Wed, Feb 20, 12:02 PM
douardda updated the diff for D1114: Update vault service config for vault 0.0.22.

rebased and fix requirements-swh.txt

Wed, Feb 20, 12:02 PM
douardda committed rDSCH058f12001ebf: cli: add a --runs option to 'swh-scheduler task list' tool (authored by douardda).
cli: add a --runs option to 'swh-scheduler task list' tool
Wed, Feb 20, 11:31 AM
douardda closed D1146: cli: add a --runs option to 'swh-scheduler task list' tool.
Wed, Feb 20, 11:31 AM
douardda committed rDSCH7f951d393737: Keep the main loaded configuration as swh.scheduler.CONFIG (authored by douardda).
Keep the main loaded configuration as swh.scheduler.CONFIG
Wed, Feb 20, 10:24 AM
douardda closed D1147: Keep the main loaded configuration as swh.scheduler.CONFIG.
Wed, Feb 20, 10:24 AM
douardda committed rDSCH73bca41c6de3: celery_backend: remove debug statements and reduce info level verbosity (authored by douardda).
celery_backend: remove debug statements and reduce info level verbosity
Wed, Feb 20, 10:24 AM
douardda closed D1145: celery_backend: remove debug statements and reduce info level verbosity.
Wed, Feb 20, 10:24 AM
douardda updated the diff for D1146: cli: add a --runs option to 'swh-scheduler task list' tool.

rebase

Wed, Feb 20, 10:24 AM
douardda updated the diff for D1147: Keep the main loaded configuration as swh.scheduler.CONFIG.

rebased

Wed, Feb 20, 10:23 AM
douardda updated the diff for D1147: Keep the main loaded configuration as swh.scheduler.CONFIG.

rebased

Wed, Feb 20, 10:22 AM
douardda updated the diff for D1146: cli: add a --runs option to 'swh-scheduler task list' tool.

add tests... and consequently fix the client part of the new backend method

Wed, Feb 20, 10:22 AM
douardda updated the diff for D1145: celery_backend: remove debug statements and reduce info level verbosity.

rebased

Wed, Feb 20, 10:21 AM

Tue, Feb 19

douardda added a comment to D398: [WIP] "packing" object storage design documentation.

One interesting project we can look at is seaweedfs which implements ideas presented in this paper of fb's Haystack.

Tue, Feb 19, 5:43 PM
douardda added a comment to T1526: Install a new VPN endpoint at Rocquencourt.

Thanks @olasd for this piece of information.

Tue, Feb 19, 12:17 PM · System administration
douardda added inline comments to D1146: cli: add a --runs option to 'swh-scheduler task list' tool.
Tue, Feb 19, 9:55 AM

Fri, Feb 15

Herald added a reviewer for D1148: config: use already loaded swh config, if any, when instantiating an Indexer: Reviewers.
Fri, Feb 15, 5:19 PM
Herald added a reviewer for D1147: Keep the main loaded configuration as swh.scheduler.CONFIG: Reviewers.
Fri, Feb 15, 5:11 PM
Herald added a reviewer for D1146: cli: add a --runs option to 'swh-scheduler task list' tool: Reviewers.
Fri, Feb 15, 5:11 PM
Herald added a reviewer for D1145: celery_backend: remove debug statements and reduce info level verbosity: Reviewers.
Fri, Feb 15, 5:11 PM

Thu, Feb 14

douardda accepted D1121: Generate pom.xml files ourselves instead of using xmltodict.unparse..
Thu, Feb 14, 2:56 PM
douardda accepted D1134: lister: Update configuration scheme change.
Thu, Feb 14, 2:40 PM
douardda committed rDVAU45d95d0ea2fd: Rewrite tests using pytest's fixtures and adapt them to recent refactorings (authored by douardda).
Rewrite tests using pytest's fixtures and adapt them to recent refactorings
Thu, Feb 14, 2:39 PM
douardda committed rDVAU8a3aad37c42e: Normalize the configuration of VaultBackend and cooker (authored by douardda).
Normalize the configuration of VaultBackend and cooker
Thu, Feb 14, 2:39 PM
douardda committed rDVAUf4ad96907d81: Make it possible to specify the config file via SWH_CONFIG_FILENAME env var (authored by douardda).
Make it possible to specify the config file via SWH_CONFIG_FILENAME env var
Thu, Feb 14, 2:39 PM
douardda committed rDVAUf0ad1f01a741: Refactor the VaultBackend to use BaseDb and pool-based db access (authored by douardda).
Refactor the VaultBackend to use BaseDb and pool-based db access
Thu, Feb 14, 2:39 PM
douardda committed rDVAU62963319751f: Add a swh.vault.api.wsgi module to instanciate the (singleton) wsgi app object (authored by douardda).
Add a swh.vault.api.wsgi module to instanciate the (singleton) wsgi app object
Thu, Feb 14, 2:39 PM
douardda committed rDVAU47ad8a9cc24c: Extract the main cli command in a dedicated cli.py module and make it an… (authored by douardda).
Extract the main cli command in a dedicated cli.py module and make it an…
Thu, Feb 14, 2:39 PM
douardda closed D1108: Refactor Vault a bit, mainly pytest, config and BaseDb-based VaultBackend.
Thu, Feb 14, 2:39 PM
douardda updated the diff for D1108: Refactor Vault a bit, mainly pytest, config and BaseDb-based VaultBackend.

Fix tests

Thu, Feb 14, 2:38 PM
douardda updated the diff for D1108: Refactor Vault a bit, mainly pytest, config and BaseDb-based VaultBackend.

Few additions to the original diff

Thu, Feb 14, 2:11 PM
douardda abandoned D1135: Extract the main cli command in a dedicated cli.py module and make it an entrypoint.

stupid arc tool

Thu, Feb 14, 2:08 PM
Herald added a reviewer for D1135: Extract the main cli command in a dedicated cli.py module and make it an entrypoint: Reviewers.
Thu, Feb 14, 2:06 PM
douardda updated the diff for D1108: Refactor Vault a bit, mainly pytest, config and BaseDb-based VaultBackend.

fix typo reported by ardumont

Thu, Feb 14, 12:17 PM
douardda added inline comments to D1108: Refactor Vault a bit, mainly pytest, config and BaseDb-based VaultBackend.
Thu, Feb 14, 12:08 PM
douardda requested changes to D1122: Use xmlschema instead of xmltodict for parsing pom.xml..
  • Commit missing stuff.
  • Fix commit again.
  • Fix requirements.txt
  • Add missing tests.
Thu, Feb 14, 9:52 AM
douardda requested changes to D1121: Generate pom.xml files ourselves instead of using xmltodict.unparse..

Please give a bit of context in your commit messages (in git). Like these 2 commits are dealing with tests for example. And 'More keys' is not a good commit message. Like 'fix a bug' is not a valid commit message, generally speaking.

Thu, Feb 14, 9:48 AM
douardda accepted D1127: rpc-server: Update http check to query with 'Accept' header.
Thu, Feb 14, 9:44 AM · Puppet recipes
douardda accepted D1128: api/server: Do not read configuration at each request.
Thu, Feb 14, 9:43 AM
douardda accepted D1129: api/server: Do not read configuration at each request.
Thu, Feb 14, 9:43 AM
douardda accepted D1130: api/server: Do not read configuration at each request.

A bit 'how you doing' there, but 1/ it's kind of consistent with other similar modules in other swh packages, and 2/ who cares?

Thu, Feb 14, 9:42 AM
douardda accepted D1117: scheduler: Update configuration according to latest development.

LGTM, but hey, these are puppets!

Thu, Feb 14, 9:39 AM · Puppet recipes
douardda accepted D1131: docs/developer-setup: Update documentation about auto-completion.
Thu, Feb 14, 9:32 AM · Development environment, Development documentation

Wed, Feb 13

douardda committed rDSCHf0a8c43f06f7: listener: make the listener's queue name independent from the hostname (authored by douardda).
listener: make the listener's queue name independent from the hostname
Wed, Feb 13, 3:25 PM
douardda committed rDSCHb423c0b0b30e: runner: fix task_run configuration bootstrap (authored by douardda).
runner: fix task_run configuration bootstrap
Wed, Feb 13, 3:25 PM
douardda committed rDSCH3d761f162de3: sql: add the swh-lister-bitbucket-* task types (authored by douardda).
sql: add the swh-lister-bitbucket-* task types
Wed, Feb 13, 3:25 PM
douardda committed rDSCH7e3f2fc03f76: task: do not send the task-result-exception event in Task.on_failure() (authored by douardda).
task: do not send the task-result-exception event in Task.on_failure()
Wed, Feb 13, 3:25 PM

Tue, Feb 12

douardda created P358 origin visit histogram.
Tue, Feb 12, 4:41 PM
douardda committed rCDFD2ed0c7833c5b: Add the vault services (api and worker) and make them avaible in swh-web (authored by douardda).
Add the vault services (api and worker) and make them avaible in swh-web
Tue, Feb 12, 1:30 PM
douardda committed rCDFDbb38cd93622e: Deploy a djangossettings.py for swh-web that allows to set a dumd asmin password (authored by douardda).
Deploy a djangossettings.py for swh-web that allows to set a dumd asmin password
Tue, Feb 12, 1:29 PM
douardda committed rCDFD0ac8db330055: Move options and arguments of the swh manage command as COMMAND (authored by douardda).
Move options and arguments of the swh manage command as COMMAND
Tue, Feb 12, 1:29 PM
douardda closed D1115: Add the vault services (api and worker) and make them avaible in swh-web.
Tue, Feb 12, 1:29 PM
douardda updated the diff for D1115: Add the vault services (api and worker) and make them avaible in swh-web.

Fix indentation in swh-vault/entrypoint.sh

Tue, Feb 12, 1:29 PM
Herald added a reviewer for D1115: Add the vault services (api and worker) and make them avaible in swh-web: Reviewers.
Tue, Feb 12, 12:25 PM
douardda abandoned D1113: Do not override the DJANGO_SETTINGS_MODULE env var if it's already set.

Actually not needed

Tue, Feb 12, 12:11 PM
douardda updated the diff for D1114: Update vault service config for vault 0.0.22.

rebased

Tue, Feb 12, 12:11 PM
Herald added a reviewer for D1114: Update vault service config for vault 0.0.22: Reviewers.
Tue, Feb 12, 12:03 PM
Herald added a reviewer for D1113: Do not override the DJANGO_SETTINGS_MODULE env var if it's already set: Reviewers.
Tue, Feb 12, 12:00 PM
douardda updated the diff for D1108: Refactor Vault a bit, mainly pytest, config and BaseDb-based VaultBackend.

Important rework of the config handling

Tue, Feb 12, 11:01 AM
douardda added a comment to D1108: Refactor Vault a bit, mainly pytest, config and BaseDb-based VaultBackend.

I'm (re)working this diff, so wait a bit before spending time reviewing it.

Tue, Feb 12, 9:41 AM
douardda committed rCDFD452185dac5ef: Very beginning of a grafana service (authored by douardda).
Very beginning of a grafana service
Tue, Feb 12, 9:38 AM
douardda closed D1094: Very beginning of a grafana service.
Tue, Feb 12, 9:38 AM
douardda committed rCDFD7000265592f0: Update the lister entrypoint to make latest cli updates (authored by douardda).
Update the lister entrypoint to make latest cli updates
Tue, Feb 12, 9:38 AM
douardda closed D1093: Update the lister entrypoint to make latest cli updates.
Tue, Feb 12, 9:38 AM
douardda committed rDDOC9e6c2f3a614d: Split the getting-started guide in 3 parts (authored by douardda).
Split the getting-started guide in 3 parts
Tue, Feb 12, 9:36 AM
douardda closed D1086: Split the getting-started guide in 3 parts.
Tue, Feb 12, 9:36 AM
douardda updated the diff for D1086: Split the getting-started guide in 3 parts.

typos

Tue, Feb 12, 9:35 AM
douardda accepted D1077: Add more type checks in metadata dictionary..

Let's keep on, but I really don't like the turn this code takes...

Tue, Feb 12, 9:32 AM
douardda accepted D1110: gbp-buildpackage: Open pgdg as a new extra debian repository.

lgtm

Tue, Feb 12, 9:23 AM

Mon, Feb 11

douardda added inline comments to D1086: Split the getting-started guide in 3 parts.
Mon, Feb 11, 5:30 PM
douardda updated the diff for D1086: Split the getting-started guide in 3 parts.

rebased and fixes reported by aardumont and vlorentz

Mon, Feb 11, 5:30 PM

Fri, Feb 8

douardda updated the diff for D1108: Refactor Vault a bit, mainly pytest, config and BaseDb-based VaultBackend.

Add missing hunks in tox.ini and requirement-test.txt

Fri, Feb 8, 5:37 PM
Herald added a reviewer for D1108: Refactor Vault a bit, mainly pytest, config and BaseDb-based VaultBackend: Reviewers.
Fri, Feb 8, 5:31 PM
douardda added a comment to D1086: Split the getting-started guide in 3 parts.

I'm a bit puzzled that with so many comments, this diff has been accepted! It should not have been so IMHO.

Fri, Feb 8, 2:14 PM
douardda accepted D1079: Add checks in the idx_storage that the same content/rev/orig is not present twice in the new data..
Fri, Feb 8, 2:12 PM
douardda accepted D1091: Make the OriginMetadataIndexer call storage.revision_get a single time for all revisions..
Fri, Feb 8, 10:42 AM

Thu, Feb 7

douardda requested changes to D1079: Add checks in the idx_storage that the same content/rev/orig is not present twice in the new data..
Thu, Feb 7, 5:59 PM
douardda accepted D1090: Split OriginIndexer.run into smaller methods..
Thu, Feb 7, 5:45 PM
douardda added a comment to D1090: Split OriginIndexer.run into smaller methods..

One drawback of this function extraction is that you now have to keep an intermediate result (the origins list). Couldn't this be avoided using generators?

That's a very small list (I'd be surprised if it ever gets over 10KB). And starting with D1092, we'll download the whole list at once anyway.

Thu, Feb 7, 5:45 PM
douardda committed rDLDSVNbf78d8b186fe: Fix the DumpMountAndLoadSvnRepository task (authored by douardda).
Fix the DumpMountAndLoadSvnRepository task
Thu, Feb 7, 5:35 PM
douardda closed D1097: Fix the DumpMountAndLoadSvnRepository task.
Thu, Feb 7, 5:35 PM
douardda added a comment to D1080: Restructure origin_metadata_indexer_test to make it easier to add more origin metadata idx tests..

(Actually, after thinking more about this, a class with setUp and tearDown looks like better than my decorator solution.)

Thu, Feb 7, 4:44 PM
douardda accepted D1092: Make OriginIndexer call storage.origin_get a single time for all origins..
Thu, Feb 7, 4:43 PM
Herald added a reviewer for D1097: Fix the DumpMountAndLoadSvnRepository task: Reviewers.
Thu, Feb 7, 4:31 PM