Page MenuHomeSoftware Heritage

olasd (Nicolas Dandrimont)
UserAdministrator

Projects (7)

User Details

User Since
Sep 7 2015, 3:25 PM (263 w, 2 h)
Roles
Administrator

Recent Activity

Today

olasd triaged T2619: Make the front page "archive size" graphs consistent with one another as Wishlist priority.
Mon, Sep 21, 5:28 PM · Web app
olasd added a comment to D3981: Support different database flavors in the SQL scripts.

Also this description of flavors should be documented somewhere (I mean somewhere else than the ci message)

Mon, Sep 21, 11:57 AM
olasd added a comment to D3981: Support different database flavors in the SQL scripts.

@douardda deduplication means it's not read_replica, so either mirror or default.

We want deduplication for default (it's the one written to by loaders), and also for mirror so they filter out duplicates from the journal.

Mon, Sep 21, 11:54 AM
olasd committed rDCOREe5312bdaa865: Reimplement swh_db_version with psycopg2 (authored by olasd).
Reimplement swh_db_version with psycopg2
Mon, Sep 21, 10:01 AM
olasd closed D3992: Reimplement swh_db_version with psycopg2.
Mon, Sep 21, 10:01 AM
olasd committed rDCORE30be93846340: Pin black to the same version as the pre-commit hook (authored by olasd).
Pin black to the same version as the pre-commit hook
Mon, Sep 21, 10:01 AM
olasd closed D3991: Pin black to the same version as the pre-commit hook.
Mon, Sep 21, 10:01 AM
olasd committed rDCOREb63d7f432cae: Fix license statements on db_utils.py (authored by olasd).
Fix license statements on db_utils.py
Mon, Sep 21, 10:01 AM
olasd closed D3987: Fix license statements on db_utils.py.
Mon, Sep 21, 10:01 AM

Fri, Sep 18

olasd added a comment to D3981: Support different database flavors in the SQL scripts.

Is this a WIP? If yes, what are the -- ? comments?

And as discussed on IRC, we might want to use unique indices in read replicas too, so we have less differences between flavors.

Fri, Sep 18, 6:12 PM
olasd added inline comments to D3989: adapt configuration for the vagrant environment.
Fri, Sep 18, 5:49 PM
olasd created D3997: Add support for database flavors to `swh db-init`.
Fri, Sep 18, 5:46 PM
olasd created D3992: Reimplement swh_db_version with psycopg2.
Fri, Sep 18, 3:58 PM
olasd created D3991: Pin black to the same version as the pre-commit hook.
Fri, Sep 18, 3:45 PM
olasd accepted D3988: loaders: Move the proxy storage filter after the buffer proxy.
Fri, Sep 18, 2:51 PM
olasd accepted D3986: loaders: Move the proxy storage filter after the buffer proxy.

Yeah, this definitely needs to happen before the new swh.loader.core is deployed.

Fri, Sep 18, 2:50 PM
olasd accepted D3985: tests: Don't check the number of created 'person' objects.
Fri, Sep 18, 2:49 PM
olasd accepted D3984: tests: Don't check the number of created 'person' objects.
Fri, Sep 18, 2:49 PM
olasd created D3987: Fix license statements on db_utils.py.
Fri, Sep 18, 2:41 PM

Thu, Sep 17

olasd added a comment to T2604: Handle multiple "database profiles" in the swh-storage (/...) SQL configurations.

Btw, sqitch uses the native scripting for the database engine (in case of postgres, psql scripts), so if we ever end up using it, we can keep this approach.

Thu, Sep 17, 8:34 PM · Storage manager
olasd added a comment to T2604: Handle multiple "database profiles" in the swh-storage (/...) SQL configurations.

So, I've first attempted something along the lines of https://www.depesz.com/2008/06/18/conditional-ddl/, which uses a function to execute DDL commands stored in a string.

Thu, Sep 17, 8:26 PM · Storage manager
olasd updated the diff for D3981: Support different database flavors in the SQL scripts.

remove spurious flag on an \else block

Thu, Sep 17, 8:08 PM
olasd added a revision to T2604: Handle multiple "database profiles" in the swh-storage (/...) SQL configurations: D3981: Support different database flavors in the SQL scripts.
Thu, Sep 17, 8:05 PM · Storage manager
olasd created D3981: Support different database flavors in the SQL scripts.
Thu, Sep 17, 8:05 PM
olasd accepted D3976: loader: Stop materializing full lists of objects to be stored..

Begrudgingly giving this an Accept.

Thu, Sep 17, 4:00 PM
olasd added a comment to T2610: Add isort pre-commit hook and configuration to all repos.

+1 to the idea (and to the specific config as I have already bikeshed it)

Thu, Sep 17, 3:35 PM · Development environment
olasd added a comment to D3972: pre-commit: Add isort hook and configuration.
In D3972#98074, @olasd wrote:

I'm okay with this change. But I think other team members should have a stab at the color of the bikeshed too.

Ack, I will wait before landing this then.

Thu, Sep 17, 2:55 PM
olasd accepted D3978: tests: Don't check the number of created 'person' objects..
Thu, Sep 17, 2:54 PM · Git loader
olasd accepted D3977: tests.get_stats: Don't return a 'person' count..
Thu, Sep 17, 2:54 PM
olasd added a comment to D3976: loader: Stop materializing full lists of objects to be stored..

I really hate this, but save from making *_add able to take generators as arguments again, this is probably the most decent way out.

Thu, Sep 17, 2:41 PM
olasd accepted D3975: Make origin_add() handle multiple occurences of an origin properly.

It took me a bit to understand what the sorteds were achieving (dedup the list while keeping it ordered), so I wouldn't mind if you turned them into an explicit loop, or at least if you added a comment.

Thu, Sep 17, 2:15 PM
olasd added a comment to D3972: pre-commit: Add isort hook and configuration.

I'm okay with this change. But I think other team members should have a stab at the color of the bikeshed too.

Thu, Sep 17, 2:07 PM
olasd added a comment to T2373: git loader OOM when loading huge repository.

So content_missing call explodes mid-air client side (`"POST /content/missing
HTTP/1.1" 200 9475383` so client received the data).

It so happens that the content_missing api is taking an unlimited amount of
bytes ids as input [1] And then "tries" to stream to the client the results
(rpc layer in the middle makes that moot).

Thu, Sep 17, 2:03 PM · Git loader
olasd added a comment to D3967: Vagrant environment to test puppet locally.

I think this should be moved to the sysadm-provisioning repository [1] instead.

[1] https://forge.softwareheritage.org/source/swh-sysadmin-provisioning/

Thu, Sep 17, 12:05 PM
olasd added a comment to D3972: pre-commit: Add isort hook and configuration.

Definitely yes.

Thu, Sep 17, 12:00 PM
olasd requested changes to D3961: Manage filebeat configuration via puppet.
Thu, Sep 17, 11:37 AM
olasd committed rDCORE6b983c28a3be: Make swh db init and swh db-init behavior much closer to one another (authored by olasd).
Make swh db init and swh db-init behavior much closer to one another
Thu, Sep 17, 11:32 AM
olasd committed rDCORE327f98d18602: Ignore stderr in db_testing.swh_db_version instead of printing it out (authored by olasd).
Ignore stderr in db_testing.swh_db_version instead of printing it out
Thu, Sep 17, 11:32 AM
olasd closed D3970: Make swh db init and swh db-init behavior much closer to one another.
Thu, Sep 17, 11:32 AM
olasd added inline comments to D3970: Make swh db init and swh db-init behavior much closer to one another.
Thu, Sep 17, 11:32 AM
olasd added a project to T1410: Kill implicit configuration : new configuration scheme: Core & foundations.
Thu, Sep 17, 10:53 AM · Core & foundations

Wed, Sep 16

olasd updated the diff for D3970: Make swh db init and swh db-init behavior much closer to one another.

Fix flake8 issue that snuck past pre-commit thanks to rebase shenanigans

Wed, Sep 16, 7:24 PM
olasd created D3970: Make swh db init and swh db-init behavior much closer to one another.
Wed, Sep 16, 7:21 PM
olasd added a comment to D3961: Manage filebeat configuration via puppet.

Thanks! This indeed looks much less fragile than the concat version.

Wed, Sep 16, 5:56 PM
olasd triaged T2604: Handle multiple "database profiles" in the swh-storage (/...) SQL configurations as Normal priority.
Wed, Sep 16, 5:32 PM · Storage manager
olasd requested changes to D3961: Manage filebeat configuration via puppet.

I don't like using concat and ERB templates to generate yaml files at all:

Wed, Sep 16, 2:49 PM
olasd accepted D3963: data/defaults: Drop global.ini file.

Bye bye .ini.

Wed, Sep 16, 2:45 PM
olasd added a subtask for T2597: Switch indexer journal client to new kafka cluster: T2603: Configuration mismatch between swh.indexer.journal_client and the configuration declared in puppet.
Wed, Sep 16, 2:07 PM · System administration
olasd added a parent task for T2603: Configuration mismatch between swh.indexer.journal_client and the configuration declared in puppet: T2597: Switch indexer journal client to new kafka cluster.
Wed, Sep 16, 2:07 PM · Puppet recipes, Indexer
olasd triaged T2603: Configuration mismatch between swh.indexer.journal_client and the configuration declared in puppet as Normal priority.
Wed, Sep 16, 2:06 PM · Puppet recipes, Indexer
olasd added a comment to D3963: data/defaults: Drop global.ini file.
In D3963#97665, @olasd wrote:

I agree with the switch to yaml, but I don't think we should have plain text yaml files in the hiera variables.

We should switch the swh::global_conf hiera variable to be a yaml dict, and the way the file is generated by the profile::swh to inline this yaml (with a "warning" header).

Or we could drop the file altogether, as it's not being used at all now.

Wed, Sep 16, 1:54 PM
olasd requested changes to D3963: data/defaults: Drop global.ini file.

I agree with the switch to yaml, but I don't think we should have plain text yaml files in the hiera variables.

Wed, Sep 16, 1:53 PM

Tue, Sep 15

olasd closed T2596: Switch main journal producer to new kafka cluster, a subtask of T2597: Switch indexer journal client to new kafka cluster, as Resolved.
Tue, Sep 15, 7:58 PM · System administration
olasd closed T2596: Switch main journal producer to new kafka cluster as Resolved.

This has been pushed to production now.

Tue, Sep 15, 7:58 PM · System administration
olasd closed T2596: Switch main journal producer to new kafka cluster, a subtask of T2520: Setup dedicated kafka cluster on new rocquencourt hardware, as Resolved.
Tue, Sep 15, 7:58 PM · System administration
olasd committed rSPSITE90b719f2c486: Use extra configs instead of raw mtu for manual interfaces (authored by olasd).
Use extra configs instead of raw mtu for manual interfaces
Tue, Sep 15, 6:36 PM
olasd committed rSPSITEdb2ebfd012b5: Anonymize messages sent to the production kafka cluster (authored by olasd).
Anonymize messages sent to the production kafka cluster
Tue, Sep 15, 6:07 PM
olasd closed D3951: Switch production over to the new kafka cluster.
Tue, Sep 15, 6:07 PM
olasd committed rSPSITEab6794381e86: Use kafka[1-4].internal.softwareheritage.org as default kafka cluster (authored by olasd).
Use kafka[1-4].internal.softwareheritage.org as default kafka cluster
Tue, Sep 15, 6:07 PM
olasd closed D3950: Allow overriding the public kafka broker naming scheme.
Tue, Sep 15, 6:07 PM
olasd committed rSPSITEce8e9942e2cd: Make branly a hypervisor host instead of the default (authored by olasd).
Make branly a hypervisor host instead of the default
Tue, Sep 15, 6:07 PM
olasd committed rSPSITE5cc8fb43d643: Allow overriding the public kafka broker naming scheme (authored by olasd).
Allow overriding the public kafka broker naming scheme
Tue, Sep 15, 6:07 PM
olasd closed D3949: Make branly a hypervisor host instead of the default.
Tue, Sep 15, 6:07 PM
olasd committed rSPSITE75a480e71f57: Configure networking on kafka hosts using the `network` profile (authored by olasd).
Configure networking on kafka hosts using the `network` profile
Tue, Sep 15, 6:07 PM
olasd closed D3948: Configure networking on kafka hosts using the `network` profile.
Tue, Sep 15, 6:07 PM
olasd committed rSPSITE366d5965fd1c: Enable the network profile when defined (authored by olasd).
Enable the network profile when defined
Tue, Sep 15, 5:45 PM
olasd closed D3947: Enable the network profile when defined.
Tue, Sep 15, 5:45 PM
olasd committed rSPSITEb0899ba477ae: Improve networks profile module documentation (authored by olasd).
Improve networks profile module documentation
Tue, Sep 15, 5:45 PM
olasd committed rSPSITE1c5a6b53e0c2: network: Add support for extra settings (authored by olasd).
network: Add support for extra settings
Tue, Sep 15, 5:45 PM
olasd committed rSPSITEb707f66950b2: network: Add support for mtu value (authored by olasd).
network: Add support for mtu value
Tue, Sep 15, 5:45 PM
olasd committed rSPSITEb9bdc1c242a3: Enable other methods (than statics) of setting up the network interfaces (authored by olasd).
Enable other methods (than statics) of setting up the network interfaces
Tue, Sep 15, 5:45 PM
olasd committed rSPSITE058c20d7ee43: Switch the networks dict to be keyed by the interface name (authored by olasd).
Switch the networks dict to be keyed by the interface name
Tue, Sep 15, 5:45 PM
olasd committed rSPSITE087cd01c91e9: Make explicit routes on the private network configurable properly (authored by olasd).
Make explicit routes on the private network configurable properly
Tue, Sep 15, 5:45 PM
olasd closed D3946: Refactor profile::network to be more extensible.
Tue, Sep 15, 5:45 PM
olasd updated the diff for D3951: Switch production over to the new kafka cluster.

Rebase

Tue, Sep 15, 5:41 PM
olasd updated the diff for D3950: Allow overriding the public kafka broker naming scheme.

Rebase

Tue, Sep 15, 5:41 PM
olasd updated the diff for D3949: Make branly a hypervisor host instead of the default.

Rebase

Tue, Sep 15, 5:41 PM
olasd updated the diff for D3948: Configure networking on kafka hosts using the `network` profile.

Rebase

Tue, Sep 15, 5:41 PM
olasd updated the diff for D3947: Enable the network profile when defined.

Rebase

Tue, Sep 15, 5:41 PM
olasd updated the diff for D3946: Refactor profile::network to be more extensible.
  • Improve networks profile module documentation
Tue, Sep 15, 5:41 PM
olasd added inline comments to D3946: Refactor profile::network to be more extensible.
Tue, Sep 15, 2:11 PM
olasd created P767 Command-Line Input.
Tue, Sep 15, 12:52 PM
olasd added a revision to T2596: Switch main journal producer to new kafka cluster: D3951: Switch production over to the new kafka cluster.
Tue, Sep 15, 11:39 AM · System administration
olasd added a revision to T2597: Switch indexer journal client to new kafka cluster: D3951: Switch production over to the new kafka cluster.
Tue, Sep 15, 11:39 AM · System administration
olasd created D3951: Switch production over to the new kafka cluster.
Tue, Sep 15, 11:39 AM
olasd added a comment to T2520: Setup dedicated kafka cluster on new rocquencourt hardware.

After doing the basic kafka config, I've done the following to create the topics:

Tue, Sep 15, 11:35 AM · System administration
olasd added a parent task for T2596: Switch main journal producer to new kafka cluster: T2597: Switch indexer journal client to new kafka cluster.
Tue, Sep 15, 11:32 AM · System administration
olasd added a subtask for T2597: Switch indexer journal client to new kafka cluster: T2596: Switch main journal producer to new kafka cluster.
Tue, Sep 15, 11:32 AM · System administration
olasd triaged T2597: Switch indexer journal client to new kafka cluster as High priority.
Tue, Sep 15, 11:32 AM · System administration
olasd triaged T2596: Switch main journal producer to new kafka cluster as High priority.
Tue, Sep 15, 11:31 AM · System administration
olasd added a revision to T2593: Public network setup for new kafka hosts: D3950: Allow overriding the public kafka broker naming scheme.
Tue, Sep 15, 11:28 AM · System administration
olasd created D3950: Allow overriding the public kafka broker naming scheme.
Tue, Sep 15, 11:28 AM
olasd created D3949: Make branly a hypervisor host instead of the default.
Tue, Sep 15, 11:25 AM
olasd added a revision to T2593: Public network setup for new kafka hosts: D3948: Configure networking on kafka hosts using the `network` profile.
Tue, Sep 15, 11:23 AM · System administration
olasd created D3948: Configure networking on kafka hosts using the `network` profile.
Tue, Sep 15, 11:23 AM
olasd triaged T2593: Public network setup for new kafka hosts as High priority.
Tue, Sep 15, 11:22 AM · System administration
olasd created D3947: Enable the network profile when defined.
Tue, Sep 15, 11:19 AM
olasd created D3946: Refactor profile::network to be more extensible.
Tue, Sep 15, 11:18 AM

Mon, Sep 14

olasd closed T2522: Basic OS setup for new kafka cluster hardware as Resolved.
Mon, Sep 14, 3:37 PM · System administration
olasd closed T2522: Basic OS setup for new kafka cluster hardware, a subtask of T2520: Setup dedicated kafka cluster on new rocquencourt hardware, as Resolved.
Mon, Sep 14, 3:37 PM · System administration