Page MenuHomeSoftware Heritage

olasd (Nicolas Dandrimont)
UserAdministrator

Projects (7)

User Details

User Since
Sep 7 2015, 3:25 PM (272 w, 2 d)
Roles
Administrator

Recent Activity

Yesterday

olasd updated the diff for D3526: Hook up listers implemented with the new pattern to the CLI.

Rebase

Wed, Nov 25, 4:36 PM
olasd updated the diff for D3425: Introduce a simpler base pattern for lister implementations..

Rebase and simplify

Wed, Nov 25, 4:07 PM

Tue, Nov 24

olasd committed rDCORE2c9ea80cab85: Do not mutate swh.core.api.serializers.ENCODERS or DECODERS (authored by olasd).
Do not mutate swh.core.api.serializers.ENCODERS or DECODERS
Tue, Nov 24, 1:24 PM
olasd closed D4572: Do not mutate swh.core.api.serializers.ENCODERS or DECODERS.
Tue, Nov 24, 1:23 PM
olasd created D4572: Do not mutate swh.core.api.serializers.ENCODERS or DECODERS.
Tue, Nov 24, 1:00 PM
olasd committed rDCOREf269e2d3d0f4: Add support for large negative integers in msgpack encoding (authored by olasd).
Add support for large negative integers in msgpack encoding
Tue, Nov 24, 12:47 PM
olasd closed D4568: Add support for large negative integers in msgpack encoding.
Tue, Nov 24, 12:47 PM
olasd added a comment to D4568: Add support for large negative integers in msgpack encoding.

I got questions... for those functions in general.

Any particular reason for having those encoders/decoders hard-coded in the main encoding/decoding functions?
(i know you are here only extending the current behavior to solve the problem at hand but i mean long term)

Or can we envision refactor those within the realm of the extra_encoders/decoders at some point?
And then encode_types becomes the simple for look around calling the encoders.
(ideally something equivalent could happen in the decoding part).

Tue, Nov 24, 12:46 PM
olasd updated the diff for D4568: Add support for large negative integers in msgpack encoding.

Move comment back to the right place

Tue, Nov 24, 12:01 PM
olasd added a comment to D4568: Add support for large negative integers in msgpack encoding.

Why don't you pass signed=True to int.to_bytes/int.from_bytes, and add "one eighth of a byte" to the length (ie. replace if rem: length += 1 by length += 1) when encoding?

Tue, Nov 24, 11:55 AM

Mon, Nov 23

olasd created D4568: Add support for large negative integers in msgpack encoding.
Mon, Nov 23, 10:46 PM
olasd closed T2656: Pause background ingestion until we get more local storage space as Resolved.

Storage has been extended. I've restored the function to its original version (without policy = 'oneshot').

Mon, Nov 23, 7:24 PM · System administration
olasd accepted D4545: templates/stage-python-tests: Set sourceFileResolver to publishCoverage.
Mon, Nov 23, 4:27 PM

Fri, Nov 20

olasd accepted D4384: debian/*: Separate python3-swh.core.db.pytestplugin package from python3-swh.core.
Fri, Nov 20, 6:57 PM
olasd accepted D4543: setup: Separate pytest-postgresql dependency and declare it when needed.
Fri, Nov 20, 2:21 PM
olasd added inline comments to D4543: setup: Separate pytest-postgresql dependency and declare it when needed.
Fri, Nov 20, 2:09 PM
olasd added inline comments to D4543: setup: Separate pytest-postgresql dependency and declare it when needed.
Fri, Nov 20, 2:03 PM
olasd added inline comments to D4384: debian/*: Separate python3-swh.core.db.pytestplugin package from python3-swh.core.
Fri, Nov 20, 11:38 AM
olasd accepted D4539: Add mandatory cloud storage configuration on bare metal storage servers.
Fri, Nov 20, 10:28 AM
olasd accepted D4534: Kafka needs a jre to run.
Fri, Nov 20, 10:14 AM
olasd accepted D4538: tox.ini: fix pytest failure trying to run non-tests script.
Fri, Nov 20, 10:12 AM

Thu, Nov 19

olasd added a comment to T2796: 2020-11-18 Datacenter operations in Rocquencourt.

After a few hit-and-miss reboots, the multipath configuration was set to only enable explicitly listed wwids, but the list of wwids was empty.

Thu, Nov 19, 5:44 PM · System administration
olasd added a comment to D4516: Move archive storage to a new server.

@vsellier I believe that we're good to push this now.

Thu, Nov 19, 5:42 PM
olasd added a comment to D4536: Log error message when that occurs.

Even though that's probably not a problem in that case, in general, please avoid f-strings or manual interpolation in logging statements, but rather use the built-in expansion which uses the extra arguments passed to logger.info. This avoids some (potentially costly) string interpolation when the logger is not enabled.

Thu, Nov 19, 5:27 PM
olasd committed rSPSITE0912a1a7d8ef: profile::mountpoints: don't do a deep merge of options (authored by olasd).
profile::mountpoints: don't do a deep merge of options
Thu, Nov 19, 5:07 PM
olasd committed rSPSITE6383e5c23772: profile::mountpoints: Only handle mountpoints when they're not absent (authored by olasd).
profile::mountpoints: Only handle mountpoints when they're not absent
Thu, Nov 19, 4:21 PM
olasd committed rSPSITEe00340c5a942: Add mountpoints for saam (authored by olasd).
Add mountpoints for saam
Thu, Nov 19, 4:11 PM
olasd closed D4532: Add mountpoints for saam.
Thu, Nov 19, 4:11 PM
olasd committed rSPSITE33b7e5fe52e6: Carry over uffizi local storage/objstorage configs to saam (authored by olasd).
Carry over uffizi local storage/objstorage configs to saam
Thu, Nov 19, 4:11 PM
olasd committed rSPSITE5984acc9a2e2: profile::mountpoints: only create directories if the mountpoint is enabled (authored by olasd).
profile::mountpoints: only create directories if the mountpoint is enabled
Thu, Nov 19, 4:11 PM
olasd closed D4528: Carry over uffizi local storage/objstorage configs to saam.
Thu, Nov 19, 4:11 PM
olasd closed D4531: profile::mountpoints: only create directories if the mountpoint is enabled.
Thu, Nov 19, 4:11 PM
olasd added inline comments to D4531: profile::mountpoints: only create directories if the mountpoint is enabled.
Thu, Nov 19, 4:02 PM
olasd added a comment to T2796: 2020-11-18 Datacenter operations in Rocquencourt.

D4516 by @vsellier moves all storage/objstorage/indexer storage clients from uffizi to saam

Thu, Nov 19, 3:30 PM · System administration
olasd added a comment to D4534: Kafka needs a jre to run.

We use the puppet java module in a bunch of other places, maybe it makes sense to directly import that (which would mean using include ::java)?

Thu, Nov 19, 3:29 PM
olasd added a revision to T2796: 2020-11-18 Datacenter operations in Rocquencourt: D4516: Move archive storage to a new server.
Thu, Nov 19, 3:26 PM · System administration
olasd added a task to D4516: Move archive storage to a new server: T2796: 2020-11-18 Datacenter operations in Rocquencourt.
Thu, Nov 19, 3:26 PM
olasd added a comment to T2796: 2020-11-18 Datacenter operations in Rocquencourt.

D4528 carries the local storage/objstorage configuration on uffizi to saam.
D4531/D4532 adds all the local mountpoints needed for this local config to work.

Thu, Nov 19, 3:25 PM · System administration
olasd added a revision to T2796: 2020-11-18 Datacenter operations in Rocquencourt: D4532: Add mountpoints for saam.
Thu, Nov 19, 3:24 PM · System administration
olasd added a task to D4532: Add mountpoints for saam: T2796: 2020-11-18 Datacenter operations in Rocquencourt.
Thu, Nov 19, 3:24 PM
olasd added a revision to T2796: 2020-11-18 Datacenter operations in Rocquencourt: D4531: profile::mountpoints: only create directories if the mountpoint is enabled.
Thu, Nov 19, 3:24 PM · System administration
olasd added a task to D4531: profile::mountpoints: only create directories if the mountpoint is enabled: T2796: 2020-11-18 Datacenter operations in Rocquencourt.
Thu, Nov 19, 3:24 PM
olasd added a task to D4528: Carry over uffizi local storage/objstorage configs to saam: T2796: 2020-11-18 Datacenter operations in Rocquencourt.
Thu, Nov 19, 3:23 PM
olasd added a revision to T2796: 2020-11-18 Datacenter operations in Rocquencourt: D4528: Carry over uffizi local storage/objstorage configs to saam.
Thu, Nov 19, 3:23 PM · System administration
olasd added a comment to T2796: 2020-11-18 Datacenter operations in Rocquencourt.

There was an issue with the indexer storage package missing the swh.indexer setuptools metadata. I've moved the metadata to the swh.indexer.storage package in rDCIDX3809bb03

Thu, Nov 19, 3:23 PM · System administration
olasd created D4532: Add mountpoints for saam.
Thu, Nov 19, 3:16 PM
olasd created D4531: profile::mountpoints: only create directories if the mountpoint is enabled.
Thu, Nov 19, 3:15 PM
olasd created D4528: Carry over uffizi local storage/objstorage configs to saam.
Thu, Nov 19, 2:49 PM
olasd committed rSPSITEb68a7b0f45e1: Delay zfs-import-cache after multipath setup (authored by olasd).
Delay zfs-import-cache after multipath setup
Thu, Nov 19, 2:37 PM
olasd committed rSPSITE59b245dd72c7: Override the multipath.service unit on bare metal storage servers (authored by olasd).
Override the multipath.service unit on bare metal storage servers
Thu, Nov 19, 2:37 PM
olasd closed D4523: Multipath setup for saam.
Thu, Nov 19, 2:37 PM
olasd added a comment to D4523: Multipath setup for saam.

Shouldn't the multipath configuration itself also be commited?

Thu, Nov 19, 2:37 PM
olasd added a comment to T2796: 2020-11-18 Datacenter operations in Rocquencourt.

On the saam software setup:

Thu, Nov 19, 1:13 PM · System administration
olasd added a revision to T2796: 2020-11-18 Datacenter operations in Rocquencourt: D4523: Multipath setup for saam.
Thu, Nov 19, 1:13 PM · System administration
olasd created D4523: Multipath setup for saam.
Thu, Nov 19, 1:13 PM
olasd updated subscribers of T2796: 2020-11-18 Datacenter operations in Rocquencourt.

The downtime, scheduled for the whole day of 18 November, was posted to status.io on 16 November (https://status.softwareheritage.org/pages/maintenance/578e5eddcdc0cc7951000520/5fb2ae1fbf590a04c7fdffb0)

Thu, Nov 19, 12:23 PM · System administration
olasd changed the status of T2796: 2020-11-18 Datacenter operations in Rocquencourt from Open to Work in Progress.
Thu, Nov 19, 11:16 AM · System administration
olasd accepted D4516: Move archive storage to a new server.

Looks fine to me except for the vagrant bit.

Thu, Nov 19, 9:45 AM

Tue, Nov 17

olasd closed T1829: Find a way to properly open the kafka brokers to the internet as Resolved.

The new cluster in rocquencourt is using the built-in Kafka ACLs now (9993a81ffc7a1c8bd519b33ae63ac1145105f624).

Tue, Nov 17, 6:33 PM · System administration, Mirror
olasd committed rSPSITEac3332f5b22c: Set broker username and password in JAAS config (authored by olasd).
Set broker username and password in JAAS config
Tue, Nov 17, 4:48 PM
olasd committed rSPSITEdd85c51cbf49: Flatten JAAS config into the main server.properties (authored by olasd).
Flatten JAAS config into the main server.properties
Tue, Nov 17, 4:48 PM
olasd committed rSPSITE9993a81ffc7a: Enable kafka ACLs (authored by olasd).
Enable kafka ACLs
Tue, Nov 17, 4:48 PM
olasd committed rSPSITEe1a942059f00: Add superusers to the kafka cluster setup (authored by olasd).
Add superusers to the kafka cluster setup
Tue, Nov 17, 4:48 PM
olasd committed rSPSITEb517cf370e3d: Allow overriding kafka configs per cluster and per broker (authored by olasd).
Allow overriding kafka configs per cluster and per broker
Tue, Nov 17, 4:48 PM
olasd closed T2789: Upgrade kafka to 2.6.0 as Resolved.

Bumped kafka[1-4] too now. The upgrade is complete on all hosts.

Tue, Nov 17, 2:57 PM · System administration
olasd added a comment to T2789: Upgrade kafka to 2.6.0.

Restarted esnode[1-3] again after pushing 87909f184325.

Tue, Nov 17, 2:53 PM · System administration
olasd committed rSPSITE87909f184325: Finalize kafka upgrade to 2.6 (authored by olasd).
Finalize kafka upgrade to 2.6
Tue, Nov 17, 2:47 PM
olasd added a comment to T2789: Upgrade kafka to 2.6.0.

Did a rolling restart of the cluster on kafka[1-4] (enabling puppet, and running the puppet agent manually on each host in turn).

Tue, Nov 17, 2:45 PM · System administration
olasd added a comment to T2789: Upgrade kafka to 2.6.0.

Did a rolling restart of the cluster on esnode[1-3] (enabling puppet, and running the puppet agent manually on each host in turn).

Tue, Nov 17, 2:40 PM · System administration
olasd changed the status of T2789: Upgrade kafka to 2.6.0 from Open to Work in Progress.

Disabled puppet on kafka[1-4] esnode[1-3].

Tue, Nov 17, 2:35 PM · System administration
olasd committed rSPSITE87e806867373: Move kafka settings to a separate hiera file (authored by olasd).
Move kafka settings to a separate hiera file
Tue, Nov 17, 2:35 PM
olasd committed rSPSITEf9682a596086: Upgrade kafka to 2.6 (authored by olasd).
Upgrade kafka to 2.6
Tue, Nov 17, 2:35 PM
olasd triaged T2789: Upgrade kafka to 2.6.0 as High priority.
Tue, Nov 17, 2:33 PM · System administration
olasd committed rSPPRIVC0135f713ce47: Refresh from private data (authored by olasd).
Refresh from private data
Tue, Nov 17, 12:47 PM
olasd committed rDMODa3b6a6448011: Drop backwards-compatibility support for RawExtrinsicMetadata.id (authored by olasd).
Drop backwards-compatibility support for RawExtrinsicMetadata.id
Tue, Nov 17, 10:12 AM
olasd closed D4487: Drop backwards-compatibility support for RawExtrinsicMetadata.id.
Tue, Nov 17, 10:12 AM
olasd committed rSPSITE417cc2a0d67d: Don't try to zip gc logfiles which get rotated differently (authored by olasd).
Don't try to zip gc logfiles which get rotated differently
Tue, Nov 17, 9:57 AM
olasd closed D4488: Don't try to zip kafka gc logfiles which get rotated differently.
Tue, Nov 17, 9:57 AM
olasd committed rSPSITE10ce17162ae3: Reload the icinga2 service when a config file gets dropped by recursion (authored by olasd).
Reload the icinga2 service when a config file gets dropped by recursion
Tue, Nov 17, 9:57 AM
olasd closed D4485: Reload the icinga2 service when a config file gets dropped by recursion.
Tue, Nov 17, 9:57 AM
olasd updated the diff for D4488: Don't try to zip kafka gc logfiles which get rotated differently.

Rebase

Tue, Nov 17, 9:57 AM
olasd updated the diff for D4485: Reload the icinga2 service when a config file gets dropped by recursion.

Rebase

Tue, Nov 17, 9:57 AM

Mon, Nov 16

olasd created D4488: Don't try to zip kafka gc logfiles which get rotated differently.
Mon, Nov 16, 7:40 PM
olasd created D4487: Drop backwards-compatibility support for RawExtrinsicMetadata.id.
Mon, Nov 16, 7:32 PM
olasd created D4485: Reload the icinga2 service when a config file gets dropped by recursion.
Mon, Nov 16, 6:36 PM
olasd updated the task description for T2783: Decommission orsay.
Mon, Nov 16, 6:36 PM · System administration
olasd updated the task description for T2783: Decommission orsay.
Mon, Nov 16, 6:21 PM · System administration
olasd updated the task description for T2783: Decommission orsay.
Mon, Nov 16, 6:20 PM · System administration
olasd changed the status of T2783: Decommission orsay from Open to Work in Progress.
Mon, Nov 16, 6:20 PM · System administration
olasd closed T2628: Recycle ceph-osd2 as objstorage/storage backend for staging, a subtask of T2502: Migrate VM storage to ceph, as Resolved.
Mon, Nov 16, 6:18 PM · System administration
olasd closed T2628: Recycle ceph-osd2 as objstorage/storage backend for staging as Resolved.
Mon, Nov 16, 6:18 PM · Staging environment, System administration
olasd closed T2627: Recycle ceph-osd1 as staging database host as Resolved.
Mon, Nov 16, 6:17 PM · Staging environment, System administration
olasd closed T2627: Recycle ceph-osd1 as staging database host, a subtask of T2502: Migrate VM storage to ceph, as Resolved.
Mon, Nov 16, 6:17 PM · System administration
olasd closed T2502: Migrate VM storage to ceph as Resolved.

All done now that orsay is being decommissionned.

Mon, Nov 16, 6:16 PM · System administration
olasd closed T2502: Migrate VM storage to ceph, a subtask of T2501: Proxmox reliability improvements (Summer 2020), as Resolved.
Mon, Nov 16, 6:16 PM · System administration
olasd committed rDCIDX440961d06c44: Drop all deprecated uses of `args` in component factories (authored by olasd).
Drop all deprecated uses of `args` in component factories
Mon, Nov 16, 3:19 PM
olasd closed D4481: Drop all deprecated uses of `args` in component factories.
Mon, Nov 16, 3:19 PM
olasd created D4481: Drop all deprecated uses of `args` in component factories.
Mon, Nov 16, 3:11 PM
olasd abandoned D4480: Drop vcversioner from requirements.

Messed up my arc diff --create.

Mon, Nov 16, 3:11 PM
olasd committed rDCIDX5de2b6e40c38: Drop vcversioner from requirements (authored by olasd).
Drop vcversioner from requirements
Mon, Nov 16, 3:06 PM