Page MenuHomeSoftware Heritage

olasd (Nicolas Dandrimont)
UserAdministrator

Projects (7)

User Details

User Since
Sep 7 2015, 3:25 PM (214 w, 1 d)
Roles
Administrator

Recent Activity

Yesterday

olasd committed rSPSITE06557c3f16ec: Replace sshkey with concat in phabricator profile (authored by olasd).
Replace sshkey with concat in phabricator profile
Tue, Oct 15, 7:04 PM
olasd committed rSPGUNIbb3ea9af8f37: Add missing reference to the azure logger in the main config (authored by olasd).
Add missing reference to the azure logger in the main config
Tue, Oct 15, 7:00 PM
olasd committed rSPGUNI1f1cd26a291f: Only ever log to the journal (authored by olasd).
Only ever log to the journal
Tue, Oct 15, 7:00 PM
olasd committed rSPGUNIad79395206c4: Silence azure.storage.common.storageclient noise (authored by olasd).
Silence azure.storage.common.storageclient noise
Tue, Oct 15, 6:50 PM
olasd committed rSPSITEeb94989697fb: Use concat to handle ssh_known_hosts (authored by olasd).
Use concat to handle ssh_known_hosts
Tue, Oct 15, 12:03 PM

Mon, Oct 14

olasd committed rSPSITEc69ad2446524: Set DJANGO_SETTINGS_MODULE on webapp gunicorn instance (authored by olasd).
Set DJANGO_SETTINGS_MODULE on webapp gunicorn instance
Mon, Oct 14, 7:59 PM
olasd closed T1467: Slow network transfers from beaubourg as Resolved.
Mon, Oct 14, 7:06 PM · System administration
olasd committed rSPSITE86011089aa8b: Make ceph_mgr_modules even more robusterer (authored by olasd).
Make ceph_mgr_modules even more robusterer
Mon, Oct 14, 6:55 PM
olasd committed rSPSITEf02171ac60a4: Add proxmox VE 5.x apt key (authored by olasd).
Add proxmox VE 5.x apt key
Mon, Oct 14, 6:52 PM
olasd committed rSPSITE756549a25954: Make ceph_mgr_modules fact more robust (authored by olasd).
Make ceph_mgr_modules fact more robust
Mon, Oct 14, 6:35 PM
olasd committed rSPSITE2fd23f99cfb5: Use proxmox repository for ceph (authored by olasd).
Use proxmox repository for ceph
Mon, Oct 14, 6:30 PM
olasd committed rSPSITE6cedc879c715: Postgresql is now up to 11 (authored by olasd).
Postgresql is now up to 11
Mon, Oct 14, 5:07 PM
olasd accepted D2133: pytest_plugin: Add support for http request (debian lister related).
Mon, Oct 14, 4:52 PM
olasd added a comment to D2133: pytest_plugin: Add support for http request (debian lister related).

(And maybe add support for https_<domain_name> for symmetry)

I'm for symmetry but I'm a bit worried about the amount of work on my pile of diffs that this implies...
(for the https part, for http only debian code is impacted so fine)
I'd rather change that after merging the package-loader branch if you don't mind.

Mon, Oct 14, 4:15 PM
olasd committed rSPSITE8d1fc8f1af1e: Install megacli from $::lsbdistcodename, not stretch (authored by olasd).
Install megacli from $::lsbdistcodename, not stretch
Mon, Oct 14, 4:05 PM
olasd requested changes to D2133: pytest_plugin: Add support for http request (debian lister related).

I'm a bit worried that this pulls http and https from the same set of files without distinction; This looks like a good catalyst for subtle single-character bugs.

Mon, Oct 14, 4:02 PM
olasd added inline comments to D2131: [POC] Add keycloak service and sample SoftwareHeritage realm.
Mon, Oct 14, 3:56 PM
olasd committed rSPSITEe3b614731010: getty isn't a kafka broker any more (authored by olasd).
getty isn't a kafka broker any more
Mon, Oct 14, 2:39 PM
olasd committed rSPSITEb4502744b987: Speed up startup of kafka by setting num.recovery.threads.per.data.dir (authored by olasd).
Speed up startup of kafka by setting num.recovery.threads.per.data.dir
Mon, Oct 14, 2:13 PM
olasd committed rSPSITE48064894316a: Let systemd know that kafka exiting with 143 is successful (thanks Java) (authored by olasd).
Let systemd know that kafka exiting with 143 is successful (thanks Java)
Mon, Oct 14, 1:53 PM
olasd added a comment to D2078: Add support for symbolic references.
In D2078#48207, @zack wrote:

Looks good, but I'm puzzled by some testing results.
The logic of extracting the branches for snapshot computation has trickled down to swh identify. (BTW, if you're up for some refactoring, this diff might be a chance to avoid the duplication, but it's certainly not a blocker here.)
So I've tried to just reuse your change in swh.model.cli too, here's the minimal diff: D2078.
Then I've dumped the branch list just before it's hashed (with P542 in swh.model.identifiers) and what I get is this:

alias HEAD 726566732f68656164732f6d6173746572
revision refs/heads/master 9768d0b576dbaaecd80abedad6dfd0d72f1476da
revision refs/remotes/origin/master c3c588713233609f5bbbb2d9e7f3fb4a660f3f72

where HEAD is an alias, but it's pointing to an object ID (this is on the sample git repo used for tests on swh-load-git, but I obtain similar results on any other repo I've tried).
What am I missing here?

Mon, Oct 14, 12:35 PM
olasd committed rSPSITEb69971caa7d2: Avoid killing a kafka broker while it's winding down (authored by olasd).
Avoid killing a kafka broker while it's winding down
Mon, Oct 14, 11:41 AM

Fri, Oct 11

olasd committed rSPSITEcb4338516b17: Update scheduler cli entry points (authored by olasd).
Update scheduler cli entry points
Fri, Oct 11, 6:22 PM
olasd committed rSPRE254de25cf5a8: Add kafka nodes (authored by olasd).
Add kafka nodes
Fri, Oct 11, 3:06 PM
olasd committed rSPREe52476df57cf: Force vault key data to use ardumont to match terraform state (authored by olasd).
Force vault key data to use ardumont to match terraform state
Fri, Oct 11, 3:06 PM
olasd committed rSPRE456f6a2785ff: Update injection machine creation script (authored by olasd).
Update injection machine creation script
Fri, Oct 11, 3:06 PM
olasd committed rSPRE2557dc4c53c2: Port to terraform 0.12 (authored by olasd).
Port to terraform 0.12
Fri, Oct 11, 3:06 PM

Tue, Oct 8

olasd accepted D2082: Add pytest fixture to allow mocking requests with data file.
Tue, Oct 8, 7:54 PM
olasd added a comment to D2082: Add pytest fixture to allow mocking requests with data file.
In D2082#48297, @olasd wrote:

Finally, you should make sure that the rationale for adding a new toplevel module for the pytest plugin is really clear (the paste or the reference to pytest/2042 really aren't).
From my tests and reading documentation further (pytest, tox), the issue really is a limitation of the way we're using the tox + pytest combination: we're running the tests on the checked out version of the module, instead of the version that's installed by tox (which is what pytest expects). If we adapt tox.ini to run tests on the installed version of the module (by running pytest [...] {envsitepackagesdir}/swh/core/..., then we can ship the pytest plugin wherever we want without triggering the dreaded ImportMismatchError.

Tue, Oct 8, 2:04 PM
olasd requested changes to D2082: Add pytest fixture to allow mocking requests with data file.

While I understand what the fixture does, its documentation is lacking (you're only really documenting the inner callback, that users won't call into, not the fixture itself). I guess most of the documentation of the callback should move to the fixture itself?

Tue, Oct 8, 1:57 PM
olasd committed rDCIDXea606e0e93a6: Avoid spamming logs with processed %d messages every message (authored by olasd).
Avoid spamming logs with processed %d messages every message
Tue, Oct 8, 1:26 PM
olasd closed D2079: Avoid spamming logs with processed %d messages every message.
Tue, Oct 8, 1:26 PM

Mon, Oct 7

olasd added a comment to T885: Vault: use objstorage streaming to store and fetch bundles.

Pluggable compression has been implemented for all objstorage backends, which means we could

  • store the (compressed) bundles in an uncompressed objstorage on azure
  • when a user requests the bundle
Mon, Oct 7, 7:06 PM · Vault
olasd created D2079: Avoid spamming logs with processed %d messages every message.
Mon, Oct 7, 6:48 PM
olasd committed rSPSITEe567862b64e4: Disable /etc/resolv.conf modification by isc-dhcp-client (authored by olasd).
Disable /etc/resolv.conf modification by isc-dhcp-client
Mon, Oct 7, 5:12 PM
olasd committed rDCIDX67e6f68fde56: Force pg_ctl path (authored by olasd).
Force pg_ctl path
Mon, Oct 7, 4:44 PM
olasd committed rSPSITEdad1a05775ca: move storage0.euwest.azure to base_storage (authored by olasd).
move storage0.euwest.azure to base_storage
Mon, Oct 7, 3:35 PM
olasd committed rSPSITEed7a09cb7ada: Don't install objstorage.cloud on all azure machines (authored by olasd).
Don't install objstorage.cloud on all azure machines
Mon, Oct 7, 2:49 PM
olasd committed rSPSITE1e0e2c141219: Add xml extension for mediawiki (authored by olasd).
Add xml extension for mediawiki
Mon, Oct 7, 2:32 PM
olasd committed rSPSITEc9c8eb55626a: Add mbstring extension for phabricator (authored by olasd).
Add mbstring extension for phabricator
Mon, Oct 7, 2:30 PM
olasd committed rSPMW93b93a2689ab: Remove removed skins (authored by olasd).
Remove removed skins
Mon, Oct 7, 2:26 PM
olasd committed rSPSITE2619e7704594: Buster has PHP 7.3 (authored by olasd).
Buster has PHP 7.3
Mon, Oct 7, 2:19 PM
olasd committed rSPSITE5d6dd2fdca9a: Move php::version to the proper defaults file (authored by olasd).
Move php::version to the proper defaults file
Mon, Oct 7, 2:13 PM
olasd created D2078: Add support for symbolic references.
Mon, Oct 7, 11:35 AM
olasd committed rDDOC3fb36ea3045a: Pass the JAVA_HOME environment variable through tox (authored by olasd).
Pass the JAVA_HOME environment variable through tox
Mon, Oct 7, 10:59 AM

Fri, Oct 4

olasd committed rSPSITEd2410180c5a9: varnish: set vcl_reload_cmd for buster and up (authored by olasd).
varnish: set vcl_reload_cmd for buster and up
Fri, Oct 4, 6:38 PM
olasd committed rSENVf4c7a8adc94d: Update octocatalog-diff facts (authored by olasd).
Update octocatalog-diff facts
Fri, Oct 4, 6:37 PM
olasd committed rSPSITE08d50ebbc924: Update varnish to 5.1.0 (authored by olasd).
Update varnish to 5.1.0
Fri, Oct 4, 6:14 PM
olasd committed rSPVARNfdf3062e6123: Move static options to the end of ExecStart (authored by olasd).
Move static options to the end of ExecStart
Fri, Oct 4, 6:12 PM
olasd committed rSPSITEce17e8e5fdac: Force the use of the Software Heritage journalbeat packages (authored by olasd).
Force the use of the Software Heritage journalbeat packages
Fri, Oct 4, 5:47 PM
olasd committed rSPSITE4f8761144466: Remove UnsetEnvironment, which is now applied before systemd builds its command… (authored by olasd).
Remove UnsetEnvironment, which is now applied before systemd builds its command…
Fri, Oct 4, 5:11 PM
olasd accepted D2073: jobs/swh-packages: add swh-search package.
Fri, Oct 4, 4:51 PM
olasd committed rSPSITEfe1741157e7d: Update java module to 5.0.1 (authored by olasd).
Update java module to 5.0.1
Fri, Oct 4, 4:47 PM
olasd committed rSPSITE52f1bf22c119: Don't map buster to sid for the Software Heritage repository (authored by olasd).
Don't map buster to sid for the Software Heritage repository
Fri, Oct 4, 4:35 PM
olasd accepted D2071: Add Elasticsearch to the base docker image..
Fri, Oct 4, 4:21 PM
olasd committed rSPSITEa44d0a4c6ab7: uffizi: add /srv/softwareheritage/objects-xfs to backup excludes (authored by olasd).
uffizi: add /srv/softwareheritage/objects-xfs to backup excludes
Fri, Oct 4, 10:52 AM

Thu, Oct 3

olasd added a comment to T1958: Performance tuning of zfs infrastructure.

The aforementioned cards have been received, installed and zfs has been setup to use them.

Thu, Oct 3, 6:00 PM · System administration
olasd committed rSPSITEc9952576e0b3: uffizi: second try's the charm (or something) (authored by olasd).
uffizi: second try's the charm (or something)
Thu, Oct 3, 5:38 PM
olasd committed rSPSITE907cc5919daf: uffizi: shut disk warnings up for read-only partitions (authored by olasd).
uffizi: shut disk warnings up for read-only partitions
Thu, Oct 3, 5:30 PM
olasd committed rSPSITEc5a7959a2adf: Specify compression on uffizi objstorages (authored by olasd).
Specify compression on uffizi objstorages
Thu, Oct 3, 5:02 PM
olasd committed rSPSITE879774948d65: Layer two objstorages on uffizi (authored by olasd).
Layer two objstorages on uffizi
Thu, Oct 3, 4:26 PM
olasd closed T2030: Certficate for https://stats.export.softwareheritage.org/ has expired as Resolved.

There was a bug in the (manually modified) Let's Encrypt config for this domain, which was silently ignored by certbot (nice), and prevented the renewal to trickle down to puppet.

Thu, Oct 3, 3:52 PM · System administration
olasd committed rSPSITEe9ccce2d3ccb: Notify apache on let's encrypt certificate changes (authored by olasd).
Notify apache on let's encrypt certificate changes
Thu, Oct 3, 3:48 PM
olasd committed rSPSITEbcdc826aa3ac: No more /tank on uffizi (authored by olasd).
No more /tank on uffizi
Thu, Oct 3, 3:48 PM

Wed, Oct 2

olasd closed D2068: test_storage: Use the db_transaction context manager instead of raw connections.
Wed, Oct 2, 5:57 PM
olasd committed rDSTO3228b18eeba9: test_storage: Use the db_transaction context manager instead of raw connections (authored by olasd).
test_storage: Use the db_transaction context manager instead of raw connections
Wed, Oct 2, 5:57 PM
olasd committed rDSTO9bbd5e2344d8: Postgres storage: introduce a db() context manager (authored by olasd).
Postgres storage: introduce a db() context manager
Wed, Oct 2, 5:57 PM
olasd committed rDSTO4c2c311a775f: test_storage: introduce a db_transaction context manager (authored by olasd).
test_storage: introduce a db_transaction context manager
Wed, Oct 2, 5:57 PM
olasd closed D2066: Postgres storage: introduce a db() context manager.
Wed, Oct 2, 5:57 PM
olasd closed D2067: test_storage: introduce a db_transaction context manager.
Wed, Oct 2, 5:57 PM
olasd committed rDSTOe8c01a257293: Replace all 'inmemory' journal writers with 'memory' (authored by olasd).
Replace all 'inmemory' journal writers with 'memory'
Wed, Oct 2, 5:57 PM
olasd committed rDSTO24ea71f77213: test_storage: Close all pool connections on teardown (authored by olasd).
test_storage: Close all pool connections on teardown
Wed, Oct 2, 5:57 PM
olasd closed D2064: test_storage: Close all pool connections on teardown.
Wed, Oct 2, 5:57 PM
olasd closed D2065: Replace all 'inmemory' journal writers with 'memory'.
Wed, Oct 2, 5:57 PM
olasd created D2068: test_storage: Use the db_transaction context manager instead of raw connections.
Wed, Oct 2, 5:25 PM
olasd created D2067: test_storage: introduce a db_transaction context manager.
Wed, Oct 2, 5:24 PM
olasd created D2066: Postgres storage: introduce a db() context manager.
Wed, Oct 2, 5:24 PM
olasd created D2065: Replace all 'inmemory' journal writers with 'memory'.
Wed, Oct 2, 5:24 PM
olasd created D2064: test_storage: Close all pool connections on teardown.
Wed, Oct 2, 5:23 PM
olasd committed rDSTOc1ce3a14b91b: Add test triggering the race condition in content_add (authored by olasd).
Add test triggering the race condition in content_add
Wed, Oct 2, 1:30 PM
olasd closed D2041: Add test triggering the race condition in content_add.
Wed, Oct 2, 1:30 PM

Tue, Oct 1

olasd committed rDSTOb827ff331507: Add missing build-dependency on python3-swh.journal (authored by olasd).
Add missing build-dependency on python3-swh.journal
Tue, Oct 1, 6:30 PM
olasd committed rDSCH6dad1f681362: Force postgresql executable to a pg_ctl that exists when running tests. (authored by olasd).
Force postgresql executable to a pg_ctl that exists when running tests.
Tue, Oct 1, 6:26 PM
olasd updated the diff for D2041: Add test triggering the race condition in content_add.

♪┏(・o・)┛♪┗ ( ・o・) ┓♪ do the rebase dance party

Tue, Oct 1, 6:07 PM
olasd accepted D2056: swh identify: add support to compute snapshot PIDs of on-disk git repo.
In D2056#47656, @zack wrote:
In D2056#47634, @olasd wrote:

Would it be possible to make that extra dependency on dulwich optional, e.g. only installed when requesting swh.model[identify]?

Good one. How about swh.model[cli] instead though?

Tue, Oct 1, 6:02 PM
olasd updated the diff for D2041: Add test triggering the race condition in content_add.

Mark race condition test as xfail

Tue, Oct 1, 5:47 PM
olasd accepted D2057: requests: Set timeout to prevent hangs when querying external services.

Sounds like a good first fix, thanks.

Tue, Oct 1, 5:45 PM
olasd added a comment to D2056: swh identify: add support to compute snapshot PIDs of on-disk git repo.

Would it be possible to make that extra dependency on dulwich optional, e.g. only installed when requesting swh.model[identify]?

Tue, Oct 1, 5:43 PM
olasd triaged T2028: swh-web should use a proper elasticsearch library to do its requests as Normal priority.
Tue, Oct 1, 3:44 PM · Web app
olasd closed T2027: CI: Run swh-graph tests with Python 3.7 as Resolved.

I've rebased the jdk11 image on top of buster using the awful, awful a3776d744f. This means python 3.7 by default, and it looks like swh.graph CI works.

Tue, Oct 1, 3:09 PM · Continuous Integration
olasd committed rCDFJa3776d744fb8: Inline all the things to build from a buster image (authored by olasd).
Inline all the things to build from a buster image
Tue, Oct 1, 2:56 PM
olasd committed rDOBJS5a37cbc9f4c2: Make the null compression use the "none" identifier instead of None (authored by olasd).
Make the null compression use the "none" identifier instead of None
Tue, Oct 1, 11:53 AM
olasd closed D2051: Make the null compression use the "none" identifier instead of None.
Tue, Oct 1, 11:53 AM
olasd updated the diff for D2051: Make the null compression use the "none" identifier instead of None.

Switch conditions to get a ==

Tue, Oct 1, 11:40 AM
olasd committed rDOBJSb8aef443aa0e: Add tests for the behavior on trailing data after the compressed blob (authored by olasd).
Add tests for the behavior on trailing data after the compressed blob
Tue, Oct 1, 11:02 AM
olasd committed rDOBJS40645dfcc065: Implement pluggable compression in ObjStoragePathSlicing (authored by olasd).
Implement pluggable compression in ObjStoragePathSlicing
Tue, Oct 1, 11:02 AM
olasd closed D2050: Implement pluggable compression in ObjStoragePathSlicing.
Tue, Oct 1, 11:02 AM
olasd closed D2049: Add tests for the behavior on trailing data after the compressed blob.
Tue, Oct 1, 11:02 AM
olasd committed rDOBJS018ca27e87fe: Add raw compression test to seaweedfs backend (authored by olasd).
Add raw compression test to seaweedfs backend
Tue, Oct 1, 11:02 AM
olasd committed rDOBJSf8d3c60e6708: Support pluggable compression in azure objstorage (authored by olasd).
Support pluggable compression in azure objstorage
Tue, Oct 1, 11:02 AM