Page MenuHomeSoftware Heritage

olasd (Nicolas Dandrimont)
UserAdministrator

Projects (9)

User Details

User Since
Sep 7 2015, 3:25 PM (354 w, 4 d)
Roles
Administrator

Recent Activity

Wed, Jun 22

olasd committed rSPSITE77b70356949d: Switch moma over to the main db for the somerset/beaubourg migration (authored by olasd).
Switch moma over to the main db for the somerset/beaubourg migration
Wed, Jun 22, 8:19 PM
olasd closed D8027: Drop args key from all service backend instantiations.
Wed, Jun 22, 7:07 PM
olasd committed rSPSITEdf2bb2a0d5f0: Drop args key from all service backend instantiations (authored by olasd).
Drop args key from all service backend instantiations
Wed, Jun 22, 7:07 PM
olasd requested review of D8027: Drop args key from all service backend instantiations.
Wed, Jun 22, 4:23 PM
olasd accepted D8024: Add return type to get_objstorage + deduplicate its code.
Wed, Jun 22, 4:03 PM
olasd added a comment to D8024: Add return type to get_objstorage + deduplicate its code.

We've run mypy on dependent modules and they seem to be happy enough.

Wed, Jun 22, 4:02 PM
olasd accepted D8014: Add type annotations to all objstorage backends.

The factory type annotations are in D8024.

Wed, Jun 22, 4:01 PM

Tue, Jun 21

olasd changed the status of T4335: Archive repo.or.cz from Open to Work in Progress.

I've scheduled the archival of the 7377 repos in one of the leftover one-shot queues.

Tue, Jun 21, 10:07 PM · Archive coverage
olasd closed D8011: Always set object id when adding objects to objstorage.
Tue, Jun 21, 7:22 PM
olasd committed rDOBJSRPL8b7223480f66: Always set object id when adding objects to objstorage (authored by olasd).
Always set object id when adding objects to objstorage
Tue, Jun 21, 7:22 PM
olasd closed D8013: Drop the now unused add_stream and get_stream methods.
Tue, Jun 21, 5:56 PM
olasd committed rDOBJS2c7e49136af6: Drop the now unused add_stream and get_stream methods (authored by olasd).
Drop the now unused add_stream and get_stream methods
Tue, Jun 21, 5:56 PM
olasd closed D8012: cache: Drop unused method get_stream.
Tue, Jun 21, 5:56 PM
olasd committed rDVAU6436f0bb3a62: cache: Drop unused method get_stream (authored by olasd).
cache: Drop unused method get_stream
Tue, Jun 21, 5:56 PM
olasd requested review of D8011: Always set object id when adding objects to objstorage.
Tue, Jun 21, 5:14 PM
olasd added a revision to T3775: Dealing with repositories with contents that produces hash conflicts (example included from GitLab): D8011: Always set object id when adding objects to objstorage.
Tue, Jun 21, 4:21 PM · meta-task, Roadmap 2022, Save Code Now, Origin-GitLab
olasd closed D8008: Set object id when calling objstorage.add.
Tue, Jun 21, 4:03 PM
olasd committed rDDATASETe31bdb26a827: Set object id when calling objstorage.add (authored by olasd).
Set object id when calling objstorage.add
Tue, Jun 21, 4:03 PM
olasd requested review of D8008: Set object id when calling objstorage.add.
Tue, Jun 21, 2:38 PM
olasd added a revision to T2309: Add support for other hash algo than sha1 in current objstorage implementation: D8008: Set object id when calling objstorage.add.
Tue, Jun 21, 2:35 PM · Object storage

Fri, Jun 17

olasd added inline comments to D7985: [provenance] Implement a naive archive multiplexer.
Fri, Jun 17, 3:42 PM · Provenance database

Thu, Jun 16

olasd added a comment to T4250: Native hadoop libraries during graph compression.

So, as was mentioned during the irc discussion, one of the possible ways forward is to:

Thu, Jun 16, 2:32 PM · Compressed graph service
olasd accepted D7993: Uncompress support '.tar.zst' extension and 'application/zstd' mime type..
Thu, Jun 16, 12:13 PM
olasd accepted D7993: Uncompress support '.tar.zst' extension and 'application/zstd' mime type..

Thanks!

Thu, Jun 16, 12:05 PM
olasd closed T4311: Package and deploy dulwich 0.20.43 in production as Resolved.

All production loaders have been restarted now.

Thu, Jun 16, 11:36 AM · System administration, Git loader

Fri, Jun 10

olasd added a comment to T4320: Upgrade gitlab instance to v15.

After killing the minio data store, we've decided (as we had planned anyway) to start again from a clean slate.

Fri, Jun 10, 6:41 PM · System administration, GitLab migration, Roadmap 2020

Thu, Jun 9

olasd added a comment to T4311: Package and deploy dulwich 0.20.43 in production.

I've restarted the staging workers (loader_git and loader_high_priority) with the new dulwich version

Thu, Jun 9, 5:36 PM · System administration, Git loader

Wed, Jun 8

olasd added a comment to T4311: Package and deploy dulwich 0.20.43 in production.

fwiw, jenkins is python3-dulwich aware.

Wed, Jun 8, 3:59 PM · System administration, Git loader
olasd added a comment to T4311: Package and deploy dulwich 0.20.43 in production.

I checked that the swh.loader.git tests are green with the new dulwich version.

Wed, Jun 8, 3:51 PM · System administration, Git loader
olasd changed the status of T4311: Package and deploy dulwich 0.20.43 in production from Open to Work in Progress.
Wed, Jun 8, 3:43 PM · System administration, Git loader
olasd added a comment to T4316: Push of swh-graph to pypi is broken.

We've asked for another bump at https://github.com/pypa/pypi-support/issues/1998.

Wed, Jun 8, 3:33 PM · System administration, Compressed graph service

May 20 2022

olasd added a comment to D7871: Add metrics in store_data on ratios of objects already stored.

Thanks for working on this, I think getting these insights will be useful.

May 20 2022, 3:12 PM

May 16 2022

olasd added a comment to T4250: Native hadoop libraries during graph compression.

4.a. properly declare this in the maven dependencies of swh.graph
4.b. ensure the container image generation pipeline and container entrypoint script properly handle this extra dependency and argument

May 16 2022, 6:32 PM · Compressed graph service
olasd accepted D7834: Use all snapshots to compute base_snapshot_reverse_branches.

Oops, thanks!

May 16 2022, 3:07 PM

May 13 2022

olasd committed rSPSITE058c80ac4085: prometheus: scrape mmca.softwareheritage.org using federation (authored by olasd).
prometheus: scrape mmca.softwareheritage.org using federation
May 13 2022, 8:49 PM
olasd added a comment to T4225: Deploy a more recent version of prometheus-statsd-exporter on all nodes.

That's deployed on all nodes and validated as working now.

May 13 2022, 4:21 PM · System administration
olasd closed T4225: Deploy a more recent version of prometheus-statsd-exporter on all nodes, a subtask of T4219: Investigate why GitHub fork detection did not bring a speed-up, as Resolved.
May 13 2022, 4:20 PM · Origin-GitHub, Git loader
olasd closed T4225: Deploy a more recent version of prometheus-statsd-exporter on all nodes as Resolved.
May 13 2022, 4:20 PM · System administration
olasd updated subscribers of T4239: Unstuck bitbucket incremental lister.

We had a mysterious error 500 issue with one buggy repository in the bitbucket API in the past, which we had reported to their Jira to no avail, but someone at Octobus (I think it was @marmoute or @Alphare ?) managed to reach out to one of the devs at atlassian to actually debug and fix the issue.

May 13 2022, 11:40 AM · Bitbucket lister
olasd added a comment to D7828: BaseLoader: Fix handling of exceptions derived directly from BaseException.

It may be worth adding a test for a loader getting a BaseException derivate in one of the early methods too. But that's a bit nitpicky.

May 13 2022, 11:35 AM
olasd accepted D7828: BaseLoader: Fix handling of exceptions derived directly from BaseException.

(you probably want to update the commit message because you're not reverting the other commit anymore !)

May 13 2022, 11:32 AM
olasd added a comment to T4225: Deploy a more recent version of prometheus-statsd-exporter on all nodes.

Puppet ran manually on staging workers now, and the new statsd exporter has been deployed. The metrics properly show up in prometheus.

May 13 2022, 10:29 AM · System administration
olasd moved T4225: Deploy a more recent version of prometheus-statsd-exporter on all nodes from in-progress to deployed/landed/monitoring on the System administration board.
May 13 2022, 10:22 AM · System administration
olasd closed D7826: Move the prometheus statsd exporter to use the upstream tarball.
May 13 2022, 10:16 AM
olasd committed rSPSITE899dca0ec287: Move the prometheus statsd exporter to use the upstream tarball (authored by olasd).
Move the prometheus statsd exporter to use the upstream tarball
May 13 2022, 10:16 AM
olasd added inline comments to D7826: Move the prometheus statsd exporter to use the upstream tarball.
May 13 2022, 10:08 AM

May 12 2022

olasd requested review of D7826: Move the prometheus statsd exporter to use the upstream tarball.
May 12 2022, 8:22 PM
olasd added a revision to T4225: Deploy a more recent version of prometheus-statsd-exporter on all nodes: D7826: Move the prometheus statsd exporter to use the upstream tarball.
May 12 2022, 8:22 PM · System administration
olasd renamed T4225: Deploy a more recent version of prometheus-statsd-exporter on all nodes from Package more recent version of prometheus-statsd-exporter to Deploy a more recent version of prometheus-statsd-exporter on all nodes.
May 12 2022, 11:35 AM · System administration
olasd added a comment to T4225: Deploy a more recent version of prometheus-statsd-exporter on all nodes.

My debian package attempts went nowhere, it seems that one of the 3rd level dependencies needs a newer golang than what's available in buster-backports, so we'll get rid of the (homegrown anyway) debian package and go the upstream binaries route.

May 12 2022, 11:34 AM · System administration
olasd merged task T4235: [As a temporary solution] deploy the statsd-exporter binary published by prometheus into T4225: Deploy a more recent version of prometheus-statsd-exporter on all nodes.
May 12 2022, 11:29 AM · System administration
olasd merged T4235: [As a temporary solution] deploy the statsd-exporter binary published by prometheus into T4225: Deploy a more recent version of prometheus-statsd-exporter on all nodes.
May 12 2022, 11:29 AM · System administration

May 11 2022

olasd committed R259:6757ba7a79ff: Add source requirements for swh-provenance-{client,server} (authored by olasd).
Add source requirements for swh-provenance-{client,server}
May 11 2022, 5:31 PM
olasd committed R259:ef2f45c9a1e8: swh-provenance-client: update requirements-frozen.txt (authored by olasd).
swh-provenance-client: update requirements-frozen.txt
May 11 2022, 5:31 PM
olasd committed R259:4b921bfb53d8: Add a script to generate frozen requirements files (authored by olasd).
Add a script to generate frozen requirements files
May 11 2022, 5:31 PM
olasd committed R259:ec2e69bf9141: Bootstrap pre-commit hooks and other repo configs (authored by olasd).
Bootstrap pre-commit hooks and other repo configs
May 11 2022, 5:31 PM
olasd committed R259:e22ed26aa5e6: Bootstrap repository legalese (authored by olasd).
Bootstrap repository legalese
May 11 2022, 5:31 PM
olasd committed rDPROVeec175540628: Ignore provenance tools in mypy for now... (authored by olasd).
Ignore provenance tools in mypy for now...
May 11 2022, 5:23 PM
olasd committed rDPROV736f413863a8: Fix codespell warning in swh/provenance/tools (authored by olasd).
Fix codespell warning in swh/provenance/tools
May 11 2022, 5:23 PM
olasd committed rDPROVb94f145e8724: postgresql: annotate transaction with the proper DictCursor type (authored by olasd).
postgresql: annotate transaction with the proper DictCursor type
May 11 2022, 5:23 PM
olasd committed rDPROV74ef01d4f4ec: Drop conflicting .gitignore (authored by olasd).
Drop conflicting .gitignore
May 11 2022, 5:23 PM
olasd committed rDPROV90ef2faef5c6: Merge swh-provenance tools in main repository (authored by olasd).
Merge swh-provenance tools in main repository
May 11 2022, 5:23 PM
olasd committed rDPROVe3bd0b354f37: Appease flake8 on the tools directory (authored by olasd).
Appease flake8 on the tools directory
May 11 2022, 5:23 PM
olasd committed rDPROVb5cbf44f3b17: Update linting tools to match the rest of swh-environment (authored by olasd).
Update linting tools to match the rest of swh-environment
May 11 2022, 5:23 PM
olasd committed rDPROVcd4be335d34b: Move all tools to swh/provenance/tools (authored by olasd).
Move all tools to swh/provenance/tools
May 11 2022, 5:23 PM
olasd committed rSENV21431f369459: Update octocatalog-diff facts (authored by olasd).
Update octocatalog-diff facts
May 11 2022, 3:56 PM
olasd accepted D7810: keycloak: Add swh.web.api.raw_object role.
May 11 2022, 1:43 PM
olasd committed R259:2742f2bc1f6a: Bootstrap README file (authored by olasd).
Bootstrap README file
May 11 2022, 11:55 AM
olasd added a comment to D7809: elastic-workers: Let the loader some time to finish gracefully.

I expect that the clean celery worker termination command would also be useful for the current prod setup

May 11 2022, 10:31 AM
olasd closed D7808: Add add-forge-now inbound email address.
May 11 2022, 10:12 AM
olasd committed rSPSITE397a88748f36: Add add-forge-now inbound email address (authored by olasd).
Add add-forge-now inbound email address
May 11 2022, 10:12 AM

May 10 2022

olasd added a comment to D7809: elastic-workers: Let the loader some time to finish gracefully.

(basically, the liveness probe could do what our current swh-worker-ping-restart script does: call swh scheduler celery-monitor --pattern "$celery_name" ping-workers a couple of times. We just need to figure out how to generate the $celery_name value consistently between the entrypoint and the liveness probe)

May 10 2022, 7:18 PM
olasd added a comment to D7809: elastic-workers: Let the loader some time to finish gracefully.

What component is providing the health http endpoint in the pod?

May 10 2022, 7:13 PM
olasd accepted D7806: cvs.loader: Decrease log level verbosity to debug.

Yeah, that's definitely too verbose for normal operation!

May 10 2022, 7:05 PM
olasd added a comment to D7805: Allow setting up logging level in core loader.

Yeah, I'm not saying it should happen now, but the way this stuff has been handled has been inconsistent/painful/annoying me for a while, so maybe it's time?

May 10 2022, 7:02 PM
olasd added a comment to D7805: Allow setting up logging level in core loader.

I'm a bit uncomfortable with adding more ad-hoc logger.setLevel calls deep in SWH code (which then make reasoning about the logging config harder), rather than ripping it all apart and properly setting up the logging machinery "the python way" with a clean, explicit dictConfig (potentially pulled from a yaml file set as an envvar?).

May 10 2022, 6:46 PM
olasd requested review of D7808: Add add-forge-now inbound email address.
May 10 2022, 6:29 PM
olasd added a revision to T3999: [Add forge now] Design a feedback tracking solution: D7808: Add add-forge-now inbound email address.
May 10 2022, 6:29 PM · Add Forge Now
olasd accepted D7807: Remove the wrong dependency added in the previous commit.

*yeet*

May 10 2022, 6:10 PM
olasd published D7807: Remove the wrong dependency added in the previous commit for review.
May 10 2022, 6:09 PM
olasd committed rDWAPPSdc5d570ea30a: Add forge now: process inbound emails (authored by olasd).
Add forge now: process inbound emails
May 10 2022, 5:09 PM
olasd committed rDWAPPS080c988fe920: Add forge now: only catch expected exceptions in request dashboard (authored by olasd).
Add forge now: only catch expected exceptions in request dashboard
May 10 2022, 5:09 PM
olasd closed D7326: Hook up processing of inbound emails for add_forge_now.
May 10 2022, 5:09 PM
olasd committed rDWAPPSf80541830b61: Add forge now: set Cc and Reply-To for admin emails (authored by olasd).
Add forge now: set Cc and Reply-To for admin emails
May 10 2022, 5:09 PM
olasd updated the diff for D7326: Hook up processing of inbound emails for add_forge_now.

Reset commit dates

May 10 2022, 4:22 PM
olasd published D7326: Hook up processing of inbound emails for add_forge_now for review.
May 10 2022, 4:19 PM
olasd committed rDWAPPS5d37baf27ec0: Add forge now: add view to retrieve RequestHistory.message_source (authored by olasd).
Add forge now: add view to retrieve RequestHistory.message_source
May 10 2022, 4:15 PM
olasd closed D7799: Add forge now: prepare fields for inbound email processing.
May 10 2022, 4:15 PM
olasd committed rDWAPPS633b4319c4ad: Add forge now: prepare fields for inbound email processing (authored by olasd).
Add forge now: prepare fields for inbound email processing
May 10 2022, 4:15 PM
olasd updated the diff for D7799: Add forge now: prepare fields for inbound email processing.

Rebase; rename all raw_message fields to message_source

May 10 2022, 3:51 PM
olasd added a comment to D7799: Add forge now: prepare fields for inbound email processing.

what about "message_source" instead of "raw_message"? IMO it's clearer

May 10 2022, 1:42 PM
olasd requested review of D7799: Add forge now: prepare fields for inbound email processing.
May 10 2022, 1:41 PM
olasd closed D7798: Add Forge Now: add a forge_domain property for use in templates.
May 10 2022, 1:38 PM
olasd committed rDWAPPS9b80cf370659: Add Forge Now: add a forge_domain property for use in templates (authored by olasd).
Add Forge Now: add a forge_domain property for use in templates
May 10 2022, 1:38 PM
olasd closed D7753: add_forge_now: add inbound email address to model.
May 10 2022, 1:37 PM
olasd committed rDWAPPS833b171c6932: Add forge now: add inbound email address to model (authored by olasd).
Add forge now: add inbound email address to model
May 10 2022, 1:37 PM
olasd requested review of D7798: Add Forge Now: add a forge_domain property for use in templates.
May 10 2022, 1:35 PM
olasd updated the diff for D7753: add_forge_now: add inbound email address to model.

Rebase and add docstring

May 10 2022, 1:11 PM

May 9 2022

olasd accepted D7793: logstash: Fix the closed index's error filtering.
May 9 2022, 4:00 PM
olasd accepted D7764: icinga checks: Activate the prometheus export on e2e tests.

🤷🤷🤷 let's try it and see...

May 9 2022, 11:19 AM