Page MenuHomeSoftware Heritage

Properly declare a centralized logging configuration and then reuse
Closed, MigratedEdits Locked

Description

We need to simplify and centralize once and for all our logging configuration so it
becomes clearer and more maintainable. They grew organically around multiple different
implementation or services (celery task, gunicorn, cli...).

@olasd proposed a clear plan for that [1]

  • lift the logic [2] into a common utility function (moved to swh.core.logging/logger)
  • Drop the celery task backend configuration logging in favor of a call to the new utility function ^, pulling its arguments from a couple of environment variables (and the celery command line if relevant). Ensure the envvars are consistent with the ones that the swh command line supports.
  • Replace the gunicorn logconfig.ini [4] and the associated setup [5] to use the same utility function, with consistent environment variables.
  • Coalesce all the ad-hoc logger.setLevel calls strewn across our codebase, as well as gunicorn's hardcoded logconfig.ini, into a (bunch of) common yaml logging setup file(s) defined in puppet and docker.
  • update all the entry points / puppet deployment manifests / ... to use that common config file and envvar

[1] Copy/pasted from the excellent plan D7805#203180

[2] https://forge.softwareheritage.org/source/swh-core/browse/master/swh/core/cli/__init__.py$137-156

[3] https://forge.softwareheritage.org/source/swh-scheduler/browse/master/swh/scheduler/celery_backend/config.py$82-138

[4] https://forge.softwareheritage.org/source/puppet-swh-gunicorn/browse/master/files/logconfig.ini

[5] https://forge.softwareheritage.org/source/puppet-swh-gunicorn/browse/master/templates/gunicorn-instance.cfg.erb$40-42

Event Timeline

ardumont renamed this task from Properly centralize celery tasks logging configuration to Properly declare a centralized logging configuration and then reuse.May 11 2022, 9:18 AM
ardumont triaged this task as High priority.
ardumont lowered the priority of this task from High to Normal.
ardumont created this task.