Page MenuHomeSoftware Heritage

Add request durations in access logs and improve logstash's integer parsing
ClosedPublic

Authored by vsellier on Feb 2 2021, 9:55 AM.

Details

Summary

Several commit to limit the review :

webapp: code format

webapp: improve access log parsing

  • fix a hardcoded path to the apache log file, filebeat was not reading it when the vhost name is not webapp.
  • add an environment field to easily identify the origin of a log entry
  • add a vhost field to easily regroup the logs of a vhost (can be also added on varnish side)

logstash: Add support an optional duration on apache logs

It will allow to monitor the front responses time per requests.
The conversion of the request and duration fields to integer is
mandatory to use them in some aggregation.
The ES indexes will have to be reindexed if we want to have the full
history.

apache: Add the request duration on access logs

It will add some visibility on the monitoring

Related to T2787

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

lgtm

Please also update the deposit.pp which can benefit from this as well ;)

Remove wrong float conversion on grok pattern

lgtm

Please also update the deposit.pp which can benefit from this as well ;)

Yes right, I will do that

Yes right, I will do that

as exchanged orally, in another diff ;)

so fine

This revision is now accepted and ready to land.Feb 2 2021, 3:06 PM