Wed, Feb 10
Fri, Feb 5
It seems there were some huge queries the last few days, the script needed to be adapted to use Long instead of Integers :
apache_logs-2021.01.14:
{
"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:68)",
"java.base/java.lang.Integer.parseInt(Integer.java:652)",
"java.base/java.lang.Integer.parseInt(Integer.java:770)",
"ctx._source.bytes = ctx._source.bytes instanceof java.lang.String ? Integer.parseInt(ctx._source.bytes) : ctx._source.bytes; ",
" ^---- HERE"
],
"script" : "ctx._source.bytes = ctx._source.bytes instanceof java.lang.String ? Integer.parseInt(ctx._source.bytes) : ctx._source.bytes; ctx._source.response = ctx._source.response instanceof java.lang.String ? Integer.parseInt(ctx._source.response) : ctx._source.response;",
"lang" : "painless",
"position" : {
"offset" : 96,
"start" : 0,
"end" : 125
}
}
],
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:68)",
"java.base/java.lang.Integer.parseInt(Integer.java:652)",
"java.base/java.lang.Integer.parseInt(Integer.java:770)",
"ctx._source.bytes = ctx._source.bytes instanceof java.lang.String ? Integer.parseInt(ctx._source.bytes) : ctx._source.bytes; ",
" ^---- HERE"
],
"script" : "ctx._source.bytes = ctx._source.bytes instanceof java.lang.String ? Integer.parseInt(ctx._source.bytes) : ctx._source.bytes; ctx._source.response = ctx._source.response instanceof java.lang.String ? Integer.parseInt(ctx._source.response) : ctx._source.response;",
"lang" : "painless",
"position" : {
"offset" : 96,
"start" : 0,
"end" : 125
},
"caused_by" : {
"type" : "number_format_exception",
"reason" : "For input string: \"4633815064\""
}
},
"status" : 400
}Thu, Feb 4
The opened apache indexes are currently being migrated with the P940 script.
The log parsing is ok.
An elasticsearch datasource was created on grafana so we can now create some graphs based on the logs on elasticsearch.
A simple dashboard to display some statistics based on the apache log was initiated[1], it appears the design is not as simple as in kibana and have some limitations but it still allows to have basic information centralized in grafana.
Tue, Feb 2
Configuration deployed for the webapp on all servers, the logs have now the duration, which is parsed on the elasticseach entries :
Jan 29 2021
Nov 17 2020
The varnish logs should be also ingested to elasticsearch to have fine grained statistics.
Nov 3 2020
Oct 26 2020
Oct 16 2020
This can be closed now.
Sep 22 2020
I think the second point mostly happened: the storage is returning statistics to the loader, but the loaders don't generally collect them.
We've definitely improved on this (notably using proper hostnames for the instance label on prom metrics). I think we should make this task more actionable if we want to keep it open.
Apr 21 2020
I'm pretty sure this is done now ;p
Feb 15 2020
Jan 27 2020
Jan 23 2020
Deployed.
Jan 22 2020
Adapting the puppet manifest so we can discriminate issues per environment in sentry.
Vault check deployed!
Deposit check deployed!
debian package this
Jan 20 2020
debian package this
Jan 17 2020
As far as i could tell so far:
- debian package this
- update puppet configuration to add the checks [1]