Related to T2591
Details
Details
- Reviewers
olasd ardumont - Group Reviewers
System administrators - Maniphest Tasks
- T2591: Filebeat fails to start on moma
- Commits
- rSPSITEc56d9612ca75: Manage filebeat configuration via puppet
Octocatalog-diff for moma :
*** Running octocatalog-diff on host moma.softwareheritage.org
I, [2020-09-21T14:17:34.928892 #4635] INFO -- : Catalogs compiled for moma.softwareheritage.org
I, [2020-09-21T14:17:36.404154 #4635] INFO -- : Diffs computed for moma.softwareheritage.org
diff origin/production/moma.softwareheritage.org current/moma.softwareheritage.org
*******************************************
+ File[/etc/filebeat/filebeat.yml] =>
parameters =>
"content": "# File managed by puppet - modifications will be lost\nfilebeat....
"ensure": "present",
"group": "root",
"mode": "0644",
"notify": "Service[filebeat]",
"owner": "root"
*******************************************
+ File[/etc/filebeat/inputs.d/deposit-non-ssl-access.yml] =>
parameters =>
"content": "# File managed by puppet - modifications will be lost\n- type: l...
"ensure": "present",
"group": "root",
"mode": "0644",
"notify": "Service[filebeat]",
"owner": "root"
*******************************************
+ File[/etc/filebeat/inputs.d/webapp-non-ssl-access.yml] =>
parameters =>
"content": "# File managed by puppet - modifications will be lost\n- type: l...
"ensure": "present",
"group": "root",
"mode": "0644",
"notify": "Service[filebeat]",
"owner": "root"
*******************************************
+ File[/etc/filebeat/inputs.d] =>
parameters =>
"ensure": "directory",
"group": "root",
"mode": "0755",
"owner": "root",
"purge": true
*******************************************
+ Profile::Filebeat::Log_input[deposit-non-ssl-access] =>
parameters =>
"fields": {
"apache_log_type": "access_log"
},
"paths": [
"/var/log/apache2/deposit.softwareheritage.org_non-ssl_access.log"
]
*******************************************
+ Profile::Filebeat::Log_input[webapp-non-ssl-access] =>
parameters =>
"fields": {
"apache_log_type": "access_log"
},
"paths": [
"/var/log/apache2/archive.softwareheritage.org_non-ssl_access.log"
]
*******************************************
*** End octocatalog-diff on moma.softwareheritage.orgDiff Detail
Diff Detail
- Repository
- rSPSITE puppet-swh-site
- Branch
- staging
- Lint
No Linters Available - Unit
No Unit Test Coverage - Build Status
Buildable 15196 Build 23430: arc lint + arc unit
Event Timeline
| site-modules/profile/manifests/swh/deploy/webapp.pp | ||
|---|---|---|
| 288 | swh/deploy/deposit.pp could use this as well ;) | |
Comment Actions
I don't like using concat and ERB templates to generate yaml files at all:
- you need to remember to escape your variables in the ERB template
- you need to keep track of indent levels across the concat fragments
- you need to have explicit indents in the ERB which is ugly
I very much prefer generating the config "data structure" as a puppet dict, then serializing that to yaml with the inline_yaml command.
Now this begs the question of how to "integrate" multiple snippets from various profiles (swh-web, swh-deposit) into this main config file.
I think the best way to do this is to support the filebeat "drop-in" input configurations:
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-reloading.html
- generate a base configuration in the filebeat profile, which contains an include for input drop-ins as well as the output config.
- have a profile::filebeat::input define which generates a drop-in file with input settings
- call the profile::filebeat::input defined resource in the swh-web and swh-deposit deployments, pointing at the apache configs
Comment Actions
Use the capability of filebeat to use a file per input.
It's a lot more simple than build a single big yaml file with several concat
Comment Actions
Thanks! This indeed looks much less fragile than the concat version.
I've left a bunch of inline style and functionality comments.
I still don't think we should be templating yaml files. Please build the config structure as a hash within the puppet manifest, and use the inline_yaml function to generate the file contents. That function also generates a "changes will be lost" header automatically.
From memory, you can have a look at the way we generate the prometheus or grafana configurations for inspiration.
| data/defaults.yaml | ||
|---|---|---|
| 2512–2514 | There should be one main logstash_hosts variable, set to the proper value, and used as an "%{alias()}" in other "service-specific" variables. This way we can change a single variable to point at another logstash server, e.g. per environment, while still keeping separate variables per profile. | |
| site-modules/profile/manifests/filebeat.pp | ||
| 4–5 | For now we've mostly avoided using class arguments for profiles, to avoid being surprised by hiera variable "autoloading". (what i mean by hiera autoloading : if you set a profile::filebeat::config_directory hiera variable, it'll override the "'/etc/filebeat'" you've set here) I don't know if that's a sensible choice, and it can certainly be reconsidered, but for now I think we should keep it consistent. | |
| 30 | misaligned => | |
| 31 | Needs an explicit owner, group and mode (probably '0644'). Also should consider using the purge attribute to remove leftover files. | |
| site-modules/profile/manifests/filebeat/log_input.pp | ||
| 3 | This should be a Hash, not an Array. (It is, after all, a hash in the configuration file itself). | |
| 5 | Needs an include profile::filebeat. That way we're sure that the directory resource is defined etc. | |
| 8 | I'd even make the inputs.d directory an explicit variable in profile::filebeat, so the directory resource there and the path here are guaranteed to match. I'll also note that you can access the variable using this scope even if it's defined in the body of the class (no need to have it as a class argument). | |
| 14 | Is this really needed? | |
Comment Actions
update the diff according the previous feedbacks :
- The logstash hosts are declared on a uniq property
- No more yaml templates ;)
- The profile::filbeat doesn't use parameters anymore
- Ensure the permissions are correctly set
- Add the purge option on the inputs.d directory