Page MenuHomeSoftware Heritage

elk: upgrade journalbeat and filebeat
ClosedPublic

Authored by vsellier on Wed, Nov 10, 12:49 PM.

Details

Summary
  • Remove the journalbeat 5.5.0 service configuration
  • remove the journalbeat user not needed anymore
  • deploy the elasticsearch packages (journalbeat and filebeat)
  • cleanup the temporary configuration
  • cleanup the old cursor position, when the version will be updated the beginning of the journal will be reingested
  • update the logstash configuration to support the messages from the old journalbeat version and the new version in parallel
  • allow to override the beat version for node with an elasticsearch version specified (swh-search)
  • upate the check_journal script to use the new registry file to check the cursor position
  • Automatically declare the journalbeat index templates from the logstash host

The mappings of the new systemlogs-7.15.1-* and swh_worker-7.15.1-*
index need to be manually declared on elasticsearch before.

Related to T3705
Depends on D6634

Test Plan
  • logstash0:
diff origin/production/logstash0.internal.softwareheritage.org current/logstash0.internal.softwareheritage.org
*******************************************
+ Apt::Pin[journalbeat] =>
   parameters =>
     "codename": "",
     "component": "",
     "ensure": "present",
     "explanation": "Use the elk stack version",
     "label": "",
     "order": 50,
     "origin": "",
     "originator": "",
     "packages": [
       "journalbeat"
     ],
     "priority": 1001,
     "release": "",
     "release_version": "",
     "version": "7.15.1"
*******************************************
  Apt::Pin[swh-journalbeat] =>
   parameters =>
     ensure =>
      - present
      + absent
     explanation =>
      - Use journalbeat packages from Software Heritage
     originator =>
      - softwareheritage
      +_
     packages =>
      - ["journalbeat"]
      + *
     priority =>
      - 990
      + 0
*******************************************
+ Apt::Setting[pref-journalbeat] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\nExplanation: Use...
     "ensure": "present",
     "notify_update": false,
     "priority": 50
*******************************************
  Apt::Setting[pref-swh-journalbeat] =>
   parameters =>
     content =>
      @@ -1,5 +1,5 @@
       # This file is managed by Puppet. DO NOT EDIT.
      -Explanation: Use journalbeat packages from Software Heritage
      -Package: journalbeat
      -Pin: release o=softwareheritage
      -Pin-Priority: 990
      +Explanation: profile: swh-journalbeat
      +Package: *
      +Pin: release a=swh-journalbeat
      +Pin-Priority: 0
     ensure =>
      - present
      + absent
*******************************************
+ File[/etc/apt/preferences.d/journalbeat.pref] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\nExplanation: Use...
     "ensure": "present",
     "group": "root",
     "mode": "0644",
     "owner": "root"
*******************************************
  File[/etc/apt/preferences.d/swh-journalbeat.pref] =>
   parameters =>
     ensure =>
      - present
      + absent
*******************************************
  File[/etc/journalbeat/journalbeat.yml] =>
   parameters =>
     content =>
      @@ -2,4 +2,10 @@
      _
       journalbeat:
      +  inputs:
      +    # Paths that should be crawled and fetched. Possible values files and directories.
      +    # When setting a directory, all journals under it are merged.
      +    # When empty starts to read from local journal.
      +    - paths: []
      +
         # What position in journald to seek to at start up
         # options: cursor, tail, head (defaults to tail)
*******************************************
- File[/etc/journalbeat]
*******************************************
  File[/etc/logstash/conf.d/filter.conf] =>
   parameters =>
     content =>
      @@ -18,4 +18,5 @@
           }
         } else if "swh-worker@" in [systemd_unit] {
      +    # Temporary rule to delete after complete migration to 7.15.1
           mutate {
             add_field => {
      @@ -23,8 +24,23 @@
             }
           }
      -  } else {
      +  } else if "swh-worker@" in [systemd][unit] {
           mutate {
             add_field => {
      -        "[@metadata][target_index]" => "systemlogs-%{+YYYY.MM.dd}"
      +        "[@metadata][target_index]" => "swh_workers-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      +      }
      +    }
      +  } else {
      +    if [@metadata][version] {
      +      mutate {
      +        add_field => {
      +          "[@metadata][target_index]" => "systemlogs-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      +        }
      +      }
      +    } else {
      +      # Temporary rule to delete after complete migration to 7.15.1
      +      mutate {
      +        add_field => {
      +          "[@metadata][target_index]" => "systemlogs-%{+YYYY.MM.dd}"
      +        }
             }
           }
*******************************************
+ File[/etc/systemd/system/journalbeat.service.d/journalbeat.conf] =>
   parameters =>
     "content": "# Managed by puppet (class profile::systemd_journal::journalbeat...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Class[Systemd::Systemctl::Daemon_reload]"
     ],
     "owner": "root",
     "selinux_ignore_defaults": false,
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/journalbeat.service.d] =>
   parameters =>
     "ensure": "directory",
     "group": "root",
     "owner": "root",
     "purge": true,
     "recurse": true,
     "selinux_ignore_defaults": false
*******************************************
  File[/etc/systemd/system/journalbeat.service] =>
   parameters =>
     ensure =>
      - file
      + absent
*******************************************
  Package[journalbeat] =>
   parameters =>
     ensure =>
      - present
      + 7.15.1
*******************************************
  Service[journalbeat] =>
   parameters =>
     subscribe =>
      + ["File[/etc/journalbeat/journalbeat.yml]", "Package[journalbeat]", "Systemd::Dropin_file[journalbeat.conf]"]
*******************************************
+ Systemd::Dropin_file[journalbeat.conf] =>
   parameters =>
     "content": "# Managed by puppet (class profile::systemd_journal::journalbeat...
     "daemon_reload": "lazy",
     "ensure": "present",
     "filename": "journalbeat.conf",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Service[journalbeat]"
     ],
     "owner": "root",
     "path": "/etc/systemd/system",
     "selinux_ignore_defaults": false,
     "show_diff": true,
     "unit": "journalbeat.service"
*******************************************
- Systemd::Unit_file[journalbeat.service]
*******************************************
*** End octocatalog-diff on logstash0.internal.softwareheritage.org

-swh-worker01

diff origin/production/worker01.softwareheritage.org current/worker01.softwareheritage.org
*******************************************
+ Anchor[apt_key 46095ACC8548582C1A2699A9D27D666CD88E42B4 present]
*******************************************
+ Apt::Key[Add key: 46095ACC8548582C1A2699A9D27D666CD88E42B4 from Apt::Source elasticsearch] =>
   parameters =>
     "content": "-----BEGIN PGP PUBLIC KEY BLOCK-----\nVersion: GnuPG v2.0.14 (GN...
     "ensure": "present",
     "id": "46095ACC8548582C1A2699A9D27D666CD88E42B4",
     "server": "keyserver.ubuntu.com",
     "weak_ssl": false
*******************************************
+ Apt::Pin[journalbeat] =>
   parameters =>
     "codename": "",
     "component": "",
     "ensure": "present",
     "explanation": "Use the elk stack version",
     "label": "",
     "order": 50,
     "origin": "",
     "originator": "",
     "packages": [
       "journalbeat"
     ],
     "priority": 1001,
     "release": "",
     "release_version": "",
     "version": "7.15.1"
*******************************************
  Apt::Pin[swh-journalbeat] =>
   parameters =>
     ensure =>
      - present
      + absent
     explanation =>
      - Use journalbeat packages from Software Heritage
     originator =>
      - softwareheritage
      +_
     packages =>
      - ["journalbeat"]
      + *
     priority =>
      - 990
      + 0
*******************************************
+ Apt::Setting[list-elastic-6.x] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\n# elastic-6.x\nd...
     "ensure": "absent",
     "notify_update": true,
     "priority": 50
*******************************************
+ Apt::Setting[list-elasticsearch] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\n# elasticsearch\...
     "ensure": "present",
     "notify_update": true,
     "priority": 50
*******************************************
+ Apt::Setting[pref-journalbeat] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\nExplanation: Use...
     "ensure": "present",
     "notify_update": false,
     "priority": 50
*******************************************
  Apt::Setting[pref-swh-journalbeat] =>
   parameters =>
     content =>
      @@ -1,5 +1,5 @@
       # This file is managed by Puppet. DO NOT EDIT.
      -Explanation: Use journalbeat packages from Software Heritage
      -Package: journalbeat
      -Pin: release o=softwareheritage
      -Pin-Priority: 990
      +Explanation: profile: swh-journalbeat
      +Package: *
      +Pin: release a=swh-journalbeat
      +Pin-Priority: 0
     ensure =>
      - present
      + absent
*******************************************
+ Apt::Source[elastic-6.x] =>
   parameters =>
     "allow_unsigned": false,
     "comment": "elastic-6.x",
     "ensure": "absent",
     "include": {
     },
     "notify_update": true,
     "repos": "main"
*******************************************
+ Apt::Source[elasticsearch] =>
   parameters =>
     "allow_unsigned": false,
     "comment": "elasticsearch",
     "ensure": "present",
     "include": {
     },
     "key": {
       "id": "46095ACC8548582C1A2699A9D27D666CD88E42B4",
       "content": "-----BEGIN PGP PUBLIC KEY BLOCK-----\nVersion: GnuPG v2.0.14 (...
     },
     "location": "https://artifacts.elastic.co/packages/7.x/apt",
     "notify_update": true,
     "release": "stable",
     "repos": "main"
*******************************************
+ Apt_key[Add key: 46095ACC8548582C1A2699A9D27D666CD88E42B4 from Apt::Source elasticsearch] =>
   parameters =>
     "content": "-----BEGIN PGP PUBLIC KEY BLOCK-----\nVersion: GnuPG v2.0.14 (GN...
     "ensure": "present",
     "id": "46095ACC8548582C1A2699A9D27D666CD88E42B4",
     "refresh": false,
     "server": "keyserver.ubuntu.com",
     "weak_ssl": false
*******************************************
+ File[/etc/apt/preferences.d/journalbeat.pref] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\nExplanation: Use...
     "ensure": "present",
     "group": "root",
     "mode": "0644",
     "owner": "root"
*******************************************
  File[/etc/apt/preferences.d/swh-journalbeat.pref] =>
   parameters =>
     ensure =>
      - present
      + absent
*******************************************
+ File[/etc/apt/sources.list.d/elastic-6.x.list] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\n# elastic-6.x\nd...
     "ensure": "absent",
     "group": "root",
     "mode": "0644",
     "notify": "Class[Apt::Update]",
     "owner": "root"
*******************************************
+ File[/etc/apt/sources.list.d/elasticsearch.list] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\n# elasticsearch\...
     "ensure": "present",
     "group": "root",
     "mode": "0644",
     "notify": "Class[Apt::Update]",
     "owner": "root"
*******************************************
  File[/etc/journalbeat/journalbeat.yml] =>
   parameters =>
     content =>
      @@ -2,4 +2,10 @@
      _
       journalbeat:
      +  inputs:
      +    # Paths that should be crawled and fetched. Possible values files and directories.
      +    # When setting a directory, all journals under it are merged.
      +    # When empty starts to read from local journal.
      +    - paths: []
      +
         # What position in journald to seek to at start up
         # options: cursor, tail, head (defaults to tail)
*******************************************
- File[/etc/journalbeat]
*******************************************
+ File[/etc/systemd/system/journalbeat.service.d/journalbeat.conf] =>
   parameters =>
     "content": "# Managed by puppet (class profile::systemd_journal::journalbeat...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Class[Systemd::Systemctl::Daemon_reload]"
     ],
     "owner": "root",
     "selinux_ignore_defaults": false,
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/journalbeat.service.d] =>
   parameters =>
     "ensure": "directory",
     "group": "root",
     "owner": "root",
     "purge": true,
     "recurse": true,
     "selinux_ignore_defaults": false
*******************************************
  File[/etc/systemd/system/journalbeat.service] =>
   parameters =>
     ensure =>
      - file
      + absent
*******************************************
  Package[journalbeat] =>
   parameters =>
     ensure =>
      - present
      + 7.15.1
*******************************************
  Service[journalbeat] =>
   parameters =>
     subscribe =>
      + ["File[/etc/journalbeat/journalbeat.yml]", "Package[journalbeat]", "Systemd::Dropin_file[journalbeat.conf]"]
*******************************************
+ Systemd::Dropin_file[journalbeat.conf] =>
   parameters =>
     "content": "# Managed by puppet (class profile::systemd_journal::journalbeat...
     "daemon_reload": "lazy",
     "ensure": "present",
     "filename": "journalbeat.conf",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Service[journalbeat]"
     ],
     "owner": "root",
     "path": "/etc/systemd/system",
     "selinux_ignore_defaults": false,
     "show_diff": true,
     "unit": "journalbeat.service"
*******************************************
- Systemd::Unit_file[journalbeat.service]
*******************************************
*** End octocatalog-diff on worker01.softwareheritage.org

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

keeping this diff as planned changes as it seems there is a conflict with the elasticsearch version for the staging and swh-search elastic nodes (redeclare the elastic::elk_version property for swh-search)

*** Running octocatalog-diff on host search-esnode4.internal.softwareheritage.org
I, [2021-11-10T12:54:23.710611 #16530]  INFO -- : Catalogs compiled for search-esnode4.internal.softwareheritage.org
I, [2021-11-10T12:54:23.998043 #16530]  INFO -- : Diffs computed for search-esnode4.internal.softwareheritage.org
diff origin/production/search-esnode4.internal.softwareheritage.org current/search-esnode4.internal.softwareheritage.org
*******************************************
+ Apt::Pin[journalbeat] =>
   parameters =>
     "codename": "",
     "component": "",
     "ensure": "present",
     "explanation": "Use the elk stack version",
     "label": "",
     "order": 50,
     "origin": "",
     "originator": "",
     "packages": [
       "journalbeat"
     ],
     "priority": 1001,
     "release": "",
     "release_version": "",
     "version": "7.9.3"    <--------------- Not good
*******************************************
  Apt::Pin[swh-journalbeat] =>
   parameters =>
     ensure =>
      - present
      + absent
     explanation =>
      - Use journalbeat packages from Software Heritage
     originator =>
      - softwareheritage
      +_
     packages =>
      - ["journalbeat"]
      + *
     priority =>
      - 990
      + 0
*******************************************
+ Apt::Setting[pref-journalbeat] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\nExplanation: Use...
     "ensure": "present",
     "notify_update": false,
     "priority": 50
*******************************************
  Apt::Setting[pref-swh-journalbeat] =>
   parameters =>
     content =>
      @@ -1,5 +1,5 @@
       # This file is managed by Puppet. DO NOT EDIT.
      -Explanation: Use journalbeat packages from Software Heritage
      -Package: journalbeat
      -Pin: release o=softwareheritage
      -Pin-Priority: 990
      +Explanation: profile: swh-journalbeat
      +Package: *
      +Pin: release a=swh-journalbeat
      +Pin-Priority: 0
     ensure =>
      - present
      + absent
*******************************************
+ File[/etc/apt/preferences.d/journalbeat.pref] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\nExplanation: Use...
     "ensure": "present",
     "group": "root",
     "mode": "0644",
     "owner": "root"
*******************************************
  File[/etc/apt/preferences.d/swh-journalbeat.pref] =>
   parameters =>
     ensure =>
      - present
      + absent
*******************************************
  File[/etc/journalbeat/journalbeat.yml] =>
   parameters =>
     content =>
      @@ -2,4 +2,10 @@
      _
       journalbeat:
      +  inputs:
      +    # Paths that should be crawled and fetched. Possible values files and directories.
      +    # When setting a directory, all journals under it are merged.
      +    # When empty starts to read from local journal.
      +    - paths: []
      +
         # What position in journald to seek to at start up
         # options: cursor, tail, head (defaults to tail)
*******************************************
- File[/etc/journalbeat]
*******************************************
+ File[/etc/systemd/system/journalbeat.service.d/journalbeat.conf] =>
   parameters =>
     "content": "# Managed by puppet (class profile::systemd_journal::journalbeat...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Class[Systemd::Systemctl::Daemon_reload]"
     ],
     "owner": "root",
     "selinux_ignore_defaults": false,
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/journalbeat.service.d] =>
   parameters =>
     "ensure": "directory",
     "group": "root",
     "owner": "root",
     "purge": true,
     "recurse": true,
     "selinux_ignore_defaults": false
*******************************************
  File[/etc/systemd/system/journalbeat.service] =>
   parameters =>
     ensure =>
      - file
      + absent
*******************************************
  Package[journalbeat] =>
   parameters =>
     ensure =>
      - present
      + 7.9.3
*******************************************
  Service[journalbeat] =>
   parameters =>
     subscribe =>
      + ["File[/etc/journalbeat/journalbeat.yml]", "Package[journalbeat]", "Systemd::Dropin_file[journalbeat.conf]"]
*******************************************
+ Systemd::Dropin_file[journalbeat.conf] =>
   parameters =>
     "content": "# Managed by puppet (class profile::systemd_journal::journalbeat...
     "daemon_reload": "lazy",
     "ensure": "present",
     "filename": "journalbeat.conf",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Service[journalbeat]"
     ],
     "owner": "root",
     "path": "/etc/systemd/system",
     "selinux_ignore_defaults": false,
     "show_diff": true,
     "unit": "journalbeat.service"
*******************************************
- Systemd::Unit_file[journalbeat.service]
*******************************************
*** End octocatalog-diff on search-esnode4.internal.softwareheritage.org

Allow to override the beat version

Thanks for working on this!

I have a few questions:

  • Do we really care about running as a separate user? Maybe avoiding a gratuitous divergence from upstream would be worth it.
  • Could you fully update the config template to the new default config?
  • Does the icinga check for the journalbeat position still work? I assume so, as the config for it is still there.
  • Should we consider moving to cursor_seek_fallback: tail to work around the issues with the old indexes needing to be reopened, when rebooting a machine? This should not apply as the cursor state should be saved on service shutdown, but maybe it isn't being read back properly...
site-modules/profile/manifests/systemd_journal/journalbeat.pp
18
  • Do we really care about running as a separate user? Maybe avoiding a gratuitous divergence from upstream would be worth it.

Nope, I keep it to remains as close as what we have today, I can let it per default and add the cleanup of the user previously created

  • Could you fully update the config template to the new default config?

Yep, I will try to automize that

  • Does the icinga check for the journalbeat position still work? I assume so, as the config for it is still there.

I think so too but I will double check to be sure

  • Should we consider moving to cursor_seek_fallback: tail to work around the issues with the old indexes needing to be reopened, when rebooting a machine? This should not apply as the cursor state should be saved on service shutdown, but maybe it isn't being read back properly...

I will also try this because the upgrade changed nothing, the current behavior is reproduced locally with the new version when I restart the vms

  • Does the icinga check for the journalbeat position still work? I assume so, as the config for it is still there.

I think so too but I will double check to be sure

good point, it seems it's not working anymore, the new version of logstash is silently ignoring the property

The cursor state is now store in the /var/lib/journalbeat/registry file and the format has changed:

journal_entries:
- path: LOCAL_SYSTEM_JOURNAL
  cursor: s=fbf433e3151442d48a41f6bbe7178f76;i=7caea;b=a12f885575a64123a63b824ecbb3848c;m=1f56598ac;t=5d0e54cb8961a;x=a04cf1cd032d5b03
  realtime_timestamp: 1637058136806938
  monotonic_timestamp: 8412043436

compared to the previous /var/log/journalbeat/cursor-state:

vsellier@logstash0 ~ % sudo cat /var/lib/journalbeat/cursor-state
s=b3186d31ac0b4c0aa8a11239fb635349;i=27764513;b=7c201d02af8f41d3ae4f0462c271b721;m=8e01a7abbf9;t=5d0e565142685;x=9d6c9621135a86d%

The check_journal script need to be updated too.

Hopefully, the format of the cursor didn't changed:

root@logstash0:/var/lib/journalbeat# grep cursor registry | awk '{print $2}' > cursor-state
root@logstash0:/var/lib/journalbeat# /usr/lib/nagios/plugins/swh/check_journal
JOURNAL LAG OK - 0s behind, 0 entries behind | entries_lag=0 monotonic_lag=0.0s;1200;3600 realtime_lag=0.0s;1200;3600

Improve the check_journal script to check the new registry file
and fallback to the old cursor_state file if not found

diff updates:

  • remove the journal beat user
  • cleanup the cursor_state file

diff update:

  • upgrade to 7.15.2
  • automatically manage the journalbeat index template from the logstash server
ardumont added a subscriber: ardumont.

lgtm

couple of absolutely non-blocking remarks inline.

site-modules/profile/files/icinga2/plugins/check_journal
96

The initial version is fine for raw print statements, it's better for log instruction to do the formatting if need be otherwise.

105

for your general information (not necessary to change), let the formatting be done by the log instruction.

site-modules/profile/files/journalbeat/manage_index_template.sh
16
This revision is now accepted and ready to land.Wed, Nov 17, 10:11 AM
  • update the log formating in python script
  • fix a couple of typos
This revision was automatically updated to reflect the committed changes.