Page MenuHomeSoftware Heritage

Update the journalbeat version package
Closed, ResolvedPublic

Description

We've been hitting by regular out-of-sync journalbeat events.
Too old events that want to get written again on closed indices.
When logstash tries to write those into closed and/or freezed indices.
that creates errors.

So far, we work around it by triggering the manually installed script [1].
It parses error logs and opens up and unfreezes those necessary closed indices.

That allows logstash to do thy bidding.
And other processes are in charge of closing those indices again.

It'd be interesting to update the journalbeat to a superior version than
the current installed one which is old. At least, to check whether that spurious
behavior continues.

We possibly want to upgrade with upstream versions as we are currently using an old
5.5 version we packaged or backported back in the day [3]

[1]

root@logstash0:~# /usr/local/bin/es_open_unfreeze_from_journalctl.sh

[2] P1135

[3]

root@logstash0:~# dpkg -l journalbeat | grep ii
ii  journalbeat    5.5.0+git20170727.1-1~swh+1~bpo10+1 amd64        Journalbeat is a log shipper from systemd/journald to Logstash/Elasticsearch

[3] See comments below with some more details T3545#69778

Event Timeline

ardumont triaged this task as Normal priority.Sep 2 2021, 12:29 PM
ardumont created this task.
17:17 <+ardumont> just out of curiosity, not that i'm on that right now
17:20 <+ardumont> on an elastic node, we'd get
17:20 <+ardumont> ardumont@esnode1:~% apt-cache show journalbeat | grep Version | awk '{print $2}' | sort -V -r | head -1
17:20 <+ardumont> 7.14.1
17:20 <+ardumont> we'd have some choice
17:22 <+olasd> ardumont: no objections. the main concern is that the upstream mapping from journalctl fields to elasticsearch documents has changed, so we'd need to adapt the (overall, very few) filters we have
17:22 <+olasd> and probably some dashboard churn

I'm referring to the setup described in their docs [1]
And my command line got executed in one of our nodes having said setup.

[1] https://www.elastic.co/guide/en/beats/journalbeat/current/setup-repositories.html

ardumont updated the task description. (Show Details)
vsellier claimed this task.
vsellier added a subscriber: vsellier.

Package upgraded in T3705.

We have restarted most the infra with the upgrades to bullseye. I have the feeling the problem occurs less often.