The check is now active.
An alert will be raised by icinga if :
update the commit message
fix indentation
According to the API (TIL the catalog can be requested like that), journal0 doesn't have the new plugins declared. So the check should be disabled as the filter is using this field
The new probe is deployed but nothing is displayed in icinga. Let's start a configuration debug session.
network schema: reactivate the description of the firewalls' group
after searching how it can be integrated with the inciga checks, the simplest way I have found is to create a script that periodically query logstash to get the statistics and return this status in this cases:
I have simulated different situations locally on the vagrant environment:
root@logstash0:~# curl -s http://localhost:9600/_node/stats/pipelines | jq '.pipelines.main.plugins.outputs'
[
{
"id": "c49a6902391a456022af4c89f0972781900d01d70cd5f312b292cb20c0d345eb",
"documents": {
"non_retryable_failures": 112,
"successes": 103692
},
"events": {
"out": 103804,
"in": 103804,
"duration_in_millis": 3529049
},
"name": "elasticsearch",
"bulk_requests": {
"responses": {
"200": 2028
},
"failures": 3,
"with_errors": 110,
"successes": 1918
}
}
]upgrade done without any problem:
Actions performed:
The checks ran without detecting any bad block on the disk.
They can be added on the zfs pool again.
changelog for the 21.1.5 version
Good day everyone,
rebase
And I think that I found how to fix the last remaining warning cited above.
good news ;)
thanks
completely remove the -e option
@anlambert I didn't succeed to work with link to the modules, but I found another way with a standard installation (i.e without the -e flag) we can force with a flag on the ci (cf D5681).
I really not sure if it's the good approach or not.
A full badblock test is launched on both disks:
root@storage1:~# badblocks -v -w -B -s -b 4096 /dev/sdb
root@storage1:~# badblocks -v -w -B -s -b 4096 /dev/sdc
thanks a lot @anlambert, I will look in that direction
Aparrently, the modules are well installed but the sphinx-apidoc is not detecting them due to the local installation
jenkins@4e5220b923d8:~/workspace/swh-environment/swh-docs$ .tox/sphinx-dev/bin/python3 -m pip list | grep swh swh.auth 0.5.4 /home/jenkins/workspace/swh-environment/swh-auth swh.core 0.13.2.dev1+g7d42035 /home/jenkins/workspace/swh-environment/swh-core swh.counters 0.7.1.dev1+g6a44a84 /home/jenkins/workspace/swh-environment/swh-counters swh.deposit 0.13.6 /home/jenkins/workspace/swh-environment/swh-deposit swh.docs 0.0.1.dev334+g044cb9b.d20210505 swh.fuse 1.0.3 /home/jenkins/workspace/swh-environment/swh-fuse swh.graph 0.3.2.dev3+g62c2fd3 /home/jenkins/workspace/swh-environment/swh-graph swh.icinga-plugins 0.3.1.dev1+g8878925 /home/jenkins/workspace/swh-environment/swh-icinga-plugins swh.indexer 0.7.1.dev4+g8f1fb0f /home/jenkins/workspace/swh-environment/swh-indexer swh.journal 0.7.2.dev8+g2972c7a /home/jenkins/workspace/swh-environment/swh-journal swh.lister 1.1.0 /home/jenkins/workspace/swh-environment/swh-lister swh.loader.core 0.22.1.dev2+g0e4bb4b /home/jenkins/workspace/swh-environment/swh-loader-core swh.loader.git 0.9.2.dev1+g15e12fa /home/jenkins/workspace/swh-environment/swh-loader-git swh.loader.mercurial 0.5.1.dev4+g8884714 /home/jenkins/workspace/swh-environment/swh-loader-mercurial swh.loader.svn 0.7.1 /home/jenkins/workspace/swh-environment/swh-loader-svn swh.model 2.4.2.dev1+gdf036ef /home/jenkins/workspace/swh-environment/swh-model swh.objstorage 0.2.3 /home/jenkins/workspace/swh-environment/swh-objstorage swh.objstorage.replayer 0.2.2 /home/jenkins/workspace/swh-environment/swh-objstorage-replayer swh.scanner 0.4.2.dev1+g30b40cc /home/jenkins/workspace/swh-environment/swh-scanner swh.scheduler 0.13.1.dev5+gbab557e /home/jenkins/workspace/swh-environment/swh-scheduler swh.search 0.8.1 /home/jenkins/workspace/swh-environment/swh-search swh.storage 0.27.5.dev4+g051b7715 /home/jenkins/workspace/swh-environment/swh-storage swh.vault 0.5.2.dev2+gf87dd54 /home/jenkins/workspace/swh-environment/swh-vault swh.web 0.0.307.dev3+gf8c750b6 /home/jenkins/workspace/swh-environment/swh-web swh.web.client 0.3.1.dev1+g4b610ad /home/jenkins/workspace/swh-environment/swh-web-client
sphinx-apidoc \ --implicit-namespaces \ --templatedir=../swh/docs/templates/ \ --maxdepth=3 \ --ext-viewcode --separate \ -o apidoc \ /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh \ /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/tests /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/tests/* /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/*/tests/* /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/*/*/tests/* /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/migrations /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/migrations/* /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/*/migrations/* /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/*/*/migrations/* /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/wsgi.py /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/*/wsgi.py /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/*/*/wsgi.py /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/*/*/wsgi.py /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/deposit/settings/* /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/web/settings/* /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/dataset/* Creating file apidoc/swh.rst. Creating file apidoc/swh.docs.rst. Creating file apidoc/swh.docs.django_settings.rst. Creating file apidoc/swh.docs.sphinx.rst. Creating file apidoc/swh.docs.sphinx.conf.rst. Creating file apidoc/swh.docs.sphinx.view_in_phabricator.rst. Creating file apidoc/modules.rst.
The disk were replaced by Christophe.
Apparently, the led of one of the disk is still on, so they need to be switched off:
root@storage1:~# ls /dev/sd* | grep -e "[a-z]$" | xargs -n1 -t -i{} ledctl normal={}
ledctl normal=/dev/sda
ledctl normal=/dev/sdb
ledctl normal=/dev/sdc
ledctl normal=/dev/sdd
ledctl normal=/dev/sde
ledctl normal=/dev/sdf
ledctl normal=/dev/sdg
ledctl normal=/dev/sdh
ledctl normal=/dev/sdi
ledctl normal=/dev/sdj
ledctl normal=/dev/sdk
ledctl normal=/dev/sdl
ledctl normal=/dev/sdm
ledctl normal=/dev/sdnnew permissions updated by puppet
fix a typo on the commit message
The replacement disks were delivered at rocquencourt :
The command seems to be delivered, I will check with the DSI how we can proceed for the installation
logstash now exposes an api server[1] which seems to return some interesting metrics on the plugin behaviors.
For example, there is a section for the elasticsearch output plugin:
"outputs": [
{
"id": "62d11c4234b8981da77a97955da92ac9de92b9a6dcd4582f407face31fd5c664",
"events": {
"duration_in_millis": 160089636,
"in": 72818126,
"out": 72818046
},
"bulk_requests": {
"responses": {
"200": 3860888
},
"successes": 3860888
},
"documents": {
"successes": 72818046
},
"name": "elasticsearch"
}
]
},I'll try to implement a small python script checking if there is other response code than 200 in a first time to identify the behavior
Perhaps it will be also interesting to check other properties like queue size :
"queue": {
"type": "memory",
"events_count": 0,
"queue_size_in_bytes": 0,
"max_queue_size_in_bytes": 0
},I checked the icinga_logstash plugin[1] to see if it can be helpful but it's more oriented to logastash instances used to ingest data from log files. There is no options to check the number of events received/sent for example.
According to the tracking page, the command has left the factory the Apr 22, 2021, The ETA is May 28, 2021*.
and the authors are now displayed on staging and production (webapp1)
The lag for the production can be followed here: https://grafana.softwareheritage.org/goto/Di2H3z9Gk
(staging has already recovered)
the swh-counters is deployed in production too:
root@counters1:~# apt dist-upgrade ... Setting up python3-swh.counters (0.7.0-1~swh1~bpo10+1) ... root@counters1:~# systemctl stop swh-counters-journal-client.service root@counters1:~# systemctl restart gunicorn-swh-counters.service root@counters1:~# systemctl start swh-counters-journal-client.service root@counters1:~# redis-cli pfcount person (integer) 7
The count of the person already starts
root@counters1:~# systemctl stop swh-counters-journal-client.service
vsellier@kafka1 ~ % /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets --all-topics --to-current --dry-run --export --group swh.counters.journal_client 2>&1 > ~/counters_journal_client_offsets.csv # revision reset vsellier@kafka1 ~ % /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets --group swh.counters.journal_client --to-earliest --execute --topic swh.journal.objects.revision # release reset vsellier@kafka1 ~ % /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets --group swh.counters.journal_client --to-earliest --execute --topic swh.journal.objects.release # checks /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets --all-topics --to-current --dry-run --export --group swh.counters.journal_client 2>&1 > ~/counters_journal_client_offsets-backfill.csv vsellier@kafka1 ~ % diff ~/counters_journal_client_offsets.csv ~/counters_journal_client_offsets-backfill.csv | less 1c1 < "swh.journal.objects.revision",25,8275180 --- > "swh.journal.objects.revision",25,0 8c8 < "swh.journal.objects.release",128,78484 --- > "swh.journal.objects.release",128,0 16c16 ...
root@counters1:~# systemctl start swh-counters-journal-client.service
root@counters1:~# date;redis-cli pfcount person Fri 23 Apr 2021 10:55:54 AM UTC (integer) 72358 root@counters1:~# date;redis-cli pfcount person Fri 23 Apr 2021 10:55:57 AM UTC (integer) 80618