Page MenuHomeSoftware Heritage

staging: upgrade swh-search elasticsearch to 7.15.2
ClosedPublic

Authored by vsellier on Nov 23 2021, 7:53 PM.

Details

Summary

The elastic_beat_version was keep in place to explicit
the elasticsearch and the beat versions are not correlated
in this server

Related to T3741

Test Plan

It was deployed in vagrant, the origin indexation is still working well.
The indexed content indexation was not tested

diff origin/production/search-esnode0.internal.staging.swh.network current/search-esnode0.internal.staging.swh.network
*******************************************
  Apt::Pin[elasticsearch] =>
   parameters =>
     version =>
      - 7.9.3
      + 7.15.2
*******************************************
  Apt::Setting[pref-elasticsearch] =>
   parameters =>
     content =>
      @@ -2,4 +2,4 @@
       Explanation: profile: elasticsearch
       Package: elasticsearch elasticsearch-oss
      -Pin: version 7.9.3
      +Pin: version 7.15.2
       Pin-Priority: 1001
*******************************************
  Archive[prometheus-elasticsearch-exporter] =>
   parameters =>
     creates =>
      - /usr/share/elasticsearch/plugins/prometheus-exporter/prometheus-exporter-7.9.3.0.jar
      + /usr/share/elasticsearch/plugins/prometheus-exporter/prometheus-exporter-7.15.2.0.jar
     path =>
      - /tmp/prometheus-exporter-7.9.3.0.zip
      + /tmp/prometheus-exporter-7.15.2.0.zip
     source =>
      - https://github.com/vvanholl/elasticsearch-prometheus-exporter/releases/download/7.9.3.0/prometheus-exporter-7.9.3.0.zip
      + https://github.com/vvanholl/elasticsearch-prometheus-exporter/releases/download/7.15.2.0/prometheus-exporter-7.15.2.0.zip
*******************************************
  Exec[cleanup prometheus exporter plugin] =>
   parameters =>
     creates =>
      - /usr/share/elasticsearch/plugins/prometheus-exporter/prometheus-exporter-7.9.3.0.jar
      + /usr/share/elasticsearch/plugins/prometheus-exporter/prometheus-exporter-7.15.2.0.jar
*******************************************
  File[/etc/apt/preferences.d/elasticsearch.pref] =>
   parameters =>
     content =>
      @@ -2,4 +2,4 @@
       Explanation: profile: elasticsearch
       Package: elasticsearch elasticsearch-oss
      -Pin: version 7.9.3
      +Pin: version 7.15.2
       Pin-Priority: 1001
*******************************************
  File[/etc/elasticsearch/elasticsearch.yml] =>
   parameters =>
     content =>
      @@ -12,3 +12,4 @@
       indices.memory.index_buffer_size: 50%
       index.store.type: hybridfs
      +ingest.geoip.downloader.enabled: false
       network.host: 192.168.130.80
*******************************************
  Package[elasticsearch] =>
   parameters =>
     ensure =>
      - 7.9.3
      + 7.15.2
*******************************************
*** End octocatalog-diff on search-esnode0.internal.staging.swh.network

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

This revision is now accepted and ready to land.Nov 24 2021, 9:11 AM

Disable the new embedded geoip database
It creates and regularly update a .geoip_database index
not used by swh_search, so removing it

...
[2021-11-24T09:01:54,160][INFO ][o.e.i.g.GeoIpDownloader  ] [search-esnode0] updating geoip databases
[2021-11-24T09:01:54,161][INFO ][o.e.i.g.GeoIpDownloader  ] [search-esnode0] fetching geoip databases overview from [https://geoip.elastic.co/v1/database?elastic_geoip_service_tos=agree]
[2021-11-24T09:01:54,164][INFO ][o.e.c.r.a.AllocationService] [search-esnode0] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[origin-v0.11][0]]]).
[2021-11-24T09:01:55,076][INFO ][o.e.i.g.GeoIpDownloader  ] [search-esnode0] updating geoip database [GeoLite2-ASN.mmdb]
[2021-11-24T09:01:55,617][INFO ][o.e.c.m.MetadataCreateIndexService] [search-esnode0] [.geoip_databases] creating index, cause [auto(bulk api)], templates [], shards [1]/[0]
[2021-11-24T09:01:56,170][INFO ][o.e.i.g.DatabaseRegistry ] [search-esnode0] downloading geoip database [GeoLite2-ASN.mmdb] to [/tmp/elasticsearch-15515843704193017823/geoip-databases/qHAqt7DFTjO8vF4Nl2ozdw/GeoLite2-ASN.mmdb.tmp.gz]
[2021-11-24T09:01:56,191][INFO ][o.e.i.g.GeoIpDownloader  ] [search-esnode0] updated geoip database [GeoLite2-ASN.mmdb]
[2021-11-24T09:01:56,215][INFO ][o.e.i.g.GeoIpDownloader  ] [search-esnode0] updating geoip database [GeoLite2-City.mmdb]
[2021-11-24T09:01:56,379][INFO ][o.e.i.g.DatabaseRegistry ] [search-esnode0] successfully reloaded changed geoip database file [/tmp/elasticsearch-15515843704193017823/geoip-databases/qHAqt7DFTjO8vF4Nl2ozdw/GeoLite2-ASN.mmdb]
[2021-11-24T09:01:59,034][INFO ][o.e.i.g.DatabaseRegistry ] [search-esnode0] downloading geoip database [GeoLite2-City.mmdb] to [/tmp/elasticsearch-15515843704193017823/geoip-databases/qHAqt7DFTjO8vF4Nl2ozdw/GeoLite2-City.mmdb.tmp.gz]
[2021-11-24T09:01:59,063][INFO ][o.e.i.g.GeoIpDownloader  ] [search-esnode0] updated geoip database [GeoLite2-City.mmdb]
[2021-11-24T09:01:59,066][INFO ][o.e.i.g.GeoIpDownloader  ] [search-esnode0] updating geoip database [GeoLite2-Country.mmdb]
[2021-11-24T09:01:59,661][INFO ][o.e.i.g.DatabaseRegistry ] [search-esnode0] downloading geoip database [GeoLite2-Country.mmdb] to [/tmp/elasticsearch-15515843704193017823/geoip-databases/qHAqt7DFTjO8vF4Nl2ozdw/GeoLite2-Country.mmdb.tmp.gz]
[2021-11-24T09:01:59,698][INFO ][o.e.i.g.GeoIpDownloader  ] [search-esnode0] updated geoip database [GeoLite2-Country.mmdb]
[2021-11-24T09:01:59,780][INFO ][o.e.i.g.DatabaseRegistry ] [search-esnode0] successfully reloaded changed geoip database file [/tmp/elasticsearch-15515843704193017823/geoip-databases/qHAqt7DFTjO8vF4Nl2ozdw/GeoLite2-Country.mmdb]
...
[2021-11-24T09:04:00,612][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [search-esnode0] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-15515843704193017823/geoip-databases/qHAqt7DFTjO8vF4Nl2ozdw/GeoLite2-City.mmdb]
...