This installs the cron on the logstash instance.
Related to T3221
Differential D5483
Puppetize the elasticsearch closing index script Authored by ardumont on Apr 12 2021, 1:35 PM.
Details
This installs the cron on the logstash instance. Related to T3221 octocatalog, vagrant happy: bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging logstash0
Found host logstash0.internal.softwareheritage.org
...
*** Running octocatalog-diff on host logstash0.internal.softwareheritage.org
I, [2021-04-12T13:34:31.298208 #12191] INFO -- : Catalogs compiled for logstash0.internal.softwareheritage.org
I, [2021-04-12T13:34:32.433237 #12191] INFO -- : Diffs computed for logstash0.internal.softwareheritage.org
diff origin/production/logstash0.internal.softwareheritage.org current/logstash0.internal.softwareheritage.org
*******************************************
+ Concat_file[profile::cron::elasticsearch] =>
parameters =>
"group": "root"
"mode": "0644"
"owner": "root"
"path": "/etc/puppet-cron.d/elasticsearch"
"tag": "profile::cron::elasticsearch"
*******************************************
+ Concat_fragment[profile::cron::elasticsearch-close-index] =>
parameters =>
"order": "10"
"tag": "profile::cron::elasticsearch"
"target": "profile::cron::elasticsearch"
"content": >>>
# Cron snippet elasticsearch-close-index
30 5 * * * root /usr/local/bin/elasticsearch_close_index.py --host esnode2.internal.softwareheritage.org --host esnode3.internal.softwareheritage.org --host esnode1.internal.softwareheritage.org --timeout 1200
<<<
*******************************************
+ Concat_fragment[profile::cron::elasticsearch::_header] =>
parameters =>
"order": "00"
"tag": "profile::cron::elasticsearch"
"target": "profile::cron::elasticsearch"
"content": >>>
# Managed by puppet (module profile::cron), manual changes will be lost
<<<
*******************************************
+ File[/etc/cron.d/puppet-elasticsearch] =>
parameters =>
"ensure": "link"
"target": "/etc/puppet-cron.d/elasticsearch"
*******************************************
+ File[/usr/local/bin/elasticsearch_close_index.py] =>
parameters =>
"ensure": "present"
"group": "root"
"mode": "0755"
"owner": "root"
"content": >>>
#!/usr/bin/env python3
import click
import datetime
import iso8601
import elasticsearch
@click.command()
@click.option('--host', '-h',
default='esnode1.internal.softwareheritage.org',
multiple=True,
help="Elasticsearch node instances")
@click.option('--timeout', '-t', default=1200)
def main(host, timeout):
hosts = host # `host` is a list of multiple hosts
today = datetime.date.today()
days = lambda n: datetime.timedelta(days=n)
es = elasticsearch.Elasticsearch(hosts=hosts, timeout=timeout)
for l in sorted(es.cat.indices(h='i,sth,status').splitlines()):
i, throttled, status = l.split()
throttled = throttled == 'true'
if throttled and status != 'open':
continue
if i.startswith('.'):
continue
date = i.split('-')[-1]
if not date.startswith('20'):
continue
date = date.replace('.', '-')
date = iso8601.parse_date(date).date()
info = es.indices.get(i)[i]
shards = int(info['settings']['index']['number_of_shards'])
if not throttled and date < today - days(7):
print('freezing', i)
es.indices.freeze(i, wait_for_active_shards=shards)
status = 'open'
if status == 'open' and date < today - days(30):
print('closing', i)
es.indices.close(i)
if __name__ == '__main__':
main()
<<<
*******************************************
+ Package[python3-click] =>
parameters =>
"ensure": "present"
*******************************************
+ Package[python3-elasticsearch] =>
parameters =>
"ensure": "present"
*******************************************
+ Package[python3-iso8601] =>
parameters =>
"ensure": "present"
*******************************************
+ Profile::Cron::D[elasticsearch-close-index] =>
parameters =>
"command": "/usr/local/bin/elasticsearch_close_index.py --host esnode2.internal.softwareheritage.org --host esnode3.internal.softwareheritage.org --host esnode1.internal.softwareheritage.org --timeout 1200"
"month": "fqdn_rand"
"target": "elasticsearch"
"unique_tag": "elasticsearch-close-index"
"user": "root"
*******************************************
+ Profile::Cron::File[elasticsearch] =>
parameters =>
"target": "elasticsearch"
*******************************************
*** End octocatalog-diff on logstash0.internal.softwareheritage.org
Diff Detail
Event Timeline
Comment Actions Adapt according to suggestions
Comment Actions I'm sure some of the assumptions behind this script are going to break, eventually. But for now it's better than nothing!
| ||||||||||||||||||||||||||||||||||||||||||||