ELK platform provides great solution aggregating and indexing various log, events within a organization. But you may not want to keep old data in Elasticsearch forever.
Also see ELK installation and configuration
To delete old data you can use “elasticsearch-curator” tool
You can simply install it as
$pip install elasticsearch-curator
Followed by some basic configuration.
create config.yml and action.yml as following
config.yml
client:
hosts:
- 127.0.0.1
port: 9200
logging:
loglevel: INFO
logfile: "/var/log/curator/actions.log"
logformat: default
blacklist: ['elasticsearch', 'urllib3']
create log directory /var/log/curator/
mkdir /var/log/curator/
action.yml – change days according to your requirement.
actions:
1:
action: delete_indices
description: >-
Delete indices older than 10 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
timeout_override:
continue_if_exception: False
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: logstash-
exclude:
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 10
exclude:
Try a dry run and check log file /var/log/curator/actions.log
/usr/local/bin/curator –config /opt/sw/curator/config.yml –dry-run /opt/sw/curator/action.yml
Once you are convinced with the logs then you can setup cronjob to auto delete old data.
0 0 * * * /usr/local/bin/curator –config /opt/sw/curator/config.yml /opt/sw/curator/action.yml