Managing old data in logstash

Share

ELK platform provides great solution aggregating and indexing various log, events within a organization. But you may not want to keep old data in logstash forever.

Also see ELK installation and configuration

To delete old data you can use “elasticsearch-curator” tool

You can simply install it as

$pip install elasticsearch-curator

Followed by some basic configuration.

create config.yml and action.yml as following

config.yml


client:
hosts:
- 127.0.0.1
port: 9200
logging:
loglevel: INFO
logfile: "/var/log/curator/actions.log"
logformat: default
blacklist: ['elasticsearch', 'urllib3']

create log directory /var/log/curator/
mkdir /var/log/curator/

action.yml – change days according to your requirement.


actions:
1:
action: delete_indices
description: >-
Delete indices older than 10 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
timeout_override:
continue_if_exception: False
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: logstash-
exclude:
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 10
exclude:

Try a dry run and check log file /var/log/curator/actions.log

/usr/local/bin/curator –config /opt/sw/curator/config.yml –dry-run /opt/sw/curator/action.yml

Once you are convinced with the logs then you can setup cronjob to auto delete old data.

0 0 * * * /usr/local/bin/curator –config /opt/sw/curator/config.yml /opt/sw/curator/action.yml