Elasticsearch
Apache access and error logs in Fluentd
To parse and add Apache2 access and error logs add this section to your /etc/td-agent/td-agent.conf Fluend config file Restart td-agent Now, access apache web pages and you will see logs going into elastic database. In Kibana management, create new index pattern like apache-* to visualize apache logs in kibana
Migrate elasticsearch data from one cluster to another
Suppose you have an existing elasticsearch cluster A and, for any reason, you want to migrate the data to cluster B.In this write up, My cluster A is on ELK 5.4 and cluster B is on ELK 6.7. This procedure is very handy to upgrade and migrate ELK on from existing to new servers. On […]
Playing with elasticsearch indices
List all indices List indices matching a pattern Delete an index Fetch available fields in an index Search something within an index List all items in an index
Syslog messages in EFK stack
You can use opensource syslog-ng tool to capture syslog messages from various sources, like remote servers, devices, local logs and so on. These logs can then be forwarded to fuentd on EFK stack. Here is a sample config of syslog-ng config to receive logs from devices and hosts on tcp and udp port 514. Received […]
EFK stack – versatile and very capable analytics platform
So far I was happily using ELK stack to feed syslog messages into Elasticsearch. In ELK stack I had used Logstash to aggregate syslogs and feed them into elasticsearch. Recently, I came across fluentd and found it quite interesting and flexible. Using fluentd with Elasticsearch and Kibana I have now build a EFK stack. EFK […]
Secure Elasticsearch and Kibana access using Apache reverse proxy
Default ELK installation may not fulfill enterprise requirements as kibana, the visualizer, works on port 5601 without any authentication and elasticsearch listens only on loopback IP on port 9200 without any authentication. You may want to run kibana on standard http/https port ie 80/443 with authentication control. Similarly, you may want to allow elasticsearch access […]
Delete old data in Elasticsearch
ELK platform provides great solution aggregating and indexing various log, events within a organization. But you may not want to keep old data in Elasticsearch forever. Also see ELK installation and configuration To delete old data you can use “elasticsearch-curator” tool You can simply install it as $pip install elasticsearch-curator Followed by some basic configuration. […]
Syslog and snmp traps in ELK – Taming ELK
ELK stack is quite capable solution for event, logs, data aggregation and parsing. It offers a very shiny yet highly flexible web frontend. You can extend it to limits you can think off. It is a perfect open source tool for end to end data analytics.So what is ELK exactly ? ELK is composed of […]