Quick and easy ELK – elasticsearch, logstash, kibana for syslog and SNMP traps

Share

ELK stack is quite capable solution for event, logs, data aggregation and parsing. It offers a very shiny yet highly flexible web frontend. You can extend it to limits you can think off.

So what is ELK exactly ? ELK is composed of three independent components.

Logstash: A very comprehensive and event collector and parser which works very well for syslog, SNMP and anything that crawls in computers.

Elasticsearch: A search engine based on Lucene. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.

Kibana: an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data.

Start building ELK stack

We will use following versions on Debian jessie

Kibana 4.3.0
logstash 2.0.0
elasticsearch 2.4.5

Install Java

aptitude install openjdk-7-jre

Install SMItools for SNMP traps

aptitude install smitools

Elasticsearch

wget -qO – https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –

echo “deb http://packages.elastic.co/elasticsearch/2.x/debian stable main” | sudo tee -a /etc/apt/sources.list.d/Elasticsearch-2.x.list

sudo aptitude update

sudo aptitude install elasticsearch

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

Upon starting you will see port 9200 in listen state

root@mka:~# netstat -nap | grep 9200
tcp6 0 0 ::1:9200 :::* LISTEN 9208/java
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 9208/java

You can tune elasticsearch config by editing /etc/elasticsearch/elasticsearch.yml
Though default config works well if all ELK components are residing on same server.
You may want to change cluster name

cluster.name: myelk

Logstash

Add Elastic.co logstash repository to debian /etc/apt/sources.list.d/.

echo “deb http://packages.elastic.co/logstash/2.0/debian stable main” | sudo tee -a /etc/apt/sources.list

sudo aptitude update

sudo aptitude install logstash

sudo systemctl daemon-reload
sudo systemctl enable logstash.service

Configuring Logstash to receive syslog events and SNMP traps

After trying many permutation combination, I have narrowed down to following configs to get syslog and SNMP together in ELK stack.
I had trouble in getting SNMP trap events because they have “.” in many fields. To mitigate it we are replacing “.” with “_” in SNMP filter section.

See https://discuss.elastic.co/t/cannot-see-snmp-trap-message-in-kibana/88833

Syslog:

root@mka:~# cat /etc/logstash/conf.d/logstash-syslog.conf
input {
tcp {
port => 514
type => syslog
}
udp {
port => 514
type => syslog
}
}

filter {
if [type] == “syslog” {
grok {
match => { “message” => “%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}” }
add_field => [ “received_from”, “%{host}” ]
}
}
}

output {
elasticsearch { hosts => [“127.0.01:9200”] }
stdout { codec => rubydebug }
}

SNMP traps:

root@mka:~# cat /etc/logstash/conf.d/logstash-snmp.conf
input {
snmptrap {
type => “snmptrap”
host => “0.0.0.0”
port => 162
yamlmibdir => “/opt/logstash/vendor/bundle/jruby/1.9/gems/snmp-1.2.0/data/ruby/snmp/mibs”
}
}

filter{
if [type] == “snmptrap” {
ruby {
code => “
event.to_hash.keys.each { |k| event[ k.gsub(‘.’,’_’) ] = event.remove(k) if k.include?’.’ }

}
}
}

output {
elasticsearch { hosts => [“127.0.0.1:9200”] }
stdout { codec => rubydebug }
}

Now start logstash

sudo systemctl start logstash.service

Or use following command

/opt/logstash/bin/logstash –verbose -f /etc/logstash/conf.d/ -l /var/log/logstash/logstash.log &

Upon starting logstash you will see ports 514(udp and tcp) and 162(udp) open in netstat output

root@mka:~# netstat -nap | grep 514
tcp6 0 0 :::514 :::* LISTEN 22572/java
udp6 0 0 :::514 :::* 22572/java

root@mka:~# netstat -nap | grep 162
udp6 0 0 :::162 :::* 22572/java

Kibana

We will install kibana in /opt directory

cd /opt
wget -qO – https://download.elastic.co/kibana/kibana/kibana-4.3.0-linux-x64.tar.gz | sudo tar -xzf
sudo useradd kibana
sudo chown kibana:kibana kibana-4.3.0-linux-x64/ -R
sudo touch /usr/lib/systemd/system/kibana.service

vi /usr/lib/systemd/system/kibana.service
[Unit]
Description=kibana
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target
[Service]
User=kibana
Group=kibana
ExecStart=/opt/kibana-4.3.0-linux-x64/bin/kibana
Restart=always
StandardOutput=null
# Connects standard error to journal
StandardError=journal
[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable kibana.service
sudo systemctl start kibana.service

Or use following command

/opt/kibana-4.3.0-linux-x64/bin/kibana -e http://localhost:9200 –verbose -l /var/log/kibana/kibana.log &

When you start kibana you will see port 5601 open in netstat

root@mka:~# netstat -nap | grep 5601
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 1205/node

Point your web browser to http://<your_ip>:5601 and navigate to settings

Here select default logstash-* pattern and time field “@timestamp” then create index. Now navigate to discover tab to watch events in Kibana.

And after all hard work you get following on platter