Bits & Bytes of Life

A blog of my thoughts and actions.

Installing Logstash on CentOS 5.7

| Comments

What is and Why use Logstash?

At my place of employment we have recently set up Zabbix for our infrastructure monitoring. To complement that system and allow easier and faster diagnosis of problems detected we have looked into building a centralized logging system for our servers and applications. This has been an ongoing search but we never found something that truly fit our needs. Recently, I came across Logstash which uses a program written in JRuby which operates like many Unix style programs. It is described to be similar to sed. Logstash is designed to chain a number of filters together to process an input source and output to many different places. One of them is to Elasticsearch which allows for easy searching, pattern matching and even correlation without needing to dump the entire system in a backend SQL database which is often slow and cumbersome to use on unstructured data like log files are.

This post will be a set of steps to install and configure a Centralized install of Logstash. It will be tested with Apache access logs.

Install Dependencies

1
2
3
4
5

# Install EPEL repo
sudo rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/$(uname -i)/epel-release-5-4.noarch.rpm

# Dependencies
yum install -y gcc gperf make libevent-devel pcre-devel tokyocabinet-devel

Download and Install Java JDK

These instructions were used from BANYM’S BLOG Use your favorite browser to download the JDK in .rpm.bin format from Oracle: http://www.oracle.com/technetwork/java/javase/downloads/index.html

1
2
3
4
5
6

# Install Java
sh jdk*rpm.bin* -noregister

# Link the JDK into your environment:
alternatives --install /usr/bin/java java /usr/java/default/bin/java 1
alternatives --config java

Use the Sources!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

# Download Elastic Search
ES_PACKAGE=elasticsearch-0.17.9.zip # Latest release as of publishing.
ES_DIR=${ES_PACKAGE%%.zip}
SITE=http://github.com/downloads/elasticsearch/elasticsearch
if [ ! -d "$ES_DIR" ] ; then
  wget --no-check-certificate $SITE/$ES_PACKAGE
  unzip $ES_PACKAGE
fi

# Install as system service
bin/elasticsearch install

# Run Elastic Search
bin/elasticsearch -f -Xmx3g -Xms3g

# Download grok
wget --no-check-certificate https://github.com/jordansissel/grok/tarball/master -O grok.tar.gz
tar zxf grok.tar.gz

# Install grok
cd *grok*
make grok
make install
ldconfig # Load the new libraries into the library path.

# If this is a 64-bit machine, create a symbolic link from /usr/lib64/ to the installed grok in /usr/lib/

# Download logstash
src=/usr/src/
mkdir -p $src/logstash
cd $src/logstash
wget http://semicomplete.com/files/logstash/logstash-1.0.17-monolithic.jar

Configure Logstash

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

cat << EOF > logstash.conf
input {
  stdin {
    format => "plain"
    message_format => "plain"
    type => "apache-access"
  }
  tcp {
    type => "apache-access"
    port => 3333
  }
}

filter {
  grok {
    type => "syslog" # for logs of type "syslog"
    pattern => "%{SYSLOGLINE}"
    # You can specify multiple 'pattern' lines
  }

  grok {
    type => "apache-access" # for logs of type 'apache-access'
    pattern => "%{COMBINEDAPACHELOG}"
  }

  date {
    type => "syslog"

    # The 'timestamp' and 'timestamp8601' names are for fields in the
    # logstash event.  The 'SYSLOGLINE' grok pattern above includes a field
    # named 'timestamp' that is set to the normal syslog timestamp if it
    # exists in the event.
    timestamp => "MMM  d HH:mm:ss"   # syslog 'day' value can be space-leading
    timestamp => "MMM dd HH:mm:ss"
    timestamp8601 => ISO8601 # Some syslogs use ISO8601 time format
  }

  date {
    type => "apache-access"
    timestamp => "dd/MMM/yyyy:HH:mm:ss Z"
  }
}

output {
  stdout { }
  # If you can't discover using multicast, set the address explicitly
  elasticsearch {
    host => "localhost"
  }
}
EOF

Run logstash

java -jar logstash-*-monolithic.jar agent -f logstash.conf -- web --backend elasticsearch://localhost/

Parse logs

time tail -n100000 /tmp/wwt-virt-extra-web15_access.log | nc 192.168.1.119 3333

View logs and Search results

Open browser to http://192.168.1.119:9292/

Comments