Using Splunk for IBM Bluemix syslog drain

One of the repeating voice of IBM Bluemix users are how to manage application logs. A while ago, I wrote a blog entry  to retrieve logs from IBM Bluemix using 3rd party logging service. However, some users may have concerns about using public SaaS, since log may contain some relevant information such as PI (personal information). I had a discussion with Daniel Toczala (one of emerging technology leaders inside IBM), and he suggested about the trial of “Splunk“. Because it offers some trial period, or community version for small team (free with some limitation), I was able to evaluate it.

Fortunately, Cloud Foundry itself does support Splunk, so this blog entry is about how I tried Splunk for IBM Bluemix syslog drain.

Setup Splunk

Firstly, I have created a virtual machine using IBM Bluemix VM(β) service. I have chosen Ubuntu as image. Installation of Splunk was very simple. Download Splunk Light from the download site, and then install it with single line of command.

# dpkg -i  ~/splunklight-6.3.0-aa7d4b1ccb80-linux-2.6-amd64.deb

The software was installed on /opt/splunk. Looks I like I need to install and patch log format to be able to integrate with Bluemix. I have follow the guideline from Cloud Foundry documentation ( here ) .  Based on the guide, I added “rfc5424-syslog” package, since this format is used in loggreator syslog drain output.

# cd /opt/splunk/etc/apps
# tar xvfz ~/rfc5424-syslog_11.tgz

Then, I modified the file “/opt/splunk/etc/apps/rfc5424/default/transforms.conf”

[rfc5424_host]
DEST_KEY = MetaData:Host
REGEX = <\d+>\d{1}\s{1}\S+\s{1}(\S+)
FORMAT = host::$1

[rfc5424_header]
REGEX = <(\d+)>\d{1}\s{1}\S+\s{1}\S+\s{1}(\S+)\s{1}(\S+)\s{1}(\S+)
FORMAT = prival::$1 appname::$2 procid::$3 msgid::$4
MV_ADD = true

After this setup, I need to open some ports on ubuntu machine to accept incoming syslog drain (5140) and Splunk web UI (8000) because IBM Bluemix VM image has firewall setup by default.

# iptables -A INPUT -p tcp --dport 5140 -j ACCEPT
# iptables -A INPUT -p tcp --sport 5140 -j ACCEPT
# iptables -A INPUT -p tcp --dport 8000 -j ACCEPT
# iptables -A INPUT -p tcp --sport 8000 -j ACCEPT

And then, I run splunk.

# /opt/splunk/bin/splunk start --accept-license

Splunk setting

I need to setup Splunk to accept syslog drain from IBM Bluemix. I need to create “Data Input” for syslog. From left hand side of Web UI, selected “Data -> Data inputs” menu.

add data input

This opens a list of inputs that Splunk (by default). Syslog drain uses TCP protocol, so I selected TCP.

select TCP

This opened a new entry for TCP. I have put 5140 as a port number. This was setup as open port for the communication, keep the rest of fields as blank and clicked “Next”.

setup listen port

I need to select “rfc5424_syslog” which I have installed for this integration.

select rfc

I selected IP for method. Index needs to be created as well.

create-new-index-blemix

I created “bluemix” as index.

add index bluemix

 

In the review, I confirmed the setting is right.

review

And then enabled this input.

enable TCP

IBM Bluemix setting

Bluemix application needs to bind syslog drain service in user space. I have created like this:

% cf cups splunk -l syslog://dummyhost:5140

Note that dummyhost is not real name, but to hide the actual host name. And then bind the service and restage the application.

$ cf bind-service myapp splunk
$ cf restage myapp

I played with my application, then I put the query string in the Splunk web UI  like:

source="tcp:5140" index="bluemix" sourcetype="rfc5424_syslog"

I got a stream of logs in my Splunk web interface.

actual log stream - need to edit

Even though this is Splunk Light, I can retain logs 500MB per day. This is quite useful for my experiment and evaluation of this integration for who needs this kind of solution.

Advertisements
This entry was posted in Bluemix. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s