Splunk

On the server download rpm file and run

rpm -ivh splunk-6.2.2-255606-linux-2.6-x86_64.rpm

• Default user/password is: admin/changeme
• Default HTTP port is: 8000
• Default Home: /opt/splunk

Full start-up log

# /opt/splunk/bin/splunk start
 
Splunk> Finding your faults, just like mom.
 
Checking prerequisites...
        Checking http port [8000]: open
        Checking mgmt port [8089]: open
        Checking appserver port [127.0.0.1:8065]: open
        Checking kvstore port [8191]: open
        Checking configuration...  Done.
        Checking critical directories...        Done
        Checking indexes...
                Validated: _audit _blocksignature _internal _introspection _thefishbucket history main summary
        Done
        Checking filesystem compatibility...  Done
        Checking conf files for problems...
        Done
All preliminary checks passed.
 
Starting splunk server daemon (splunkd)...
Done
                                                           [  OK  ]
 
Waiting for web server at http://127.0.0.1:8000 to be available... Done
 
 
If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com
 
The Splunk web interface is at http://splunkserver:8000

On the client download rpm file and run

rpm -ivh splunkforwarder-6.2.2-255606-linux-2.6-x86_64.rpm

On the web console:

Setting -> Forwarding and Receiving -> Add Receiving -> Define listen Port (9997)

Create new index rts on splunk enterprise web

In this example we are creating a new source type for haproxy as that is not available by default for splunk. Steps

• Copy /opt/splunk/etc/system/local/props.conf from /opt/splunk/etc/system/default/props.conf
• Created a new sourccetype block for haproxylog

[haproxylog]
category = Web
pulldown_type = true
EXTRACT-haproxy_httplog = haproxy\b.*? (?<client_ip>\d+\.\d+\.\d+\.\d+):(?<client_port>\d+) \[[^\]]+\] (?<frontend_name>\S+) (?<backend_name>[^/]+)/(?<server_name>\S+) (?<request_time>\d+)/(?<queue_time>\d+)/(?<connect_time>\d+)/(?<response_time>\d+)/(?<total_time>\d+) (?<status_code>\d+) (?<response_size>\d+) \S+ \S+ (?<flags>\S{4}) (?<process_connections>\d+)/(?<frontend_connections>\d+)/(?<backend_connections>\d+)/(?<server_connections>\d+)/(?<retries>\d+) (?<server_queue_size>\d+)/(?<backend_queue_size>\d+)(?: \{(?<request_headers>[^\}]*)\})?(?: \{(?<response_headers>[^\}]*)\})? "(?<method>\S+)\s+(?<uri>[^"]+?)(?: HTTP\S+)?"
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
description = HAProxy single line log
disabled = false

Changed minimum disk free space required to 2GB from 5GB.

Add forward server as the splunkserver on the designated port

/opt/splunkforwarder/bin/splunk add forward-server splunkserver:9997 -auth admin:changeme

Modify /opt/splunkforwarder/etc/system/local/inputs.conf and add a monitor stanza to monitor haproxy.log, specified sourcetype and index

[default]
host = splunkclient

[monitor:///var/log/haproxy.log]
sourcetype = haproxylog
index = rts

Modify /opt/splunkforwarder/etc/system/local/outputs.conf to add forwarding

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = splunkserver:9997

[tcpout-server://splunkserver:9997]

/opt/splunkforwarder/bin/splunk restart