====== Splunk ======
===== Install =====
==== Server (Host Name:splunkserver) ====
On the server download rpm file and run
<code bash>
rpm -ivh splunk-6.2.2-255606-linux-2.6-x86_64.rpm
</code>
  * Default user/password is: admin/changeme
  * Default HTTP port is: 8000
  * Default Home: /opt/splunk

Full start-up log
<code>
# /opt/splunk/bin/splunk start
 
Splunk> Finding your faults, just like mom.
 
Checking prerequisites...
        Checking http port [8000]: open
        Checking mgmt port [8089]: open
        Checking appserver port [127.0.0.1:8065]: open
        Checking kvstore port [8191]: open
        Checking configuration...  Done.
        Checking critical directories...        Done
        Checking indexes...
                Validated: _audit _blocksignature _internal _introspection _thefishbucket history main summary
        Done
        Checking filesystem compatibility...  Done
        Checking conf files for problems...
        Done
All preliminary checks passed.
 
Starting splunk server daemon (splunkd)...
Done
                                                           [  OK  ]
 
Waiting for web server at http://127.0.0.1:8000 to be available... Done
 
 
If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com
 
The Splunk web interface is at http://splunkserver:8000
</code>

==== Agent (Host Name:splunkclient) ====
On the client download rpm file and run
<code bash>
rpm -ivh splunkforwarder-6.2.2-255606-linux-2.6-x86_64.rpm
</code>

===== Configuration =====
==== Server ====
=== Define Listen Port ===
On the web console:
<code>
Setting -> Forwarding and Receiving -> Add Receiving -> Define listen Port (9997)
</code>
=== New Index ===
Create new index rts on splunk enterprise web
=== New Source Type ===
In this example we are creating a new source type for haproxy as that is not available by default for splunk. Steps
  * Copy /opt/splunk/etc/system/local/props.conf from /opt/splunk/etc/system/default/props.conf
  * Created a new sourccetype block for haproxylog
<code>
[haproxylog]
category = Web
pulldown_type = true
EXTRACT-haproxy_httplog = haproxy\b.*? (?<client_ip>\d+\.\d+\.\d+\.\d+):(?<client_port>\d+) \[[^\]]+\] (?<frontend_name>\S+) (?<backend_name>[^/]+)/(?<server_name>\S+) (?<request_time>\d+)/(?<queue_time>\d+)/(?<connect_time>\d+)/(?<response_time>\d+)/(?<total_time>\d+) (?<status_code>\d+) (?<response_size>\d+) \S+ \S+ (?<flags>\S{4}) (?<process_connections>\d+)/(?<frontend_connections>\d+)/(?<backend_connections>\d+)/(?<server_connections>\d+)/(?<retries>\d+) (?<server_queue_size>\d+)/(?<backend_queue_size>\d+)(?: \{(?<request_headers>[^\}]*)\})?(?: \{(?<response_headers>[^\}]*)\})? "(?<method>\S+)\s+(?<uri>[^"]+?)(?: HTTP\S+)?"
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
description = HAProxy single line log
disabled = false
</code>
=== Other settings ===
Changed minimum disk free space required to 2GB from 5GB.

==== Client ====
=== Configure to monitor haproxy.log ===
Add forward server as the splunkserver on the designated port
<code bash>
/opt/splunkforwarder/bin/splunk add forward-server splunkserver:9997 -auth admin:changeme
</code>

== Log forwarding ==
Modify /opt/splunkforwarder/etc/system/local/inputs.conf and add a monitor stanza to monitor haproxy.log, specified sourcetype and index
<code>
[default]
host = splunkclient

[monitor:///var/log/haproxy.log]
sourcetype = haproxylog
index = rts
</code>

Modify /opt/splunkforwarder/etc/system/local/outputs.conf to add forwarding
<code>
[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = splunkserver:9997

[tcpout-server://splunkserver:9997]
</code>

=== Restart forwarder ===
<code bash>
/opt/splunkforwarder/bin/splunk restart
</code>