Module 5: Getting Data into Splunk Enterprise/Cloud

Question 1

What is a Splunk intermediate forwarder?

Answer 1

• Last hop on the on-prem network
• Need to UF credentials from Splunk Cloud
• Gather data and parse if necessary
• Send data across the internet to Splunk Cloud for indexing

Question 2

What is a Splunk Universal Forwarder?

Answer 2

Specifically designed to run on production servers
o	Lightweight
o	Minimal CPU and mem usages
o	No web interface
A separate install binary
Use UF for defining Splunk inputs

Question 3

What is a Splunk Heavy Forwarder?

Answer 3

Splunk Enterprise binary configured as a forwarder (outputs.conf)
o Requires forwarder license
o Does everything except indexing
o Creates more network traffic than UF
Accepts all input types and can parse raw data
Can be used as IF for other UF
o As a mid-tier component
o Aggregate data from UFs, parse, and route to Splunk cloud
o Can be a single point of failure

Question 4

What are the forwarder best practices?

Answer 4

• UFs are the preferred method of getting data in to Splunk cloud
• Use a HF only when necessary
o Parsing data before going into cloud
o When apps are not allowed on Splunk cloud
• Recommend ratio of forwarders to indexers is 2 UFs to 1 indexer

Question 5

What are sourcetypes?

Answer 5

• Splunk uses source types to categorise data
• Can be used while searching to filter data
• Defined in props.conf on the indexer or heavy-forwarder
• Automatically assigned by Splunk or explicitly assigned by the data administrator.

Question 6

What are the three phases of indexing?

Answer 6

Input phase
Parsing phase
Indexing phase

Question 7

What is the input phase?

Answer 7

o data is read from data sources such as files and network
o Metadata such as host, source, and sourcetype are applied
o Most configuration is in inputs.conf (some in props.conf)
o Operates on the entire data stream

Question 8

What is the parsing phase?

Answer 8

o	Data is broken in to events
o	Timestamp extracted.
o	Most configuration in props.cnf (some in transforms.conf)
o	Operates on individual events
o	Event level transformations

Question 9

What is the indexing phase?

Answer 9

o Segment events that then can be searched
o Build index
o Licence meter runs before the data is writing to disk
o Raw data and index files are written to disk (in buckets)

Question 10

The Indexing pipeline can be viewed from the ___.

Answer 10

monitoring console (monitoring console > indexing > performance > indexing performance:instance)

Question 11

What is the parsing pipeline?

Answer 11

Responsible for character encoding and line breaking

Question 12

What is the merging pipeline?

Answer 12

Responsible for timestamp extraction and merges multi-line events

Question 13

What is the typing pipeline?

Answer 13

Responsible for punctuations extraction and regex replacement

Question 14

What is the indexing pipeline?

Answer 14

Responsible for indexing.

Question 15

What are three use cases for using a HF?

Answer 15

• Parse and route data before sending to indexers
• Firewall rule simplification, can allow just the heavy forwarder through the firewall as opposed to all UF
• Use addons such as DBconnect and HTTP event collector.

Question 16

How do you install a UF in Unix?

Answer 16

• Download the tar ball and unzip in desired location
• Install UF in a separate file system (example /opt/splunkforwarder)
• The command ‘Splunk’ is located in $SPLUNK_HOME/bin that can be used to start/stop UF
• Default admin password is changeme
• Employ config management tools such as ansible or puppet

Question 17

How do you install a UF in Windows?

Answer 17

• Can use MSI wizard or msiexec.exe CLI
• Can be installed as a domain admin
• Use msiexec.exe with /quite an AGREETOLICENSE=yes for complete silent install
• Can enable windows event logs and perfmon ingestion (recommended)
• Runs as a windows service

Question 18

What must be done to set up forwarding?

Answer 18

• Set up receiving on the indexers. Primary config file is inputs.conf
• Set up forwarding on the UF. Primary config file is outputs.conf

Question 19

How do you set up receiving?

Answer 19

• Must be done before forwarders can send data
• Using gui – Settings > forwarding and receiving > configure receiving
• CLI – Splunk enable listen
• Default port is 9997.

Question 20

How do you set up forwarding?

Answer 20

• Use CLI or outputs.conf to set up forwarding
• CLI - ./Splunk add forward-server
• Edit outputs.conf

Question 21

What are the four ways to add inputs?

Answer 21

• Using deployment server to distribute inputs.conf
• Using Splunk CLI]
o $SPLUNK_HOME/bin/slunk add monitor [location & filename]
• Editing inputs.conf manually
• Using Splunk add-ons

Question 22

What are some additional forwarder configurations?

Answer 22

Forward data selectively to multiple indexers
Load balancing
Compression
SSL
Buffering (queueing)
Index acknowledgement

Question 23

How do you configure forwarding data selectively to multiple indexers?

Answer 23

• Define two tcpout destinations in outputs.conf

* Use _TCP_ROUTING in inputs.conf to selectively send data

Question 24

How do you configure compression?

Answer 24

• Enable compression on both indexer and forwarder
• Reduces network traffic, but slightly increase CPU usage
• Compressed=true on the forwarder outputs.conf and indexer inputs.comf

Question 25

How do you configure SSL?

Answer 25

* Enabling SSL automatically enables compression * Can increase CPU usage * Specify cert files in the conf files

Question 26

How do you configure buffering?

Answer 26

* Configure maxQueueSize in outputs.conf to enable queuing * Default is 500kb when useACK is false. 7mb when useACK is true * When load balancing is enabled, queueing occurs only when all indexers are not available

Question 27

How do you configure index acknowledgement?

Answer 27

* Ensures data is received by indexers * Protects against data loss * Forwarders resend data not acknowledged by indexer * A wait queue is created with 3x size of maxQueuSize * Enabled by using useACK = true in outputs.conf

Question 28

How do you configure load balancing accross multiple indexers?

Answer 28

* Specify a list of indexers in the server list in outputs.conf * Every 30 seconds, forwarder will randomly select an available indexer from the list * Frequency can be updated using autoLBFrequency setting (not recommended)

Question 29

Name the troubleshooting steps when troubleshooting a forwarder?

Answer 29

``` Use SPL • index=_internal host= Check if forwarder is running • $SPLUNK_HOME/bin/Splunk status • Ps -ef | grep Splunk Check forwarder configuration • $SPLUNK_HOME/bin/Splunk list forward-server Check indexer configuration • $SPLUNK_HOME/bin/Splunk display listen Check forwarder splunkd log • $SPLUNK_HOME/var/log/Splunk/splunkd.log ```