Module 2: Index Management

Question 1

Splunk stores data in ___, which are organised in directories and files in disk.

Answer 1

indexes

Question 2

What are the features of a Splunk index?

Answer 2

• Indexes are stored in $SPLUNK_HOME/var/lib/Splunk
• the directory location is customisable for each index
• Indexes contain raw data and index files
• Indexes can be created by an admin
• There are many prebuilt indexes such as_internal, _audit and main.

Question 3

What is inside a Splunk index?

Answer 3

• Raw data stored in a compressed format
• Tdsidx files - time series index files that point to the raw data
• Metadata – files such a sources.data, sourcetypes.data and hosts.data

Question 4

Indexes store data in ___. __ are a set of directories organised by age.

Answer 4

Buckets

Question 5

What are the bucket types?

Answer 5

Bucket types are hot, warm, cold. froze, thawed

Question 6

Hot buckets

Answer 6

• newest data
• open for read and write
• can be more than one hot bucket in an index
• searchable
• roll to warm when reaching certain size, or upon a Splunk restart

Question 7

Warm buckets

Answer 7

• created when hot buckets roll
• not open for writing, but searchable
• reside in the same directory as hot buckets but renamed
• roll to cold buckets when exceeding max setting.

Question 8

Cold buckets

Answer 8

• Starting from the oldest bucket (based on time), warm buckets roll to cold
• Reside in different directory from hot and warm
• Searchable
• The directory location can be configured
• Possible to save cost by using cheaper storage.

Question 9

Frozen buckets

Answer 9

• After cold buckets age out based on policy, the roll to frozen
• Default action is to delete, but can be configured to archive.
• coldToFrozenDir or coldToFrozenScript in indexes.conf configures archiving
• Archived frozen buckets are not searchable

Question 10

Thawed buckets

Answer 10

• Frozen buckets can be thawed
• Thawed buckets are rebuilt into index and searchable
• Location can be configured in indexes.conf
• No age restriction
• Use Splunk rebuild command to rebuild data into the index

Question 11

Where is the bucket location of hot/warm buckets?

Answer 11

$SPLUNK_HOME/var/lib/Splunk/indexname/db

Question 12

Where is the bucket location of cold buckets?

Answer 12

$SPLUNK_HOME/var/lib/Splunk/indexname/colddb

Question 13

Where is the bucket location of frozen buckets?

Answer 13

• Frozen buckets are deleted by default. Can be optionally archived

Question 14

Where is the bucket location of thawed buckets?

Answer 14

$SPLUNK_HOME/var/lib/Splunk/indexname/thaweddb

Question 15

How are hot buckets named?

Answer 15

hot_v1_

Question 16

How are warm buckets named?

Answer 16

• Warm – db_newesttime_oldesttime_localid
o Local id is the ID of the bucket
o Newest and oldest time are in UTC epoch time in seconds

Question 17

Why create multiple indexes?

Answer 17

• Security – RBAC options set by index

* Retention policies are applied at index level

Question 18

What is index data integrity?

Answer 18

• Set in GUI, or enableDataIntegrityControl=true in indexes.conf
• Allows to ensure indexed data has not been tampered with
• Creates hash files (SHA 256) as the data is indexed
• Check intregrity: ./Splunk check-intregrity -index [index name]
• Regenerate hash: ./Splunk generate-hash-files -index [index name]
• Hash files are stored in the rawdata directory within the index

Question 19

___.conf is used to configure Splunk indexes and their properties

Answer 19

indexes

Question 20

Where is the default indexes.conf saved?

Answer 20

default indexes.conf in $SPLUNK_HOME/etc/system/default, but do not edit this. Create a new indexes.conf in $SPLUNK_HOME/etc/system/local

Question 21

What is the structure of indexes.conf?

Answer 21

• Default stanza for defining global properties
• If a property is defined outside of any specific stanza, at the top of the file, it is considered a global property
• If a property is defined at both global level and in a specific stanza, value in the specific stanza takes precedence
• If there are multiple definitions of the same settings in a stanza, last setting wins

Question 22

What is the fishbucket?

Answer 22

• fishbucket is a special Splunk internal index that is automatically created
• keeps track of ingestion progress of monitored files and directories
• located in $SPLUNK_HOME/var/lib/Splunk/fishbucket
• upon restart, using fishbucket, Splunk can start ingesting from where it left off.

Question 23

How can you use the fishbucket to reindex files?

Answer 23

• reindex a particular file (btprob command)
o ./Splunk cmd btprob -d $SPLUNK_HOME/var/lib/Splunk/fishbucket/splunk_private_db –file –reset
• Reindex all monitored files (remove entire fishbucket directory)
o Rm -rf /opt/Splunk/var/lib/Splunk/fishbucket
o You must restart Splunk forwarder

Question 24

What is data retention?

Answer 24

• must define a retention policy for the data indexed
• retention policy applied at index level
• set maxTotalDataSizeMB and/or frozenTimePeriodInSecs. maxTotal overrides frozenTime
• indexes.conf is used to configure retention policy

Question 25

the frozenTimePeriodinSecs is set to _ __ by default.

Answer 25

6 years

Question 26

What happens to expired data?

Answer 26

• When the bucket rolls from cold to frozen, by default the data is deleted • If coldToFrozenScript is configured, the scrip is executred • If coldToFrozenDir is configured, Splunk moves the expired buckets to this directory. This is what you do if you want to archive frozen buckets • To restore expired data, copy the archived buckets to thaweddb location and rebuild o ./Splunk rebuild $SPLUNK_HOME/var/lib/Splunk//thaweddb/

Question 27

How do you back up Splunk?

Answer 27

* must regularly back Splunk * back up warm and cold buckets, and entire etc directory * hot buckets can't be backed up without stopping Splunk, as Splunk is actively writing to them * in a clustered environment, you may not need to back up data as it is replicated. * in distributed environments, ensure you backup search heads and heavy forwarders

Question 28

How do you delete events in a Splunk index?

Answer 28

* best way is to let the data expire instead of deleting * can only do a virtual delete, not removed from disk * even admins can't delete data by default * create a user with a can_delete capability * run a search to list the desired events to be deleted, pipe the delete command

Question 29

How do you delete ALL events from a Splunk index?

Answer 29

* extremely dangerous command in production * Destroys index data from disk * Syntax: Splunk clean all -index * If you do not specifiy an index name, all indexes are considered for deletion