Your organization has figured out that critical business insight exists in machine data. As the Splunk website points out, “Machine data contains a definitive record of all activity and behavior of your customers, users, transactions, applications, servers, networks and sensors”. And “Splunk Enterprise turns that (machine) data into real-time operational intelligence”.
“Splunk Enterprise is the industry-leading platform for operational intelligence. It (quickly) makes all of an organization’s machine data accessible, usable and valuable to the people that need it” – www.Splunk.com
So, how does Spunk accomplish that? Answer: by automatically indexing data. What does that mean? It means that as data is input to Splunk it “indexes it” so it can be queried in the most optimal and almost unlimited way.
So, what does that mean? If you have some experience with databases, you’ll understand that database indexes are (using straightforward terms) “copies of selected pieces of the data in a database saved in such a way as to support the speediest of query performances”. And Splunk is similar:
Splunk stores its “indexed data” in flat files known as indexes or indexers.
Right out of the gate
When you install Splunk, a number of pr-econfigured indexes are set up. These are the “main” (main) index, the “internal” (_internal) index and the “audit” (_audit) index.
The Main index is the Splunk “default” index. This is where all processed data will be stored by Splunk (unless otherwise specified by a Splunk administrator. An administrator can create indexes, edit index properties, delete indexes, and move indexes).
The internal index will hold Splunk’s internal logs and processing metrics.
The audit index will contain events related to the file system change monitor, auditing, and all user search history.
Management
The first step you take towards managing your indexes is to understand what indexes are currently setup and available (perhaps other that the standard ones listed above). To view a list of the indexes (in Splunk Web) you can click on Settings and then select Indexes:
From the indexes page you can see all indexes currently defined (notice the pre-configured indexes: audit, main and internal):
Splunk index management can be kept simple or, become very complex. There are a number of “index management” tasks you can perform, such as creating new indexes, removing indexes, moving indexes, configuring index properties, and so on.
Typically, index management becomes critical in a larger, multiple Splunk instance environment, but even when you use a simple, single Splunk server, it is a good idea to know your indexes!