So after figuring out the greatness of Splunk’ing the statistical data from a USP I thought I’d go on to share more of what I’ve come across.
First off to shorten the time frame in which you pull your data so that it’s nearest to real-time as possible. Within your command.txt file, define the longrange value to only be one hour like so:
longrange -000100:
This will definitely help with the amount of time it takes to pull the data.
Now depending on the number of group’s you have defined to export, you can set a lower shortrange value like so:
shortrange -0010:
This will export the data for the last 10 minutes.
Now to get an idea on how long it will take to export this data, in my command.txt file I have defined those shortrange and longrange values above and the following groups:
group PhyPG ; Parity Groups
group PhyLDEV ; Logical Volumes
group PhyProc ; Micro-Processor usage
group PhyCSW ; Access Paths and Write Pending
group PG ; Parity Group Statistics
group LDEV internal ; LDEV usage in PGs, Ext Vol Groups or V-VOL Groups
group Port ; Port usage
group PortWWN ; Stats for HBAs connected to ports.
group LU ; LDEV usage Summarised by LU Path
It only takes 40-50 seconds to export the data. The reason I leave the shortrange value at ten minutes and have crontab run my export tool every 10 minutes is due to the fact it takes Splunk around 5-7 minutes to index the 500+ csv files that the export tool generates.
Next I’ve come across an issue, well more it was designed that way, with Splunk where it does not define column headings at fields if they begin with numbers or characters such as _ and -. This obviously can cause an issue due to the fact that the exported LDEV and PG data will begin with numbers (00:00:00, 00:01:00, etc..). So to solve this problem what you need to do is define within transforms.conf that CLEAN_KEYS = false … This will turn off the use of key cleaning that Splunk enables by default.
Finally, I’ve begun writing a Splunk app for use after indexing this data. If anyone has any interest in it, please feel free to let me know. It should be on Splunkbase by the end of May 2011.
Happy Splunk’ing!