Getting Hitachi Performance Data into Splunk

I’ve been Splunk’ing A LOT recently and one of the most recent accomplishments was getting Hitachi Performance Data out of a USP (Universal Storage Platform) and in to Splunk. So I thought I’d write quickly on how it can be done.

Hitachi has a tool called “Export Tool” which basically connects to the SVP of your Hitachi USP as a given user, collects all of the data you tell it to (Port stats, LUN/LDEV stats, etc…) and output’s it into numerous CSV files. All of the options (SVP IP, user/pass, data to collect, etc…) can be configured within the Export Tool command file. This tool comes on the software package with your USP or if your microcode has been upgraded then you’ll need to obtain the correct copy from your Hitachi CE.

So here’s what I’ve done with the Export Tool. It runs every hour, on the hour, and gathers the data I need. It saves it, in CSV format, to a local folder. The CSV files begin with 6 lines of data that identify the time, your USP SVP, etc… this data causes issues when indexed with Splunk because for Splunk to automagically map the fields according to the header of the CSV file the header must be on the first line of the file. So to get around this I’ve used a sed one-liner to remove the first 6 lines of every CSV file that the Export Tool generated.

After a successful finish using sed all of the CSV files are then copied to the directory that Splunk monitor’s for the expected Hitachi data. Splunk is configured to monitor this directory, capture all the csv files and index them accordingly. For best optimization the files are of course indexed into their own index.

For building searches around the indexed data and making pretty charts I recommend looking at the Performance Manager User’s Guide. Pages 168 onward contain all the knowledge of what file stores what information and so on. I also then recommend printing off the “Storage Utilization by Host Report” that is accessible through the Hitachi Device Manager. This report will allow you to understand which hosts contain which LDEVs, the Port/HSD(iSCSI Target)/LUN identifiers, etc… Trust me, I could not have understood how to map the fields and build charts without this report. For example after the CSV’s are indexed by Splunk some of the field’s that are mapped will look like this:

CL4_F_01_HCMD8d00__0000

You can make general sense out of this field ID but using the report that I mentioned earlier you can quickly look through the hosts to the find the matching Port/HSD(iSCSI Target)/LUN identifier. In this case it would be: CL4-F/HCMD8d00/00 And viola! You then have the host this field maps to, Parity Group, LDEV, etc…

So good luck to all those out there Splunk’ing this type of data and let me know if you have any questions or want to share ideas!