Using Splunk to monitor Postilion events

Simply put, Splunk is a data crunching engine for IT data. It is a remarkable piece of software, extremely easy to install, configure and use. And it even comes free if you don't want to use its enterprise features and your data isn't too large.

The Postilion payment system comes with an embedded monitoring facility that is based on event generation. Postilion monitoring is nicely layered across the entire payments engine and the Postilion SDK has a namespace dedicated to it, so Postilion developers can also benefit from this infrastructure.

But the monitoring facility, as nice as it is, leaves a lot to be desired in terms of searching. Any person that has operated the Postilion payment system can testify to that effect. Although there are consoles to query events using some criteria, the needs of the operational staff are always impossible to predict.

Using Splunk to index Postilion events opens a whole range of new possibilities as far as monitoring the payment system goes. There are two main ways to send your events to a Splunk instance:

1. Write a custom program or data export that takes the latest events and dump them to a directory indexed by Splunk.
2. Use the SDK to create a custom scribe that will feed SPlunk either indirectly (through files) or directly (through a UDP socket).

 

Click for large image

Once the events start flowing into Splunk, you can use all of Splunk facilities to search your data. Some of the cool things you can do:

• Convert any piece of disparate data in fields. Splunk does a great job at extracting fields out of raw data, allowing for better querying.
• Generate reports and charts based on fields.
• Create all sorts of monitors and alerts. Sure, Postilion has alerts as well but Splunk really excels in this area. For example, you can create an alert that will notify staff is a cutover event isn't indexed at a specific time.