This was my first time attending a Splunk .Conf so I was eager to feel the event; to gauge the excitement about the products and get a sense for how Splunk might succeed with its ambitious plans for growth in an ever more competitive market.
David Hodgson, September 30, 2016
The family goes to Orlando
Boasting 3 days of in-depth training, 185 technical sessions, inspiring keynotes each day, and the booths of 70 technical partners, .Conf2016 did not disappoint the nearly 5,000 attendees in terms of the intensity of the event and the velocity of interactions between us all. Obviously CEO Doug Merritt had primed his troops because in the kickoff keynote he quoted what they say internally at Splunk: “If you ever want to be inspired go out and talk to a customer”.
The fervor that the Splunk user base feels for the product brings back memories of VMware and SAP when they were cool, and promised change and progress. Perhaps because it’s a weird election cycle, Millennials are looking to technology rather than politics to shape their future. I don’t know for sure, but definitely this conference felt like the best sort of family gathering where people actually liked each other, wanted to collaborate on building solutions and wanted to bring new members into the fold.
The conference was held in the Dolphin & Swan at Disneyworld, Orlando. By the time we got to the Tuesday event night and a roaming party around the Hollywood studios park, the atmosphere was very much of one big family having fun together.
Learning how to get machines learning IT
Splunk is the clear market leader in providing a pragmatic platform for machine learning. The results have been real and beneficial whether its detecting intrusions from unusual data access patterns or predicting trends that can be addressed to optimize IT service delivery. A big theme at .Conf2016 was the power of Machine Learning and how it is shaping Splunk’s products.
In practice Machine Learning is very different from what we usually think of as Artificial Intelligence. AI seeks to build computer models that can emulate the functions of human brains. We expect that an AI would perceive its environment and exhibit goal seeking, purposefully behavior that is understood by humans. Ideally it would interact with humans to both receive input and augment our decision making abilities. By contrast Machine Learning is a sub-area of AI that is focused on pattern recognition that allows the system to “learn” and predict based on history, but without their being a rational explanation for that response that a human could understand. Machine Learning relies on the consumption of masses of granular data that can be processed with statistical analysis to make predictions and uncover “hidden insights” about relationships and trends. These “insights” are not necessarily causalities that have an explanation that humans could understand and replicate.
As a solution Splunk differentiates itself from the similar platforms like the ELK stack (Elastic Search, Logstash, Kibana) and Hadoop mainly through its functional completeness and ease of use. But it is proprietary and somewhat expensive to use with costs scaling based on the amount of data ingested daily. To accommodate customers concerns about growing costs and their desire to embrace open source technologies, Merritt announced at Conf2016 that Splunk Labs was enabling integration with Elastic Search, Spark, and Kafka- showing Splunk’s openness to adaptation to what customers are asking for in the field. The announcement was well received and is probably the answer both to customer needs and to Splunk can ensure continued popularity.
From a Syncsort perspective our Ironstream product has been focused on getting data to Splunk directly but customers have increasingly asked us to support a Kafka pipe to split data between Splunk and Hadoop. With Splunk’s new open architecture announced at .CONF2016 we will now plan to follow suit.
Splunking IT Operations
One of the significant areas of success that Splunk has had is in the area of monitoring tools for IT infrastructure. The normal users are Enterprise IT teams that need to monitor a broad array of platforms. They need to contextualize events by gathering data from connected platforms and using Splunk to do basic time-based correlation and advanced pattern recognitions. The rate of environmental change in hardware, software and connected devices makes traditional tools almost impossible to integrate and Splunk Enterprise offers a much simpler and more effective approach
For the last two years Syncsort has partnered with Splunk to add the mainframe platform to those monitored and this has proven to be an essential ingredient for the some of the world’s biggest IT organizations that have mainframes.
On the first day Merritt introduced the concept of Data as the DNA of IT, driving evolution and change. On Wednesday Andi Mann carried the theme further in his keynote “Re-Imagining IT” saying
“Digital transformation needs to be in your DNA; not passionately pursuing it is an existential challenge and threat to your individual and organization’s future success”.
Mann focused his discussion on the new 2.4 release of IT Service Intelligence (ITSI) that was unveiled at the conference. The main new capabilities of value are:
- Anomaly detection using machine learning
- Adaptive thresholds and tells you what the norms and thresholds should be for any time of the day, week, etc.
- Intelligent events with contextualized data wrapped in them
- End-to-end visibility of business services richly visualized for LOBs in the new “glass tables”
Syncsort also unveiled our latest work which was integration of mainframe data for ITSI 2.4. We demonstrated this with glass tables visualizing an online banking system from a mobile device to a mainframe running CICS and DB2. The Syncsort ITSI module is available for download from Splunkbase at no cost.
One of the most widely adopted use cases for Splunk is security and compliance. As normal you can roll your own very effectively using the Splunk Enterprise platform or you can add pre-built power features with Splunk’s premium app Enterprise Security (ES)
In her keynote Haiyan Song, SVP Security Markets described how alert based security is no longer adequate and stated that Machine Learning is now required to address internal and external threats. Splunk’s answer is User Behavioral Analytics or UBA.
At the conference Splunk announced new features in ES 4.5 and UBA 3.0 that were aimed at providing CISOs and their teams with operational intelligence. The highlights were:
- The Adaptive Response initiative allowing partners to openly integrate SIEM technology
- Glass tables available for advanced visualizations of the underlying data
- Enterprise hardening for the Caspida acquisition to create UBA as a product
Song described how UBA has the ability to understand and correlate user sessions across platforms and devices. She also brought on Richard Stone from the UK Ministry of Defence who explained how they are leveraging Splunk ES and UBA to create a DaaP (Defence as a Platform) ecosystem. To Stone this is a single information environment in which anyone with the appropriate credentials can access it from any point, enter a familiar environment, and access any information. He challenged us to “Date to Imagine” saying that the biggest constraint in security is our imagination.
Syncsort again extends these solution to the mainframe offering data integration to ES for RACF via the Ironstream product.
A new concept unveiled at .Conf2016 is a solution for DevOps. This is perhaps not surprising given Andi Mann’s background and he will be the champion for this new product. The solution uses the underlying capabilities of Splunk Enterprise to take a data-integration approach to deliver three areas of value:
- End-to-end visibility across every component in the DevOps tool chain
- Metrics in glass tables to show LOBs that code meets quality SLAs
- Correlation of business metrics with code changes to drive continual improvement
Splunking the Mainframe
One of the greatest things for me about the show was the number of people interested in the Syncsort booth. Even people who were not familiar with mainframes were interested to learn how we are Splunking the Mainframe!
Our CEO Josh Rogers delivered a phenomenal Cube interview that explained our strategy of moving data from Big Iron to Big Data (BIBD) platforms. Our deliverables and direction resonate with customers and prospects alike who are as excited with what we are doing as they are about Splunk!
During his appearance on the CUBE at .conf2016, Syncsort CEO Josh Rogers defined the Big Iron to Big Data (BIBD) challenge where customers need to take core data assets being created thru transactional workloads on mainframe and move them to next generation environments for analytics.
With the pace that things are moving across this market I am looking forward to .returning to .Conf in 2017 when it will be held in Washington DC, my home town. I know that both Splunk and Syncsort will have learned more and developed more, inspired by our customers. I can’t wait to see what we will have co-created and what evolves next from the data-DNA of IT.