In the past, year big data has elevated from a hot topic in the enterprise to one of the most buzzed about, and potentially overhyped, phrases of the year. Big data has huge disruptive potential and the flood of attention should be no surprise. A recent IDC report stated that the business analytics software market grew by 14.1 percent in 2011 and will continue to grow to reach $ 50.7 billion in 2016, all driven by the focus on big data.
As one of the managing directors at Lightspeed Venture Partners, I spend a lot of time talking to companies about how they are using technology, such as big data, and meeting with entrepreneurs who are developing the next big disruptions in technology. I believe that to harness the power of this data revolution and gain a competitive edge, companies need to be able to do more than create and query their big data stores. They need to focus on making this big data fast, intuitive and easy to manipulate in new and interesting ways.
So what does that mean?
Companies need to do more than just store the data. Recent innovations in scale-out storage technology make it relatively inexpensive for companies to capture massive amounts of data. In many ways, that is how the big data conversation started. Companies such as Cloudera, Mapr, Vertica (acquired by HP) and Datastax are doing a great job of delivering the infrastructure required to hold and manage big data in a typical enterprise. (Full disclosure: Mapr and Datastax are both Lightspeed Venture Partners portfolio companies.)
Holding the data is step one of the process. The next challenge is how to use that data to help your business make better decisions. Right now, most companies are relying on data scientists to mine these raw stores of information. That’s a start, and we’ve seen some leading-edge companies make significant early revenue gains and cost savings with the help of data scientists. But, they are expensive, extremely scarce, non-real-time, and they don’t scale.
A new generation of startups is looking to democratize data science by building on top of the basic big data platforms to turbo-charge the speed, intuitiveness and collaborative methods by which businesses can extract value from the new flood of information.
The first challenge is to make this data fast. Today, it can take minutes or hours to get a response or glean a new insight buried in the typical enterprise’s mountain of data. As a result all questions must be carefully scripted and planned in advance. This limits the flexibility and agility of business questions that can be posed.
But in a world where big data can perform instantaneously or “at the speed of thought,” the results are dramatically different. When a user can maintain an unbroken train-of-thought, a fluid interplay starts to occur between asking an initial question, getting a response, refining and asking additional questions, and ultimately getting to a new, unanticipated “Eureka!” moment. Think Google Instant for the enterprise. There are a number of startups that are attacking this problem, including Qubole, Boundary, DataDog and several other stealth companies. (Full disclosure: Qubole and Boundary are both Lightspeed Venture Partners portfolio companies.)
The second layer of disruptive innovation relates to delivering dramatically improved experiences for navigating and manipulating data that has become so large that traditional spreadsheets, reports and charts would need millions of rows and pages to represent it (in simple terms, making data intuitive and easy to analyze).
New companies are focusing on a combination of AI (artificial intelligence), visualization, faceted search and social collaboration tools to empower hundreds or even thousands of ordinary business users to collectively mine, share and evaluate big data sets and gain insight without the need for a data scientist in the middle.
The emergence of self-service BI is allowing ordinary business users to drive the data warehouse for the first time and thereby eliminate expensive IT and data-scientist intermediaries. Historically, these intermediaries have been necessary to process requests and program reports, which ultimately constrained the business analysis process. Some of the most exciting companies innovating in this space include Tableau, Cliktech and Edgespring. (Full disclosure: Edgespring is a Lightspeed Venture Partners portfolio company.)
The big data revolution is in its early innings, and it is about to get even more exciting. So while the term may be overhyped, the massive potential for companies to take advantage of these new innovations makes it worth all of the extra attention.
Ravi Mhatre is a managing director of Lightspeed Venture Partners (@lightspeedvp), where he focuses on investments in enterprise IT, mobility, and Internet and cloud-based services and applications. You can follow him at on Twitter at @RMTacct.
We will be discussing the challenges of big data and scalable analytics at GigaOM’s Structure: Europe conference in Amsterdam, October 16 and 17.
Image courtesy of Flickr user altemark.
