Is 2013 the year Hadoop uptake turns into a ‘tornado’?

Karmasphere CEO Gail Ennis told me recently she thinks “2013 is going to be the year when we see [Hadoop adoption] go a lot more mainstream and [turn] into a tornado.” I like the prediction, as much for its imagery as for her near-term certainty. Hadoop right now is like a funnel cloud spiraling above the ground: it took some strong forces (famous users, lots of hype and lots of venture capital) to create the cloud, but it’s still too high up to do real damage. When it hits the ground, though, watch out.

Ennis and her company Karmasphere are among the companies trying to bring Hadoop down to ground level or, in corporate IT parlance, to business users rather than data scientists and systems engineers. The easier Hadoop is to use, the more insights companies get to discover and the easier it is to justify paying for it.

Breaking down (or setting up) barriers

Karmasphere has been trying to help data analysts build better Hadoop applications for a couple years, but on Monday it took a big step forward with the 2.o release of its namesake software. The new version focuses on self-service and collaboration, letting business analysts, data analysts and others share a workspace in which they can team up on datasets to make the most of their individual skills. Most importantly, though, they can do so without calling in the IT department for help accessing their data or running a job.

It’s a trend we’ve reported on numerous times — one that Karmasphere and Datameer helped kick off, and that has since been given new life (GigaOM Pro subscription req’d) by Microsoft and a slew of other startups ranging from Platfora to Qubole. That’s a good thing for numerous reasons, including that business users and IT personnel are getting sick of each other, Ennis said — “IT wants to teach users to fish.”

When it’s easier to access and analyze data stored in Hadoop (Karmasphere 2.o, for example, lets users write SQL-like queries while also connecting to their favorite BI tools and analytics software), business departments and IT both get a little more space. That’s also means it’s easier to sell Hadoop products because everyone knows their roles and what items are coming out of whose budgets. Individual departments buy and manage business tools such as Karmasphere, while IT buys and manages system software such as Cloudera or Hortonworks.

You scream, I scream, we all scream for Hadoop

Datameer, is also doing its part to make the Hadoop tornado touch down, announcing on Monday a new version of its spreadsheet-for-Hadoop product that runs on a single computer. Among the usability and data-access improvements to its own 2.0 release (such as a WYSIWYG infographic creator) are a Workgroup edition that resides on a single server and supports up to 1TB and 50 users, and a personal edition that installs on a laptop or desktop computer.

Such small deployments might seem like an oxymoron — a single server, much less a single laptop is hardly big data — but big data is a misnomer itself. The real benefit of technologies such as Hadoop is being able to analyze new types of data in new ways, regardless how much you have. Small companies, departments or even individuals might not need a whole Hadoop cluster, but still might want to analyze their unstructured data in ways that other BI tools won’t let them.

Datameer’s personal edition is particularly compelling because of the flexibility it allows. In the case of one beta customer, VP of Marketing Joe Nicholson told me, an employee was able to import some data onto his desktop and, in a few nights working at home, draw a correlation between temperature and the amount of ice cream and water his company sells. The company had been struggling with this correlation for a while, but because of Hadoop the employee was able to analyze the company’s traditional sales data against publicly available weather data.

Given that it’s halfway through 2012, Ennis’s prediction of 2013 being the year that Hadoop turns into a tornado and sucks up everything its path seems fair enough. What Karmapshere and Datameer are doing — along with the availability of antipicated products from Platfora, Continuuity and other stealth-mode startups, cloud-based Hadoop services, and non-stop innovation on at the distribution layer — will make Hadoop a lot harder to resist, at least.

Feature image courtesy of Shutterstock user Melanie Metz.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.