Are companies addicted to Hadoop?

The size of Hadoop deployments appears to have tripled since October, according to statistics from Cloudera. If accurate, they help prove assumptions that Hadoop usage grows quickly once organizations wrap their heads around how to use it. This seems especially true in industries where better customer insights are tied directly to more revenue.

In his DBMS2 blog this morning, database expert Curt Monash quotes Cloudera VP of Customer Solutions Omer Trajman as stating that his employer counts 22 Hadoop clusters (not counting non-Cloudera users Facebook and Yahoo) in production that are managing more than a petabyte of data apiece. Additionally, Trajman said the average cluster size has grown to more than 200 nodes, roughly tripling since the company conducted a survey of attendees at its Hadoop World conference in October 2010.

I relied in part on that Hadoop World survey in writing a GigaOM Pro report in March (subscription required). What’s interesting, if Trajman’s stats are accurate, is how fast the high end of Hadoop clusters has grown. At that point, the average cluster size was about 67 nodes, but it was less than 40 nodes once you excluded three outliers claiming 999 or 1,000 nodes. Trajman notes that most clusters are still less than 30 nodes, but the increasing number of large clusters is boosting the average size.

Presumably, there are also a greater number of those starter 30-node clusters, which eventually will grow closer to the 200-node average.

This aligns with the findings of a Karmasphere survey conducted contemporaneously with the Hadoop World survey. In the Karmasphere survey, 32 percent of respondents expected to be managing a cluster between 10 and 49 nodes in a year, compared with 36 percent claiming they presently were managing a cluster in that range. However, 61 percent expected to manage clusters between 60 and 1,000-plus nodes in a year, compared with only 32 percent managing that size cluster at the time of the survey.

Also noteworthy is Trajman’s observation that the biggest population of large clusters is in the advertising space. This shouldn’t be surprising: Using analytics to improve targeted advertising directly results in more money. We have highlighted this trend before, specifically as a part of video-advertising strategies, and, earlier this week, as a component of social-media-based ad targeting.

Of course, Facebook, Yahoo and Google (which uses its MapReduce-plus-Big-Table system, on which Hadoop is based) also rely heavily on Hadoop to serve the advertising programs that drive their revenues. And they all have huge deployments spanning multiple clusters around the world. Yahoo now manages more than 42,000 Hadoop nodes.

The amazing thing about this purported growth is that we’re just getting started. Many see a market that is over a billion dollars for Hadoop products, but it’s nowhere near that right now. As Hadoop best practices and technologies improve along with the overall understanding of big data across industries, who knows how large deployments will become. Perhaps today’s SMB will be tomorrow’s Yahoo.

Feature image courtesy of Flickr user miheco

Related content from GigaOM Pro (subscription req’d):

  • Defining Hadoop: the Players, Technologies and Challenges of 2011
  • Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
  • Putting Big Data to Work: Opportunities for Enterprises



GigaOM — Tech News, Analysis and Trends