Hadoop through the years: A GigaOM retrospective

by

A few years before we had a Structure:Data conference dedicated to big data — and, by proxy, Hadoop — GigaOM spotted Hadoop’s promise and began trying to spread the word about and advance the discussion around this groundbreaking technology. Now that Hadoop is 10 years old (give or take), we thought now would be a good time to look back on how Hadoop has influenced our events and editorial over years. This is the final installment in our four-part Hadoop anthology that has already covered its birth, present and future.

Think about this like Hadoop’s greatest hits, but know that there will be more to come. Although the big data discussion is moving away from Hadoop somewhat, it’s still an integral — if not the integral — part of the discussion around data infrastructure. We have two great panels on Hadoop at our Structure:Data conference March 20-21 in New York (which include participants from Facebook, Platfora, Continuuity and EMC’s Pivotal Initiative (whose leader Paul Maritz will also be speaking), among others) and will keep up with all things Hadoop and data for the next 10 years.

The biggest news

  1. Hadoop-focused startup Cloudera raises $ 5 million (March 15, 2009)
  2. Friends on the move: Hadoop, AOL & PayPal  (Aug. 10, 2009)
  3. Survey: Hadoop is great, but challenges remain (Sept. 29, 2010)
  4. Yahoo suggests MapReduce overhaul to improve Hadoop performance (March 17, 2011)
  5. Meet MapR, a competitor to Hadoop leader Cloudera (March 24, 2011)
  6. EMC makes a big bet on Hadoop (May 9, 2011)
  7. Exclusive: Yahoo launching Hadoop spinoff this week (June 27, 2011)
  8. Microsoft’s Hadoop play is shaping up, and it includes Excel (Feb. 28, 2012)
  9. VMware aims for Hadoop on VMs with ‘Serengeti’ project (June 13, 2012)
  10. Cloudera makes SQL a first-class citizen in Hadoop (Oct. 24, 2012)

The best analysis

  1. The data mining renaissance (April 10, 2009)
  2. Is Hadoop champion Cloudera the next Red Hat? (Oct. 2, 2009)
  3. Meet the big data equivalent of the LAMP stack (Aug. 1, 2010)
  4. As big data takes off, the Hadoop wars begin (March 25, 2011)
  5. Hadoop’s civil war: Does it matter who contributes the most? (Oct. 7, 2011)
  6. 5 low-profile startups that could change the face of big data (Jan. 28, 2012)
  7. What it really means when someone says Hadoop (Feb. 6, 2012)
  8. Hadoop jumps through hoops, becomes mainstream (March 3, 2012)
  9. Why the days are numbered for Hadoop as we know it (July 7, 2012)
  10. A few stats, rumors and stories on Hadoop’s rapid growth (Nov. 9, 2012)

The coolest users … aside from Yahoo

Facebook

  • Facebook open sources Corona — a better way to do webscale Hadoop
  • How Facebook moved 30 petabytes of Hadoop data
  • Facebook is collecting your data — 500TB a day
  • How Facebook keeps 100 petabytes of Hadoop data online
 Netflix

  • Netflix analyzes a lot of data about your viewing habits
  • Netflix shows off how it does Hadoop in the cloud
 Etsy

  • Etsy unveils its infrastructure (and its SuperMicro love)
  • How Etsy handcrafted a big data strategy
eBay

  • Under the covers of eBay’s big data operation
  • Making the web more efficient 1,000 servers at a time
 The smart grid world

  • How to use open-source Hadoop for the smart grid
  • Opower, the big data energy player to beat
 Obama for America

  • Vote for me: How data will change the 2012 election
  • How Obama’s data scientists built a volunteer army on Facebook
Yelp BloomReach Ancestry.com
LinkedIn Quantcast Disney
Orbitz Klout Twitter
The medical world Climate Corporation Skybox Imaging
Tumblr Intuit @Walmartlabs
Zions Bancorporation LivePerson The enterprise security world

Taking Hadoop to the stage

The Hadoop Meetup (May 1, 2008)

Cutting (center) flanked by Baldeschwieler and Om Malik at GigaOM’s Hadoop Meetup in 2008.

Cutting (center) flanked by Baldeschwieler and Om Malik at GigaOM’s Hadoop Meetup in 2008.

Next-generation data stores (Structure 2008; start at 57:00)

document.getElementById(‘wpcom-iframe-form-e9c9dd541881ae5993bf1f684bd62e12’).submit();

Hadoop, NoSQL and webscale data (Structure 2009)

document.getElementById(‘wpcom-iframe-form-7488481157425ee65fa274480fdb9050’).submit();

The big data tsunami (Structure 2010)

document.getElementById(‘wpcom-iframe-form-7e7a4a2d21ea8e0ad27bcc692727cad9’).submit();

Hadoop and beyond (Structure: Data 2011)

document.getElementById(‘wpcom-iframe-form-122220630b85e27f757d13181f9f8427’).submit();

What’s next for Hadoop? (Structure: Data 2012)

document.getElementById(‘wpcom-iframe-form-48d96581b9e832beabd0a8b7e2a86b9d’).submit();

Mike Olson on Hadoop (Structure: Data 2012)

document.getElementById(‘wpcom-iframe-form-6b5f1e48ce8af570a22d43717f23559c’).submit();

Analyzing data with HBase (Structure: Data 2012)

document.getElementById(‘wpcom-iframe-form-06bc295244a48be990273e97f862ca80’).submit();

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

  • How search can unlock the power of big data
  • The importance of putting the U and I in visualization
  • Infrastructure Q1: Cloud and big data woo enterprises


GigaOM

Related Posts: