The New York Times published an article on Friday asking in its headline whether big data is a economic dud. Numerous quoted professors gave reasons why the analogy of data to oil doesn’t work, nor does the one of data to the electrical grid (and here I thought that analogy was about cloud computing).
All in all, it was kind of foolish. Here’s why.
1. Data is the new oil
Period. End of story. But we shouldn’t conflate data with big data. Data already has had significant effects on business and I can’t imagine anyone really taking issue with that statement. Whether or not it’s “big,” there’s no denying that all successful web companies and most successful companies of all types rely pretty heavily on data to make better decisions around marketing, operations or whatever.
Ask Google, Amazon, Walmart, Disney, GE, Target — and the list could go on for days — how important data is to them. The oil boom already happened, and it was great. Now, equal or not, it’s data that’s taking industry and commerce to the next level.
2. Big data is like oil sands
Big data is similar to plain, old data, but different in some important ways. Think about it like this: data is crude oil drilled from Saudi Arabian wells, while big data is the oil sands gunk that has lawmakers and businesspeople arguing so vehemently in the United States and Canada.
If regular data comes from sources such as customer information, purchase history and store sales numbers, big data comes from sources like social media, web searches, sensors and clickstreams. It takes a lot more work to turn that stuff into something usable. Right now, gains from big data at most companies (Google and Facebook aside), might be minimal at best.
This infographic from BloomReach helps illustrate how hard it complicated it can be to do big data right.
With that in mind, the question of whether big data is an economic dud is a little more fair. One might ask if all that effort is really worth the trouble to learn some potentially trivial insights about customer behavior. Maybe not, but that assumes big data is all about consumers’ personal data.
3. Big data and the web are not one and the same
A worldview of big data that’s limited to the web — which is the one the Times article espouses — is really missing a lot. Big data started on the web and a lot of technological innovation still happens there, but it’s making its way out. (Sometimes slowly — Sonic, a large fast-food chain, is just now getting started with big data, for example.) Ultimately, whatever effect the web has on the global economy might well be a drop in the bucket compared with how big data will affect other areas.
From a business perspective, you can already see what’s coming just by looking at Hadoop. It’s likely going to be the predominant big data platform going forward, and it’s constantly evolving to do new things. Companies big and small are using it (the vast majority of the Fortune 500 is at least doing proofs of concept, I’m told), some augmenting existing processes and others doing entirely new and innovative things they couldn’t otherwise do. The two biggest vendors focused solely on Hadoop — Cloudera and Hortonworks — are both talking about going public in the next year or two.
Looking broader, though, we can see even more promise on the horizon — the socioeconomic effect of which could be immense. There are companies like Planet Labs and Skybox using data to change the way we view Earth, and there are companies like Climate Corporation trying to rethink the economics of agriculture and technologies like real-time kinematics trying to make farming more efficient. Big data technology helped the United States track down Osama bin Laden and is (civil liberty concerns aside) helping governments identify potential terrorists and criminals all the time.
Rayid Ghani, once chief scientist for Obama for America, is pushing big data for social good at the University of Chicago and with a new startup called Edgeflip.
Data helped win a presidential election in 2012. I spent two hours last week (which I’ll detail in an upcoming post) talking about a litany of university projects trying to improve various social, municipal and health services using advances in data analysis. These include everything from aiding elections in Kenya to predicting hospital readmissions.
Some very smart people people think data analysis can do everything from predicting jet engine failures to curing cancer. Max Levchin thinks data can help women get pregnant. We have speakers at our upcoming Structure: Europe conference from both CERN and the European Space Agency. I’m guessing they’ll have a thing or two to say about how important big data technologies are to them.
Deep learning, arguably the holy grail for predictive analytics on big data sources like text, images and genomic data, is just making its way into the mainstream.
4. It’s way too early to call the game
I think a more accurate criticism of big data is probably to say it’s suffering from a case of being overhyped — vendors says their technologies do lots of things, but they don’t always spell out how much work that’s going to be. (Being overhyped, by the way, is not the same as being all hype.) As I’ve written before, anyone now getting upset that big data isn’t a perfect set of technologies and methodologies probably wasn’t paying much attention to what experts have been saying for years. But being overhyped now doesn’t mean big data can’t have a significant impact over time.
Source: Gartner Hype Cycle for Emerging Technologies
Yeah, it’s possible big data could turn out to be an economic dud, but there’s plenty of evidence that it’s just getting started in some significant areas. I’m gonna wait to see how all this plays out.
Feature image courtesy of Shutterstock user chuckstock.
Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.
- How to use big data to make better business decisions
- Takeaways from the second quarter in cloud and data
- The importance of putting the U and I in visualization