When your business is to insure farmers against the effects of bad weather, you’d better have some seriously accurate data on your side. Mother Nature, after all, can be somewhat unpredictable — even that old, reliable Farmers’ Almanac doesn’t prevent the billions of dollars worth of crop loss that farmers experience each year because of inclement weather. The Climate Corporation thinks it has an answer to the problem, though, and that answer is big data.
The Climate Corporation (nee WeatherBill) has been a darling of the venture capital world — and the farming community — thanks to its deceptively simple model for protecting farmers against the weather. Farmers give Climate Corp. their growing plans and locations for the year, and Climate Corp. gives them customized policies based on the specific details of their situations. If a covered weather event happens — an occurrence that presumably results in some degree of crop loss — the farmer gets paid immediately. The company provides a fuller (and better illustrated) explanation here.
Under the covers, however, the company’s business is a tad more complex. Like, the kind of complexity that comes with being accurate to a geographic area as narrow as a farmer’s specific field, and granular down to the level of average soil moisture. According to Climate Corp. Founder and CEO David Friedberg, his company knows the shape of each of the 20 million fields in the United States, as well what was grown each year, the outcome in terms of crop yield, the water-holding capacity of the soil, and other critical metrics. To bastardize a classic idiom: You can’t make a weather omelet without cracking 30 years worth of data-rich eggs.
How much data are we talking about?
Here’s how much data Friedberg and his team of 21 Ph.Ds. have and how they analyze it:
- About 200TB of historical data stored in Amazon S3.
- Monthly predictive simulations and daily forecasts for each location it insures.
- Each simulation:
- Uses between 4,000 and 5,000 computing cores on Amazon Elastic MapReduce.
- Considers 10,000 scenarios for each of the next 730 days.
- Analyzes 5 trillion data points.
- Consumes 20TB of uncompressed data.
- Data volumes have grown about 15x since last year.
It’s worth noting, though, that even with all its data and simulations, Climate Corp. isn’t in the business of predicting the weather, just giving probabilities. “‘Predict’ is a strong word in weather,” Friedberg explained, denoting an ability to say with some certainty what will happen and when. Friedberg cares about statistics — determining the probability that something will happen over a particular timeframe, then analyzing the accuracy of those probabilities to further improve the models.
Farmers know their tech
But looking beyond the sheer computing power and data science that Climate Corp. puts behind its insurance policies, the most impressive part of the company’s business model might be that it actually gets farmers to use the web-based service to improve their output. Friedberg isn’t too surprised, though.
“Farmers are not just fly-by-night small business owners who decide overnight they’re going to start selling soda and candy,” he said. Rather, they’re often the products of generations of inherited knowledge and experience on how to grow the best crops possible, and the most of them. They know what it costs them to plant an acre and what they need to grow to make a profit. But “the one thing they can’t control is the weather,” he added.
If a tool can help them accurately account for the possibility of bad weather and thus maximize the volume of crops they produce, they’ll learn to use it. As the climate that farmers depend on continues to evolve, that might be more true than ever. A statistical mind through and through, Friedberg wouldn’t call it “climate change,” but did note a trend toward warmer temperatures and volatility of extreme weather events that makes it more difficult to get a sense of what might be coming.
And it’s not like farmers are the luddites that popular culture might make them out to be. “Now, they’re using GPS equipment in their self-drviing tractors and combines,” Friedberg said. “They’ll sit back and watch Hulu on their iPad while the tractor drives itself.” Apparently, they’ll delve into some big data, too.
Feature image courtesy of Roger Kidd.
Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.
- Infrastructure Q1: Cloud and big data woo enterprises
- Big data: beyond analytics
- A near-term outlook for big data
