How Reed Hastings’ busy 2005 winter vacation led Netflix to embrace big data

Netflix may owe its love of big data to CEO Reed Hastings’ 2005 winter vacation: Businessweek took a detailed look at Netflix’s history and the strategic tech decisions the company has made over the past few years this week. One of the previously-unreported tidbits was related to the Netflix Prize, which the company used to encourage researchers to develop a better recommendation algorithm a few years ago. Turns out that was a direct response to the way Hastings spent his holidays eight years ago.

Apparently, Hastings disagreed with his engineers about the best way to serve up recommendations. He believed that Netflix could just recommend new DVDs based on the star rating people gave movies. As in: Want to watch a new movie? Then check out these titles that others with similar interests have rated highly. His staff disagreed, and wanted to look at a whole range of other indicators, including the things people searched for on Netflix’s website.

From the story:

“Hastings spent two weeks over his Christmas vacation pounding away on an Excel spreadsheet with millions of customer ratings to build an algorithm that could beat the prediction system designed by his engineers. He failed.”

Of course, that lesson – more data is better – has been a key part of Netflix’s streaming business. The company is tracking all kinds of usage behavior, including every time a subscriber pauses or skips a movie, the order in which titles are consumed and more.

Businessweek reporter Ashlee Vance goes on to say that Hastings’ failed holiday hackathon led to the creation of the Netflix Prize. This $ 1 million competition pitted teams of researchers against each other with the goal of improving the Netflix algorithm by at least 10 percent. The irony is that Netflix never actually used the winning algorithm, because it had shifted most of its efforts towards streaming by the time the contest finally concluded.