Wanted: Data Scientists To Turn Information Into Gold

The fact that Foursquare is looking for a data scientist, first reported earlier this week, is hardly news. They’re just pursuing what many companies will eventually do as they try to tap the river of data they’re generating to improve their products or build new business opportunities. Indeed, data scientists will increasingly becoming vital employees as companies create and use more and more data.

Before we move on, what is a data scientist? Hilary Mason, a data scientist at Bit.ly, has a good definition. It’s someone who can obtain, scrub, explore, model and interpret data, blending hacking, statistics and machine learning. It’s a set of skills that go beyond many existing job titles and it’s increasingly in demand. Here’s a look at some recent news on the data scientist front:

Foursquare posted a job listing for a data scientist with “experience with prediction or recommender systems, search and ranking algorithms, and classification algorithms.” The job would seem to support Foursquare’s move into local recommendations.
Earlier this month, LinkedIn lured away Google Scientist and Tech Lead Daniel Tunkelang to work on data projects.
And in the last day, we’ve seen some great visualization work by a Facebook data scientist showing the world map of social connections, something Mathew wrote about.

But data scientists won’t just be important for social networks sitting on mountains of user data. Many companies that deal with consumers are able to generate a lot of data on users using analytical and consumer behavioral tools. And with data-as-a-service providers and others able to provide additional datasets, companies are able to work with a lot more information. The key will be digging deep into all that data and figuring out how to apply it to maximize revenue and improve products.

Katie Tucker, a senior partner at executive recruiting firm Korn/Ferry, said the hunt on data gurus, czars and architects has kicked into high gear in the last year. She said there was a 200 percent increase from 2008 to today in searches for executives with sophisticated data mining or data analytics capabilities. As companies emerged from the recession and expanded their marketing budgets, they started to look for more efficient ways to build their businesses and become even more relevant to their customers. And with newer data warehousing appliances and management and processing tools such as Hadoop helping companies capture and manage more data, executives have looked to the information right under their noses for insight. Tucker said that, in turn, has prompted a big upswing in searches for data professionals, similar to the way the Sarbanes-Oxley Act’s reporting requirements elevated the importance of auditors.

“This is the first time in my 15 yeas I’ve ever seen data as such a big focus,” for companies, said Tucker. “Just as with Sarbanes-Oxley, you have something that was good discipline and good business becoming front and center.”

Todd Levy, CTO of URL-shortener Bit.ly, said companies have built out a lot of more obvious products without having to delve deep into data. But now, the challenge is to get better insights out of the data emanating from a company to help steer it toward new opportunities. And that requires more data knowhow. “We’re moving into a area where the low-hanging fruit has been culled and you need to take advantage of more advanced resources now,” he said.

Over the last year, location based service and ad network Where has built up a data team of 12 people, made up of quantitative analysts, PHds and data scientists. CTO Ivan Mitrovic said the ramp-up was necessary as the company launched its own mobile ad network. The team has not only made sense of the 3 billion monthly actions on the ad network but they were also able to point the company toward a new business: a local discovery engine for consumers that recommends places to go. That’s the value of data scientists said Mitrovic, in helping turn raw data into new business opportunities. “When you give them the data, they find products in the data that you didn’t know existed,” he said.

Foursquare’s Dennis Crowley seemed to echo this point recently at the AllThingsD mobile conference, where he seemed almost unsure about what Foursquare will turn into. Why is that? Because all that data could help the company transition in a number of ways. It could be as I wrote earlier about just being a better recommendation engine for local places. But the data could help Foursquare become a larger “crowdsourced city guide,” ”a social utility that intersects with the real world,” or just a “stats engine.” That’s what his data scientist will help him sort out.

Roger Ehrenberg, founder of IA Ventures which focuses on Big Data startups, said the promise of all this data is creating highly personal experiences for consumers, anticipating what they want and providing it to them. He said successful companies will lean on data scientists to start delivering contextual and relevant information to users, without them having to ask for it.

“I think one of the things underlying this theme is the issue of a user’s ability to be passive but have an experience created for them on the basis of this data,” he said. “Companies will look at what’s happening, who’s around you and will intuit by time of day and your interests and will serve up contextual offers and content.”

Now to be sure, many companies won’t need a data scientist right away. Most just don’t deal with the kinds of volume that necessitates such a hire. But make no mistake, this is where the world is going. With the price of processing, storage and broadband dropping and data tools emerging, it’s easier than ever for companies to wade into Big Data. And as they do, they’ll need more employees to make sense of it all and show them the way.

Related content from GigaOM Pro (subscription req’d):