Machine data is for people too

Machine-generated data, the non-intelligible zeros and ones that are generated by sensors and other devices, is no longer just for geeks.

Erik Swan of Splunk at Structure:Data 2012

The data itself, which can log files from servers, or from sensors attached to appliances or air pollution monitors, is usually “gibberish,” said Erik Swan, CTO and co-founder of Splunk, a pioneer in aggregating and processing machine data, at the Structure: Data2012 event in New York on Wednesday. ”It’s not designed for end users.”

But that doesn’t mean end users aren’t using it.

During last year’s Fukushima radiation crisis when people felt the Japanese government was not providing enough information about radiation levels, an ad hoc network of citizens worked together to connect Geiger counters and link them up to Pachube.com.

“Within 10 days, there were more than 1,000 feeds updating several times per minute,” said Usman Haque, founder and CEO of Pachube.com, the British startup that aims to make this sort of data more available and understandable to mere human beings.

But the data aggregation was just the start. People wrote apps — like the Android-based Winds of Fukushima app that melded geo location and wind data to predict where the radiation would spike next, Haque said.

“Some were domain experts and some were just people who were willing and worried,” he said, but the upshot was that consumers were able to use this data for a very important purpose.

Swan agreed that the world is at the beginning of that growth curve.

“This is like the paradigm shift around e-commerce years ago when users didn’t buy much at first but then Amazon came along and showed [e-commerce’s] power, then all companies became commerce companies and consumers expected everything to be available that way,” Swan said.

In the future, consumers will expect to have a “data relationship” with every company they do business with and to have a dashboard based on their prior transactions and other pertinent interactions to help them make decisions.

Some non-technology issues — like the ownership of all that data — still need to be resolved, however. ”Consumers are shocked to find they don’t own their data. In some cases someone else might own it and in some countries, no one can own it,” Haque said.

There is also a dearth of applications — the Fukushima app notwithstanding — that tap into that data.

Haque likened the current state of affairs to the 1995 “green-screen” era of computing . “We have no idea of what’s coming in 5 years… but we see people out there using these systems now, building services, even individuals automating their homes using the Nest thermostat. It’s just a question of providing discoverability into this world [of machine data] that’s already being built.

Watch the livestreamof Structure:Data here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.