The White House opens the data floodgates, and now the real work will begin

President Obama on Thursday signed an executive order making open data the default policy for the federal government. This is a hugely significant move, although one that can and will have its openness and thus, significance, chipped away over time. But it’s a good start.

While we may never get the full extent of government spying on citizens in machine-readable formats, the Census, FDA testing, EPA and myriad other data will offer a treasure trove of information for years to come. And by making it open and machine readable, it will presumably be harder to bury such data in physical warehouses or behind crazy fees.

The executive order is here. Essentially it requires the government to do the following:

Figure out what data the agencies have, and make an index
List all of their publicly available data in a public place
List all of the information that could be made public, even if it is not yet available

But the order attempts to address an almost existential question about moving from an organization where data is assumed TO BE hidden to one where it assumed to be open. John Wonderlich over at the Sunlight Foundation put it well in his blog on the announcement:

“Most importantly, though, the new policies take on one of the most important, trickiest questions that these policies face — how can we reset the default to openness when there is so much data? How can we take on managing and releasing all the government’s data, or as much as possible, without negotiating over every dataset the government has?

How can the public (or policymakers) request what they don’t know exists? How can CIOs manage what they haven’t surveyed?”

He concludes that this order will address a lot of these issues, and I hope that will actually happen. As someone who has submitted Freedom of Information Act requests only to get back boxes of redacted and almost meaningless documents, the hunt for government information — or information that is supposed to be publicly available — can be daunting, exhausting and ultimately fruitless.

Of course, as was shown when the SEC started making its records available online using XML, the greater visibility of those documents, notably the Reg-D filings that indicated a private company had picked up funding, prompted the agency to include less information in those documents. They still made them public online, but also made them less useful in some cases.

Also, this order notes that privileged information, law enforcement information, national security information, personal information, or information that agencies can’t disclose because it is prohibited by law, are all off the table when it comes to the order. This isn’t unexpected, but it can be used to create loopholes where agencies (or private companies working with the government) can attempt to hide data it doesn’t want to share.

But, as the White House release notes, government data such as Global Positioning System data and weather data have been open for decades and have helped create some awesome new services for citizens. I’m sure that today’s news will open up plenty of great data sets that entrepreneurs can start using to build amazing new apps.