When it comes to presidential elections, it helps to know your way around some disruptive technologies. The team of technologists that helped re-elect Barack Obama – led by Obama for America CTO Harper Reed and comprised largely of other political novices and accomplished hackers — certainly had that going for them. However, when the prize is the highest office in the land — and possibly the fate of the free world — it also helps to know your role.
A presidential campaign is not a tech startup; it has to innovate on tight deadlines and in an environment where failure really is not an option. So although it had to move fast, for example, Reed’s team couldn’t afford to re-invent the wheel because it could maybe shave 5 milliseconds off of page-load time for a web page.
Or, as Reed put it a phone call with me on Monday morning: “Our goal [was] to be the force multiplier, not to be a technology experiment.”
It was there to make sure the president’s foot soldiers — the folks who really do affect the results — could execute their ground-game without having to worry about technology failing them. Reed’s team just had to take the tools at its disposal and use them to their fullest extent so that old-world and potentially time-consuming techniques such as calling phones and knocking on doors were done as efficiently as possible.
For example, Reed explained, his team didn’t invent the tool for making and handling phone calls, but it gave that tool legs. “We made it so it could stand up and take all the calls,” he said. “Innovation was scale. Innovation was not falling down.”
Here’s how Reed and company fulfilled their duty.
Reed (center rear) and team on election day.
It made the cloud work for it …
Reed said the the vast majority of the campaign’s infrastructure was hosted on Amazon Web Services, with only the analytics platform and some “nominal” other pieces residing on physical gear. This was a fairly big change from 2008, when Obama for America was running only a few minor tasks in the cloud. A reliable cloud service is a game-changer, he said, because smart campaigns no longer have to worry about buying servers, engaging contractors or negotiating software licenses.
AWS, he said, was particularly helpful because of all the services it offers. In order to ensure visitors to President Obama’s website had the same experience wherever they were located, the team decided to “light up boxes all over [the country]” and use the CloudFront content-delivery network feature to route data from the closest virtual server. When it wanted a key-value store, it went to DynamoDB.
As for that phone-call tool and the millions of calls it has to support, Auto-Scaling, a method for adding additional resources in mere minutes, saved the day. “We used the hell out of it,” Reed said. When volume was higher than expected, he added, “we could just turn that to 11.”
… even when the cloud crashed
But relying so heavily on the cloud wasn’t always easy, because the cloud isn’t always reliable. Amazon’s cloud went down a few times between June 2012 and election day — including as late as Oct. 22 — and brought a lot of web properties down with it. “Both times we survived,” Reed said, “but it was hard.”
Obama for America stayed online because it embraced devops (the tight alignment between application development, engineering and operations that cloud computing enables) and the smart architectural strategies employed by cloud computing pioneers such as Netflix. (Paraphrasing a discussion thread from popular programming site Hacker News, Reed noted that if a developer can still use Netflix while AWS is down, then the developer screwed up. If Netflix is down, then AWS is really down.) That means spending a little more money to replicate databases and applications across geographic regions and generally being smarter about where the various application components run and how they’re connected to each other.
AWS gives you tools to do amazing things, he explained, “but you’re responsible for screwing it up.” At the beginning, he added, everything might be chill and people are working on how to creatively architect a reliable cloud application, but eventually “you start valuing uptime over experimentation.” At that point, you just have to accept the gravity of your mission and say “we’re going all in.”
Reed thinks keeping applications up in the cloud will get easier thanks to some of the innovative companies and technologies working in that space. He pointed to cloud-management company RightScale, the Tungsten Replicator engine for MySQL and the general platform-as-a-service space as things he thinks are really cool but weren’t quite ready for primetime in 2012 — at least for something as deadline- and mission-critical as a presidential campaign.
It understood how the web works
Aside the from finer points of cloud computing infrastructure, Reed said Obama’s tech team also really understood how the web has evolved since 2008. Take, for example, the advent of Twitter and Facebook as forces to be reckoned with in terms of voter engagement. Then-candidate Obama got a lot of attention for his 5 millionish social media connections, but President Obama now has roughly 10 times that many across Facebook and Twitter. That’s a lot of people from which to spark a network effect, and a lot of data to analyze.
“Being able to work at that scale is amazing,” Reed said.
And it’s not the just the president’s follower counts that have grown — social media platforms have evolved significantly and people are far more familiar with how they work. Facebook Pages, for example, were created after 2008 in part because having profile pages for candidates and companies didn’t work out too well, and Facebook Connect didn’t exist either. Twitter has also grown into a widely popular platform, while MySpace is all but gone. Reed said it was critical that team Obama take the platforms built in the four years between elections and use them the way they are being used today.
Or take the details of how people access those sites to begin with. The president’s technology team knew (no doubt thanks in part to its analytics team that has received so much attention post-election for its data mining efforts) that voters in urban areas where Obama was counting on high voter turnout were more likely to use a mobile phone rather than a laptop as their primary means of internet access. They’re also more likely to use Android devices than iOS devices, as are many potential swing voters who generally don’t care too much about technology. So, Reed said, the team designed apps to run on multiple operating systems and used responsive design to ensure apps ran well on whatever devices voters were using.
It disrupted with data
Reed’s team worked closely with that vaunted analytics team, he said, and what the Obama for America team really did better than Mitt Romney’s team was disrupt the status quo with regard to how it used data. He’s not too keen on jumping on the bandwagon calling out the failings of Romney’s Project Orca (“I never would wish technology failing on any sort of opponent or enemy,” Reed said) but he will acknowledge that the analytics team Obama had been putting together for more than a year leading into the election was a major differnentiator.
He compares the advanced modeling and analytic techniques of his comrades and guys like Nate Silver to MP3s, thus making traditional pollsters and political “experts” akin to the music industry. They both had been going about their business for decades without competition, and they both reacted violently when their worlds were disrupted. However, these are smart people, and Reed expects they’ll come around in the next election cycles.
But for now, he said, “That’s a hard position to be in when you’re no longer relevant.”
It had a great leader
In the end, though, Reed thinks the success of Obama’s campaign all boiled down to having a good leader. It wasn’t just Obama’s good fortune of running in during the era of Twitter and Amazon Web Services that let him forever change the way campaigns are run. The president is the one who set the tone about how the campaign would function and what it would focus on, Reed said, and the president put in place the campaign leaders who followed through on his vision by hiring the right people down the chain.
If Reed does come back to politics, it will take another candidate with Obama’s vision and ability to generate excitement among the populace to get him back in the saddle. But for now, Reed said, “I’m definitely ready for a little break.”
Feature and Obama for America photo courtesy of Harper Reed.
 |
 |