There are lots of events that enjoy near-universal attention from both the media and the general public: the Super Bowl, the World Cup, and last year’s royal wedding, to name a few. But there’s only one kind of event that brings the most sophisticated live web-publishing tools to their knees: an Apple event.
Not Again
As Apple unveiled the latest iPad in San Francisco Wednesday morning, a familiar sense of dread began to invade newsrooms around the tech media world — the live-blogging tools were breaking. Apple does not allow or provide live video streams of its trademark events, and so the updates from those events are breathlessly dictated by a few dozen or so publications and published “live” alongside still photos of executives demonstrating the products.
These tools are notoriously unreliable when it comes to Apple coverage. Cover It Live, one of the first off-the-shelf live-blogging tools now owned by Demand Media, has often struggled under the weight of Apple events despite years of experience. It did so again on Wednesday, forcing Erica Ogg, our Apple correspondent and woman on the scene, to abandon the service in favor of posting old-school updates to a WordPress blog post that required readers to manually refresh the page to see new entries.
ScribbleLive, which builds blogging modules for publishers and provides back-end services for customized live blogs from the likes of CNET, The Verge and Techcrunch, also struggled under the load. The Verge’s editor-in-chief Joshua Topolsky was clearly incensed at the problems, tweeting at one point: “Sorry guys. ScribbleLive is screwed. We won’t be using their service again.”
It’s a maddening situation for web publishers: reader demand for these live blogs increases every year, but only those editorial organizations with ample resources and the right technical expertise can afford to build out their own live-blogging infrastructure. Most others have to rely on third-party services that seem able to accommodate nearly any other event with aplomb, yet time and time again are flattened by demand for Apple coverage.
What Happened
Cover It Live, which failed entirely under the weight of demand for iPhone 4S news back in October, said it experienced “minimal 1-2 minute delays” during Wednesday’s event, which is laughable to anyone who was sitting at our editorial table Wednesday morning at GigaOm’s office in San Francisco. In addition to being quite slow, the service was posting updates inconsistently: At one point, my colleague Janko Roettgers and I sat side-by-side comparing the live stream our on laptops and watched dumbfounded as Erica’s updates appeared in different order on our respective screens. Some updates were eight to ten minutes late, and test comments left by other colleagues were even later.
In a statement, Cover It Live said “traffic yesterday was around 2-3x that of the iPhone 4S event. Traffic across all of the types of events we cover varies widely, but Apple events generate around 10x the traffic of other events. The audience and events also converge in an extremely short period of time, which creates a burst of more like 50-75X normal traffic.”
Demand Media’s Steve Semelsberger, who runs the division that oversees Cover It Live, said in an interview that about 2 million people read Cover It Live blogs on Wednesday, as compared to the 1 million that read its blogs during the Academy Awards. Apple events create what he called “a front-door issue,” in which millions of readers burst into live blogs as if they were running to grab choice seats at a concert, logging in or firing up live blogs within the same 10 to 15 minute period.
Cover It Live users had very different experiences on Wednesday, he said, ranging from the debacle that we experienced to more stable performances on sites like Macworld. Jason Snell, longtime Macworld editor-in-chief, confirmed that his site’s live blog performed much better than in past experiences with Cover It Live, however “I wouldn’t call it ideal — I think it dropped some of our entries and definitely frustrated us — but it didn’t crash entirely, which was something.”
Engadget’s Tim Stevens was one of the few tech media editors who only had to worry about covering the event itself, as opposed to managing the failure of his tools. Engadget’s approach was home-grown, built atop the Blogsmith platform with assistance from parent company AOL, and “the basic idea is it gives the ability to instantly view the action in real-time, separating text from photos so that neither overwhelm the other,” Stevens said. Although the final numbers were still being tallied, the end result was what Stevens believed was the single biggest traffic day in Engadget’s history with no blog downtime.
Mea Culpa
For its part, ScribbleLive apologized on its corporate blog for its performance. “Since the last major Apple event in October, we devoted significant resources to prepare for today. We believed we solved the problem. We didn’t,” wrote Michael Del Monte, CEO of ScribbleLive, in a post Wednesday. Del Monte said that a configuration setting involving how ScribbleLive’s databases connect to its Web servers was not set correctly to handle the traffic associated with an Apple event.
This image shows the load on ScribbleLive’s servers during Wednesday, March 7th, when Apple unveiled its latest iPad.
Del Monte told me in an interview Thursday that it’s nearly impossible to predict the traffic load that can accompany an Apple event. “How do you simulate hundreds of thousands or millions of people hitting your web site in real time? There’s not a lot of service providers that can simulate those traffic levels,” he said, nevertheless promising to find a way to simulate those types of traffic levels–for ScribbleLive, 50x to 100x normal event traffic–before the next Apple event. My colleague Derrick Harris has a few suggestions: companies such as Soasta and Mu Dynamics offer those kinds of performance-testing tools, although it’s still hard to know exactly how much interest an Apple event will generate.
Even the big guys can be knocked off rhythm by a huge event: when reports of Michael Jackson’s death first appeared on June 25, 2009, and millions flocked to Google in hopes of confirming those rumors, Google’s servers briefly treated the influx of traffic as a massive denial-of-service attack on its infrastructure.
“At the end of the day, we are a startup that is trying to handle traffic that one of these larger organizations have to handle,” Del Monte said, comparing the problems his company is enduring to those experienced by Twitter during the days when the “fail whale” became an Internet icon.
Twitter used to go down reliably during Apple events, but the site has dramatically improved its reliability even under the load of events like Wednesday’s iPad event or a Japanese television show called “Castle in the Sky,” which saw 25,088 tweets per second during a pivotal episode. During the fourth quarter of this year’s Giants-Patriots Super Bowl, Twitter recorded around 12,000 tweets per second.
So, Twitter?
At one point Wednesday morning, frustrated at the current state of live-blogging tools and all too aware of how much pressure live blog writers and editors are under even when the tools work properly, I posed a suggestion on Twitter: “So, @twitter, you need to make a live-blogging tool.” Apparently that was a little too broad a statement for those who haven’t covered an Apple event for a big tech media site, a number of whom derisively snarked that Twitter itself is a live blogging tool.
Tweets per second over the course of March 7th, 2012, containing the word "iPad"
It certainly is in one respect, but professional publishers need more than what Twitter currently provides. You can embed Twitter updates from reporters through a widget and put that into a blog post, but posting images in line with those updates is not easy, and of course the updates themselves are limited to 140 characters. This approach also doesn’t allow publishers to differentiate themselves from the competition with their own branding and custom features, which hardcore readers of technology live blogs have come to expect.
Twitter has (more or less) solved the hard part: the back-end reliability from an infrastructure that was designed to accommodate real-time streams of traffic as opposed to serving up static Web pages. Were it able to design a professional product that could allow sites to customize the front-end appearance while taking advantage of Twitter’s infrastructure, its sales people could book appointments with the leaders of every major technology publishing organization for a demonstration and charge real money for the product and associated services.
So long as Apple events continue to drive massive traffic to tech publishers, there is a glaring need for a more reliable product and a business opportunity that is totally up for grabs. But for now, tech media professionals and live-blogging service providers will continue to approach Apple events with both anticipation and dread: When you slip and fall on the biggest stage in tech publishing, people remember.
Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.
- Forecast: Tablet App Sales To Hit $ 8B by 2015
- Web startups: How to guard against security breaches
- Personal tools lead to practical business
