Cloud benchmarks show smaller providers coming out ahead — but they’re still benchmarks

Big-name cloud service providers capture the tech-press headlines day after day, but lesser-known players might deserve more recognition, at least based on new benchmarks out from Compuware.

The company found that, although Rackspace and Windows Azure were consistently fast in taking an application request, processing it and delivering it back, a handful of competitors are actually faster and more consistently so, while other providers are available more often. The figures from Compuware represent the average response times over a year of monitoring performance on 38 cloud facilities, as part of Compuware’s normal Global Provider View cloud-benchmarking application.

Comparing average response time, consistency (as a standard deviation of response time) and availability, Compuware has found that three of the top five data centers are run by Layered Tech (in Illinois, Missouri and Texas). There’s also Qube with its New York data center and Connectria’s Missouri data center. Altogether, Connectria came out looking the best. It ranked fourth for speed, eighth in consistency and second in availability.

Cloud providers ranked from left to right, when factoring in average response time, consistency (standard deviation of response time) and availability. Source: Compuware Global Provider View

Rackspace took sixth place, while Azure was No. 9. As for Amazon Web Services, its EC2 service from Northern California and Northern Virginia data centers came in 26th and 29th overall, respectively.

Benchmarks don’t guarantee that companies will get the same results. Far from it — application type, price and other variables are in play, and interested parties would do well to run their own tests, as my colleague Derrick Harris explained in a 2011 GigaOM Research article that explores similar benchmark reports (subscription required). Another moving part is the number of features available in each cloud. AWS, for example, has DynamoDB, Redshift, CloudFront and others, which makes it more like platform than just a bunch of servers to use.

Even with these variables, Stephen Pierzchala, application-performance management technology strategist at Compuware, drew some insight from the data analysis. “If you’re just looking at pure performance, you can look at other vendors, rather than simply the names that you know,” Pierzchala said. “There are other cloud providers that seem to be delivering a standard application with pretty good performance, and it may fit into our budget, or may fit into our business needs a little bit more in terms of delivery.”