Want to have the job of telling developers that “their baby is ugly”? That’s what researchers on Bing’s experimentation platform do, according to a post by Dr. Ronny Kohavi on the Search Quality Insights blog titled “Large Scale Experimentation at Bing“. The post introduces a research paper, “Online Controlled Experiments at Large Scale“, to be presented next week at an international conference on Knowledge Discovery and Data Mining in Chicago that illustrates how Bing uses large scale experiments, which can be a humbling process, according to the blog post:
When ideas are evaluated objectively in a controlled experiment, less than a third move the metrics they were designed to improve, and in an optimized domain like Bing, that number is lower. Humbling! We joke that our job in building Bing’s experimentation platform is to tell our developers and program managers that their new baby is ugly.
We’ve long noted that different users see a different Bing, as we get reports from around the world on features that may or may not show up elsewhere. Indeed, according to the blog post, Bing’s use of controlled experiments are growing exponentially, and Dr. Kohavi quotes Mike Moran’s book Do It Wrong Quickly – How the Web Changes Old Marketing Rules: “If you have to kiss a lot of frogs to find a prince, find more frogs and kiss them faster and faster”.
In fact, Bing is conducting over 250 tests on any typical day. Here’s a chart from the blog post showing the rise of experimentation over time:
If it feels like there’s no single Bing any more, well, there isn’t:
Almost every user is in some experiment when they visit Bing. In fact, almost every user is in about 15 different experiments. There is no single Bing any more, but rather each user falls into one of the 30B possible experiment combinations.
Like many companies, Microsoft is making more and more use of “big data”, both for its own purposes and as business intelligence services it’s selling to other companies. Simon Bisson over at CiteWorld wrote today about one example of a company using Power BI for Office 365, an Excel add-in that “provides everyone with powerful new ways to work with data in Excel and Office 365. It’s pretty clear that the era of big data is upon us, and the “old days” of beta testers and manual feedback are becoming a thing of the past.