Search as Twitter historical archive? More like Twitter’s greatest hits

Between downloading the archive of all your tweets ever and entering search queries to bring up more-than-a-few-days-old tweets, it would seem like the world, or the historical twittersphere at least, is your oyster. But in reality? It’s a little more complicated, and a little less complete than that.

Twitter announced last week that historical tweets would now be included in search archives, whereas in the past only tweets from the last few days would appear in results. Twitter has been unfolding its search ambitions over the past year, figuring out how to surface relevant information in a search product that for a long time, hasn’t been all that relevant.

“Now we’re able to look back at old tweets and think about how they should look and how people can relive past moments on Twitter,” said Sam Luckenbill, a senior engineering manager in charge of search at Twitter. “So you can find specific events in the past that weren’t previously available.”

Most of the company’s opportunities in search lie with real-time search, since companies like Oreo want to take advantage of current trends, like the Super Bowl, rather than old news, and Twitter’s clearly betting on the service’s value as a second screen to television with the Bluefin acquisition. But a more complete, historical search provides serious opportunities for the company around monetization (marketers can purchase promoted tweets for specific search terms) and intent (knowing that someone is looking certain information is more valuable than passively viewing something, even if you obviously intended to follow that brand or person.)

But even as the company is now including historical tweets in search results, it’s not all tweets ever that you can uncover — it’s really only a small percentage. The company has worked to develop an algorithm that will figure out which are the most interesting tweets that people would want to revisit, and then ranks those higher than potentially less interesting tweets.

While the company naturally wouldn’t disclose exactly how the algorithm was put together, they said Twitter is attempting to figure out both the most relevant tweets for a specific person and search term (so likely based on your interest and friend graph) and the most engaging tweets overall (which could be reasonably understood as those with lots of retweets, replies, etc.)

“You’ll see tweets from people you care about about and topics that you care about even if the word you chose was a bit vague,” Luckenbill said.

And the filtering doesn’t just apply to tweets that come up in your specific search results — it goes for the tweets Twitter chooses to make available to all searchers as well.

“We’ve started with a small percentage of tweets, looking at the ones that have been engaged with most. And then we plan to steadily increase the size of the index over time to capture more and more,” Luckenbill said. “We’re going to make more and more tweets available that are older, but it’s not necessarily that we want all the tweets ever to be in the index. Because we don’t want to put tweets that get no engagement ever in the index.”