The cloud is a wonderful thing, as you probably know from reading our coverage on the Structure blog and posts from my colleagues Stacey and Derrick — among other things, it allows us all to keep almost unlimited backup copies of our documents, photos, music and other files on a remote server somewhere, and thanks to services such as Amazon’s AWS suite, such backups are pretty cheap as well — pennies a gigabyte. At least, they’re supposed to be. But when the cloud meets an Internet service provider’s bandwidth cap (something that is unfortunately becoming more and more commonplace) it can be a less than happy experience.
I know this all too well. And while some of what happened is my fault, it’s probably not that unusual, so I thought it might be helpful to tell people about it.
Some readers may recall that I wrote recently about my problems with bandwidth usage, and how I thought I had solved them. In a nutshell, my ISP — a Canadian cable and media conglomerate called Rogers Communications — started warning me that my household was using huge amounts of bandwidth, far more than I had ever used before. One particular day last month, the online bandwidth meter showed that we had consumed 75 gigabytes of data, more than three-quarters of our 95-gigabyte allotment for the month.
As I described in my last post, I wracked my brain to try and figure out where this could be coming from. At first, I (and the Rogers technician I spoke to) thought that it was our wireless network, which was unencrypted. So I locked it down with a 64-bit password — but the downloads continued at huge levels, sometimes 20 or 30 gigabytes a day. I interviewed all three of my daughters, aged 13 to 21, and my niece, who is living with us. All denied downloading huge amounts, but one daughter said she had been using a Bit Torrent program for some Japanese TV shows.
I checked every computer in the house to make sure there were no programs running in the background or viruses or malware — we have seven computers in all, including four desktops and three laptops, as well miscellaneous wireless devices like iPhones and a streaming media box connected to the TV. The download usage didn’t even budge. I even borrowed a network switch to try and see which ports on the network were using all this data.
Then a day or two ago, I got an email from Amazon with the bill for my Amazon AWS service. I have about 25 gigabytes of photos, music and other documents backed up to Amazon’s S3 server cloud, which usually costs me about $ 3 a month — but this time, the bill said $ 109. Suddenly, a light bulb went on (one that probably should have gone on before then, I admit): that would explain how we could have been downloading 20 or 30 gigabytes a day without our computers overflowing with data. What if something was backing up all that data multiple times from Amazon’s S3 cloud?
I checked JungleDisk, which is the software from Rackspace that I use to manage the S3 instance I rent, but I didn’t have it set to sync or automatically mirror the data. So what could have happened?
After fishing around on the Internet, I finally found a support forum posting at Amazon’s AWS site that described exactly what had happened to me — except it was the user’s mother, and she had somehow download more than 800 gigabytes in a month without realizing it. The culprit, apparently, was either Windows indexing the files or an anti-virus program scanning them, or both. Since JungleDisk maps the Amazon cloud folder as a network drive, Windows and some other programs simply treat it as a regular drive and download all the files to scan them — even multiple times.
Since the amount of data that Amazon says I consumed in April (a little over 600 gigabytes) is almost exactly the same as the amount of extra bandwidth that my ISP says I used in the month, this seems to be the solution. And luckily for me, Rogers has a maximum bandwidth over-usage charge of $ 50 per month. If it didn’t have this cap on a cap, I would be liable for almost $ 1,000 based on the company’s per-gigabyte usage fees. So I have to pay $ 100 to Rogers for overage charges and $ 100 to Amazon.
A fairly cheap lesson, comparatively speaking, but a lesson nevertheless. And yet another way in which bandwidth caps threaten to turn us all into network administrators and bandwidth cops, whether we like it or not.
Thumbnail photo courtesy of Flickr users Ryan Franklin and Arthur Caranta
Related content from GigaOM Pro (subscription req’d):
- Report: Delivering Content in the Cloud
- Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
- Defining Hadoop: the Players, Technologies and Challenges of 2011
