PANOS IPEIROTIS opened an e-mail from Amazon Web Services in April to let him know that he had racked up $720.85 in charges so far that month. A professor at New York University’s business school, as well as an entrepreneur, he regularly consumes bandwidth and virtual-computer rental time across multiple web services run by Amazon.com, but never quite so much. He checked his live balance shortly after the e-mail had been sent and found charges had already topped $1,000, mostly for data transfer to the tune of nine terabytes over a mere day and a half. Worse, charges continued to accrue at a rate of $30 an hour, the result of about 250 gigabytes of data transfer during each interval.
So what went wrong? In a nutshell, Mr Iperiotis became the victim of the internet’s growing complexity, where individually benign systems all too easily metamorphose into a malicious bug. More specifically, he almost got squashed in an unexpected conflict between Amazon and Google software.
Amazon provides accounting of data transfer consumption by directory in its Simple Storage Service (S3), which offers web hosting to anyone from individuals to corporations. That let Mr Ipeirotis narrow down the problem to a folder he uses to contain automatically updated charts derived from work performed at Mechanical Turk, Amazon’s automated system for assigning tasks to human beings and collecting the results (about which Babbage wrote a year ago).
When this did not suffice, he turned on a more detailed log to grab information about the individual files requested by a remote party, and discovered that 250GB was being sucked up every hour by Google’s “Feedfetcher“. This Google agent was designed to retrieve RSS feeds, the specially formatted web documents that maintain a list of the latest news items or updates for a blog or some other site. RSS feeds are constantly refreshed, and are typically checked for updates every hour.
The problem is that, by default, Amazon’s S3 service does not automatically set much metadata, as information about a given file’s attributes is known, for uploaded documents. Feedfetcher, meanwhile, relies on information that most web servers provide: how long the retrieved file should be stored, or “cached”, before it discarding and retrieving a new copy. Without it, Feedfetcher presumed that it should never store the file. This metadata can be set by an Amazon S3 user either manually or through program code when a file is uploaded. (For wonks, there are at least four metadata items that relate to a file’s caching: an explit expiration date, a directive on how long to cache, a timestamp of last modification and a unique file tag that changes whenever a file is updated.)
Mr Ipeirotis realised that he had used Google Docs to create a spreadsheet which contained links to images stored in his Amazon S3 directory. The file would, he thought, only be updated with any new images when he viewed it. Instead, with the spreadsheet open on one of his machines or one of his colleagues (he’s unsure of which), Google automatically examined the cached images each hour to update the document. Because the Amazon S3 chart images lacked information to tell Feedfetcher to retain them, Google retrieved new copies each time.
The mystery solved, Mr Ipeirotis removed the image references from his spreadsheet and changed access controls to deny retrievals of the S3 files in question. In the meantime, Google gave him some technical advice, while Amazon waived his bandwidth bill as a courtesy. A few weeks later Amazon updated its CloudWatch service used by web-services customers to monitor their online activity. It now includes options to send e-mail, text messages and other alarms when users cross thresholds they set for assorted charges; Google seems to have no plans to tweak its Feedfetcher agent. Either way, expect more unintended consequences.
Powered by Facebook Comments