So The Site With the LAMP had three outages in three days. Did I say three in three? But my previous post mentioned only two? Sorry about that. There was another one immediately after that was publish. In all three occasions downtime was caused by a surge in demand for Amazon EC2 spot instances.
The chart shows several surges since the 27th of September. On the 29th, the price of an M1.large instance (the kind of virtual machine used to power this site) shot upto USD 15.00 per hour. Usually these are priced at USD 0.12 per hour, so it represents a 125 fold increase!
I had set the upper limit for my instance at USD 2.00 per our thinking that a safe limit. And it really ought to be safe because the 'on demand' pricing is only 0.34 per hour. How in his right mind would pay slightly more for the some service when the same service is available by the same company for a much lower price? I am pretty sure a few sysadmins and network managers would have lost their jobs. Some people would have burnt up a months hosting budget in just in just five hours!
While you might think that it's small businesses, free lancers or bloggers who rely on amazon spot instance, the truth is very different, even foursquare and many other large websites and web apps are running on Amazon EC2 spot instances. It looks like there might have been a DDoS attack on such a company. Whoever the real target was, I am pretty sure that they managed to keep their service online at great cost (or we might have heard about it). But looks like the attackers left behind a hell of a lot of collateral damage.
- Program mode requires special privileges, e.g., root or TrustedUser. »
- « Site Offline Twice in Two Days