Twitter over capacity and SimpleDB Errors

2010 April 5 at 15:40 » Tagged as :twitter, cloud, january effect,

As I write this, 'Twitter is over capacity'. The fail whale as it's popularly known is flying with the help of a few tweety birds. In between reloads the broken robot (or is that an  alien) with the 'Something is technically wrong' message also pops up. By quite a coincidence, simpleDB, Amazon's cloud hosted nosql solution is throwing up intermittent errors. There was a running task on one of our server that was attempting to movie about 11 million records from a Mysql database into SimpleDB, every 50,000 records or so the underlying API produced the following error:

PHP Warning: SimpleDB::batchPutAttributes(): ServiceUnavailable: Service AmazonSimpleDB is currently unavailable. Please try again later in /var/www/****/SimpleDB/sdb.php on line 1021 Items FAILED ErrorCode: ServiceUnavailable

However at about the time that twitter went offline, we noticed almost every other insert producing an error. When that happens we have coded our script to sleep a few seconds and retry the insert. The second attempt was completed successfully almost everytime. I went to tweet about it to find that twitter has gone offline. AFAIK, twitter has not publicly stated what their database platform is. They might do so at chirp so for the moment I am going to hazard a guess that they are using SimpleDB to store some of their data. All their static content is served out of Amazon S3 so this is not a wild guess (I hope). I am now going to leap to the conclusion that the two errors are related and make a fool of myself.