It's a good thing I have long hair because it's been a week of tearing at the hairs. Short ones at that.
First on the 8th of june we had a spike of traffic and one of our servers crashed. This server onfortunately ran the database for the shopping cart. If that wasn't bad enough the next day there was a huge power problem at the colocation facility and all the machines in the building were constantly restarting.
Unfortunatley one of their routers didn't restart, it went out and stayed out. That meant even the machines that stayed up (such as our own) were inaccessible. Now colocation facilities are supposed to be immune to power supply and network problems but as we all know 99% uptime is a big lie.
Apparently the service that we use for payments have a very lousy hosting company. Frequent outages are the story of the day. The bad news is
that they had another one of those on the 11th and did little about it till the 13th. They claimed it was intermittent. Well their server came up intermittently.