Beware Of The Spot (Amazon EC2 Pricing)

If your service is running on EC2, you probably already know about Spot Instances.  But for those of you who don't, this is a service where Amazon allows you to bid on unused capacity in their data-centers.  The pricing varies based on supply and demand, but is typically somewhere between 25% and 50% the cost of a "reserved" instance (e.g., right now, a "Small" on-demand instance is $0.085/hr, while the spot cost is $0.027/hr).
Amazon has done a lot of work in making these very easy to use, everything from the bidding, to the setup, to VPC integration. As you might imagine, this has not gone un-noticed, and there are any number of companies that have gone the spot right towards their infrastructure - using them to soak up the load associated with excess capacity. You even have guides out there on how to setup and use spot instances, a sure sign that its ubiquitous.
So, whats the catch?
As it turns out, the correct question is, what are the two catches?

1) Dumb Users
To put this very simply, just because the spot prices are cheap now, doesn't mean that they will always be cheap. Users, for whatever reason, assumed that the spot rate could never go above the on-demand rate.
<Bzzzzz>Wrong</Bzzzzz>
Last time I checked, there was nothing that capped spot prices.  Supply and demand, and a resulting price

2) It is not a Market
You have to remember that unlike a market, there is no true price discovery here.  You have one seller (a monopoly), and the seller also owns the exchange (and as the financial crisis should have taught everyone by now, that is a Bad Thing.  Apparently that lesson has gone un-noticed).  The fact that it looks like a market is probably a "feature" on Amazon's part, but it is most certainly not inherent in the pricing!

Jonathan Boutelle (of SlideShare fame) has a writeup on GigaOm about this. The lessons are remarkably obvious, but hey, sometimes it is the obvious that you don't pay attention to. 

Never EVER bid more than you are willing to pay for a server on the spot market. This is the most important lesson. Don’t even bother doing “convenience bidding” of double or triple the on-demand price: when the price starts to spike it will easily go way beyond any rational price. Do you want to be the gal who explains to the CEO why the company is paying $100 an hour for servers?

Don’t run all your infrastructure on spot market machines. In fact, don’t run more infrastructure than you are prepared to lose on spot machines. We use a thumb rule of 50 percent at SlideShare, since our system can easily survive 50 percent of our machines disappearing at one time (which is what will happen during a price spike).

Write the code that manages your cloud infrastructure so that it responds intelligently to spot market price spikes. If you can’t get a spot machine at a reasonable price, your code should automatically request an on-demand server.

Write the code that manages your cloud infrastructure so that it responds intelligently to spot market price spikes. If you can’t get a spot machine at a reasonable price, your code should automatically request an on-demand server.