High Performance SQL @ Google

There is a paper out by Google on how they migrated from MySQL to F1 for AdSense.
Its fascinating reading on its own, but its got some key take-aways for scaling and reliability.  I'd recommend going and reading it, but these two are worth emphasizing

Scaling

Eliminate the R in ORM.  In short, instead of building out a nice layer (hibernate, whatever) between your DB and your code, go the other way, and deliberately expose the workings of your DB to your application developers. 
This actually makes tremendous sense - in many ways the advent of NoSQL has been due to people specifically picking data stores that map to their application requirements.  Need a key-value store? Use Riak.  Need a document store wht guaranteed writes?  Use CouchDB.  Hiding the specifics of your data store behind an ORM layer is - increasingly - becoming irrelevant, and Google quite probably just made it official.

Fault-Tolerance

Use 5 replicas.

5? Why 5?  Why not 3?

Because, as Google puts it, with 3 replicas, all it takes is one data-center outage + one machine crash for you to be SOL, but with 5 replicas, you're pushing the black-swan way, waaaaay down the line.

And that, boys and girls, is what you need to worry about when you are working at Google-scale.  (Come to think of it, its also what we worry about at our scale, which, while not Google Scale, is still quite scale-y, and has 5 replicas...)

Anyhow the paper --->

 Abstract

Many of the services that are critical to Google’s ad business have historically been backed by MySQL. We have recently migrated several of these services to F1, a new RDBMS developed at Google. F1 implements rich relational database features, including a strictly enforced schema, a powerful parallel SQL query engine, general transactions, change tracking and notification, and indexing, and is built on top of a highly distributed storage system that scales on standard hardware in Google data centers. The store is dynamically sharded, supports transactionally-consistent replication across data centers, and is able to handle data center outages without data loss.

The strong consistency properties of F1 and its storage system come at the cost of higher write latencies compared to MySQL. Having successfully migrated a rich customerfacing application suite at the heart of Google’s ad business to F1, with no downtime, we will describe how we restructured schema and applications to largely hide this increased latency from external users. The distributed nature of F1 also allows it to scale easily and to support significantly higher throughput for batch workloads than a traditional RDBMS.

With F1, we have built a novel hybrid system that combines the scalability, fault tolerance, transparent sharding, and cost benefits so far available only in “NoSQL” systems with the usability, familiarity, and transactional guarantees expected from an RDBMS.