Prometheus at scale with Thanos

-
If you’re using Prometheus at scale, you’ve probably run into one or more of the following issues
  1. 1. Data retention costs money, the more historical data you keep, the more SSD you need, and that gets expensive.
  2. 2. You scale prometheus by sharding, and the ensuing contortions you go through to get a unified view involves all sorts of madness with federation, grafana queries, and whatnot.
  3. 3. An HA setup inevitably runs into the entertainment associated with de-duplicating prometheus data.
  4. 4. If you run into more than one of the above, then whatever you do ends up being somewhat Rube-Goldberg-ish, with associated maintenance headaches, replicability, inefficiency, etc.
Enter Thanos, which integrates transparently with Prometheus, and pretty much takes care of all of the above issues, and remarkably elegantly at that. It consists of a bunch of modular components that can be snapped together to — transparently! — provide a unified view across your existing Prometheus setup, while — inexpensively! — letting you scale.
Unified View 
At the simplest level, it provides a “meta” service (the Querier) that sits in front of your existing prometheus servers. You can now hit this single end-point for everything — no more worrying about federation, no more grafana chaos (since a query can now “hit” multiple prometheus servers!), automatic de-duplication of data, oh, this is good stuff indeed!
(Technically speaking, you also add a sidecar to each prometheus server. The Querier and the sidecars all gossip away and figure out whats what)
Scaling
The same prometheus sidecars can also be configured to back their data up to GCS or S3. The cool thing is that if you now add the Store component in front of GCS/S3, then the Querier will — transparently! — hit that along with the existing prometheus servers, and you’ll never notice the difference. The point here being that you can reduce the retention period on your prometheus servers (object storage is way, way cheaper than the SSD on your prometheus instance!), and not worry about losing historical data — the Querier will happily get historical data from GCS/S3, and current data from prometheus for optimal Grafana goodness 😆
Downsampling
As a bonus, the Compactor component will trawl through GCS/S3, and happily compact and downsample all your historical data, for when you want to look over historical data over large time periods (“show me memory pressure over the last year!”). It does this without clobbering the raw data, which means you get the ability to drill down into the gory stuff at will, but without the latency associated with having to decompress a bajillion samples
It’s really, really good stuff, and it works remarkably well. For a much more detailed description, do check out this blog post. And for those who are interested, there is an excellent writeup on the architecture and design on GitHub.

Comments

Flávio Stutz said…
We've done something like this but using plain Prometheus in hierarchical federation for sharing scrapes and doing aggregations through federation to avoid too much metrics diversity (our deployment has thousand of servers and we have hundreds os apps). The storage is low because we need just to keep some history on the last aggregated level, which has low diversity. We've released the utility we created for configuring Prometheus dynamically at https://github.com/flaviostutz/promster
September 8, 2019 at 3:50 PM
Post a Comment

Popular posts from this blog

Cannonball Tree!

-
Image
Spotted this one just outside the Ravindra Kalakshetra in Bangalore .  Its called the Cannonball Tree , but is - technically (i.e., in Latin, if thats your metier) - Couroupita Guianensis , or ನಾಗಲಿಂಗ ಪುಷ್ಪಾ  in Kannada. The fruits look like Cannonballs (well, d-uh), and the flowers are really quite amazing, and pretty ludicrously distinctive...
Read more

Erlang, Binaries, and Garbage Collection (Sigh)

-
Image
The nice thing about garbage-collection is that it is makes memory management absolutely transparent to you Of course, the above is true right up to the point when it stops being transparent, at which point it does a phase change from nice to clusterf**k of biblical proportions .  There are entire posts, books, and suicide notes that have been written about this, and I have no great desire to add to them, save with the minor comment that paranoia about garbage-collection is still justified , albeit only in the edge-cases for most people. That said, I'll give you that Erlang GC is in many ways much, much better positioned in this regard than, oh, Java, but it still rears up and bites you in the butt when you least expect it. ( More information about Erlang's GC vis-a-vis Java here , and Jesper's truly excellent description of Erlang's GC here ).  A story from the past should serve to illustrate this. Once upon a time, in a magical land far far away - well, Her
Read more

Visualizing Prime Numbers

-
Image
From Jason Davies , we have El Patron de Los Primos , probably one of the coolest prime number visualizer that I've seen.  For every Natural number, he draws a periodic curve that intersects at that number and each of its multiples.  Needless to say, prime numbers are touched by only two curves (itself and 1 ). You can scroll to the right, and move up the prime food change. (The image above is a screen grab.  Go here to see the actual thing)
Read more