Stateful Services — Scaled Simply (sorry, couldn’t resist the alliteration )

-
#Wallaroo pipelines (°) can distribute/fan-out their data-processing to multiple back-end nodes — in a map-reduce kinda. A (the?) cool aspect to this is that the processing can be stateful.
The way it works is surprisingly simple — they
  • Break “global state” into non-intersecting partitions, based on keys
  • Map keys to partitions with a “partition function”
  • Distributed partitions to nodes (redundantly, for reliability)
  • Send input data to partition/node based on the input key
  • Rebalance partitions when a node is added or removed
It’s a clever trick, check out the details at https://goo.gl/vAUEeS
(°) Wallaroo applications are composed of one or more pipelines. A pipeline starts from a source, a point where data is ingested into the application. It is then composed of zero or more computations or state computations. Finally, it can optionally terminate in a sink, a point where data is emitted to an external system.

Comments

Post a Comment

Popular posts from this blog

Cannonball Tree!

-
Image
Spotted this one just outside the Ravindra Kalakshetra in Bangalore .  Its called the Cannonball Tree , but is - technically (i.e., in Latin, if thats your metier) - Couroupita Guianensis , or ನಾಗಲಿಂಗ ಪುಷ್ಪಾ  in Kannada. The fruits look like Cannonballs (well, d-uh), and the flowers are really quite amazing, and pretty ludicrously distinctive...
Read more

Erlang, Binaries, and Garbage Collection (Sigh)

-
Image
The nice thing about garbage-collection is that it is makes memory management absolutely transparent to you Of course, the above is true right up to the point when it stops being transparent, at which point it does a phase change from nice to clusterf**k of biblical proportions .  There are entire posts, books, and suicide notes that have been written about this, and I have no great desire to add to them, save with the minor comment that paranoia about garbage-collection is still justified , albeit only in the edge-cases for most people. That said, I'll give you that Erlang GC is in many ways much, much better positioned in this regard than, oh, Java, but it still rears up and bites you in the butt when you least expect it. ( More information about Erlang's GC vis-a-vis Java here , and Jesper's truly excellent description of Erlang's GC here ).  A story from the past should serve to illustrate this. Once upon a time, in a magical land far far away - well, Her...
Read more