And you are here why?

Apache Arrow   is many things, but the really important bit is that it specifies a columnar memory format for data that is language-independent, set up for flat or hierarchical data, and has a   really   nifty set of accompanying libraries for zero-copy streaming, IPC, analytics, and the like. The question then is, why would you even begin to worry about storage formats? Well, you probably don’t, or won’t, until you bump up against the reality of large scale data-processing, and moving this data across multiple databases, which is when life starts getting hairy. The thing is, when you actually look at how databases store data, they have all sorts of additional stuff stored alongside the data. Fun stuff like references to log-records (your recovery-manager needs to be able to process the logs y’know?), MVCC timestamps, index information, and lords know what else. And given that different databases have different mechanisms for doing logging, concurrency, indexes, a...

There is a paper out by Google on how they migrated from MySQL to F1 for AdSense . Its fascinating reading on its own, but its got some key take-aways for scaling and reliability.  I'd recommend going and reading it, but these two are worth emphasizing Scaling Eliminate the R in ORM .  In short, instead of building out a nice layer (hibernate, whatever) between your DB and your code, go the other way, and deliberately expose the workings of your DB to your application developers.  This actually makes tremendous sense - in many ways the advent of NoSQL has been due to people specifically picking data stores that map to their application requirements .  Need a key-value store? Use Riak .  Need a document store wht guaranteed writes?  Use CouchDB .  Hiding the specifics of your data store behind an ORM layer is - increasingly - becoming irrelevant, and Google quite probably just made it official. Fault-Tolerance Use 5 replicas. 5? Why 5...

Theres a ridiculously cool map out from IDV Solutions titled 56 years of tornado tracks, by F-Scale . It is, as you would expect, 56 years of tornado tracks overlayed on a map of the U.S. And boy oh boy, is it ever a doozy - check out the hard stop as you move west... Small version below, but click to embiggen...

Check out the latest Scientific American - its got a great article on water usage by country , and what it is used for. For my money though, the fascinating part is that A vast amount of water is used to produce the food and products that nations consume. Large population is the greatest factor, but inefficient agriculture or dependence on water-intensive cuisine can exacerbate demand; meat consumption accounts for 30 percent of the U.S. water footprint. Given the sheer amount of water that India uses (and exports!), the importance of the monsoons makes sense - around 80% of India's water comes from the Monsoon , as does a huge chunk of its energy.  Its pretty harsh - this is a ton of water and its all dependant on one - finicky - annual event.  As the Economist puts it Yet the drought underlines a grim truth. India’s extremes of hydrology, poverty and population present vast difficulties for water management which it has never mastered. And they are growing. Increasin...

Dan Woods has an article at Forbes about " How Hadoop and SAP HANA can accelerate Big Data Startups ".  Its pretty hard to read - you have to get past the obvious shilling for SAP  (the byline is a bit of a giveaway - " He has written several books and created other research and educational content for SAP "), and after that, you have to ignore the Hadoop-centric nature of the post (there are other fish in the sea, you know?) You have to get all the way to the end of the article before you get to the meat, which boils down to ... can SAP make it as easy to experiment with SAP HANA as it is to download and use open source? ... will developers buy into SAP’s efforts at being open and making SAP HANA easy to use? (If it is) Priced too high or with onerous terms, SAP HANA won’t make sense to startups. And these points, in the end, are what its all about. A significant - and somewhat under-reported - aspect of the current BigData boom is that people ar...

D'you remember this ridiculous article in The Database Journal on " The Hidden Cost of Scaling in NoSQL " ? I won't go into it again - if I remember correctly, I summed up my take on the whole article as  Mind you, they did miss the following very important points too Baby Seals :  NoSQL developers club baby seals with joy and abandon, and make gloves with their flippers Puppies : Using NoSQL makes you want to kick puppies ( and kittens!) Nazis : Hitler used a NoSQL database Seriously, if this isn't sufficient reason for The Database Journal to be shut down, well, I can't think of a better one... Now Sreedhar Kajeepeta in InformationWeek has tried to one-up them with the provocatively titled NoSQL Everywhere? Not So Fast.  The difference, this time, is that instead of a series of ad hominem attacks, this article takes the Cool-Whip approach ( An argument that is so content-free you end up nauseated, and acquiesce by default ).  For ...

stats.minecraft.net puts together a sweet redis + node.js + socket.io app to do this. Its pretty wicked - a realy nice example of BigData / Analytics in the modern style, go check it out ! (note:  its live, the image below is just that, an image, go to stats.minecraft.net   instead!)

I've been having this long argument w/ a colleague about the merits of SQL vs NoSQL, which I'm anot going to get into here, but recently it devolved into claims along the lines of "The CAP Theorem is Wrong" (no, thats not my quote). It took me a while to figure out his point, but it turned out to be a fairly common misunderstanding, viz., If you have predictable Network Partitions, you can guarantee C, A, and P . i.e., if you know, beforehand, exactly how your network is going to get blowed up, then you can protect against it, and have Consistency and Availability and Partition Tolerance .  This is absolutely true! Huh? Say what? Well, yeah.  Note the weasel phrase " if you know, beforehand ".  This is not unlike saying, "If you know, beforehand, what the winning lottery numbers are going to be, you can win the lottery every time ". The point about Partition Tolerance is that it needs to handle arbitrary / random scenarios.  ...

You can absolutely go read up on Eventual Consistency , the CAP theorem , etc. to understand Eventual Consistency, quite a bit of which is good.  They tend to skimp on something I find quite important though, i.e., the role of White Lies in an Eventually Consistent environment. Depending on the architectural context, you can end up with a system where there is a requirement for consistency, but only once all the Business Processes are taken into count.   Or, to phrase it differently, if you are willing to lie Just a Little , you can relax constraints without getting yourself into trouble. Consider my company, which, to put it technically, does all sorts of big cloud telephony thingies.  These thingies are all donewith clumps of servers ("clump" --> another technical term).  At the simplest level, what we have is just a big honking phone system.  Y'know, the kind where people call each other and leaves voicemails and stuff. So, consider a fairl...

Its part of the human condition - the desire to solve all the problems you encounter with the tools that you currently have.  Its not necessarily a bad thing - when Goongrah was being attacked by the Bear in the days of yore, he defended himself with the club he was carrying, since saying " Yo Mr. Bear.  Hang on whilst i go fetch my spear from yonder cave.  And by the way, Ugh "  wasn't really an option. But ye gods, has this ever evolved into some remarkable insanity, especially in The World Of Development.  Everything, but everything needs to be done using the same tools.  People will go out of their way, and twist themselves into knots to retrofit all their problems to their one solution.  Take databases for example - till fairly recently people had two hammers (if that) - that they could use MySQL / PostgreSQL / Oracle for their persistence needs, and memcache for their key-value store.  And to be serious, some of the more devout ones...

InfoChimps  is offering cloud-based BigData platform to provide out-of-the-box (or is that out-of-the-cloud ?) Hadoop service.  Plenty of details at the link , as well as GigaOm commentary here . The new   Infochimps Platform   is essentially a publicly available version of what the company has built internally to process and analyze the data it stores within its marketplace. As Infochimps CEO Joe Kelly puts it, the company is “giving folks … the iPod to our iTunes.” Me, I think this is the start of something really good. The thing is, there is a ton of activity in the Hadoop space, with everybody and their brother tossing their hat into the ring providing some set of ancillary services based on Hadoop. The problem with all of these are that they presume you know what you are doing ! Needless to say, you don't. Really. You...Just...Don't. Let me be a bit clearer - the vast majority of people out there who could use Hadoop-based services fall i...

SpaceCurve just scored $2.7M in new funding. And, why do you care? Well, you may not, but they are doing something interesting vis-a-vis the merging of BigData and Spatial Data .  And yes, that is a deliberately vague description, but thats pretty much how things work in the BigData world nowadays, don't you think? Seriously though, think of it this way -   Spatial data is not inherently easy to deal with in traditional terms (think How do I get the distance between two points?  How do I find all the data-elements in this cube? etc. ).  The big SQL players all have Spatial extensions to their DBs , as well as some of the NoSQL players ( CouchDB , MongoDB , and Cassandra ).  Now, add in "Location Services" - where you need to keep track of not just data scattered in space, but time too.  e.g., Your Foursquare check-ins last week, your GPS readings last week, etc. And *then*, add in connections between the entities generating this data....

There seems to be a pretty potent Meme floating around right now viz. The Future Of NoSQL - To Be Borged .  The general argument seems to be some combination of  Databases are complex beasts We've spent eons simplifying access to these data stores (SQL) BigData revolves around an even more complex set of problems Everybody is building custom solutions to these problems Regression to the mean will result in them all becoming somewhat similar SQL stores will just add these common features All your bases will belong to them  As you might imagine, I quite disagree with this. Oh yes, I fully expect SQL DBs to incorporate some NoSQL features over time, but it really isn't the same as when Object Oriented Databases went to the Great Dustbin In The Sky.  As I've mentioned before, NoSQL stores are more solution-oriented - there is a huge and thriving ecosystem, and the vast majority of them are designed around specific types of solutions.  Can they be ...

Monash has a slightly different categorization of BigData, breaking it up into Size vs Structure .  The main idea is that while we tend to think of BigData as The Three (or Four) Vs , there is actually a distinction here between    Size : Volume, Velocity    Structure : Variety (,Variability) To  quote... Relational big data   is data of high volume that fits well into a relational DBMS. Multi-structured big data   is data of high volume that doesn’t fit well into a relational DBMS.   Alternative: Poly-structured big data. Conventional relational data   is data of not-so-high volume that fits well into a relational DBMS.   Alternatives: Ordinary/normal/smaller relational data. Smaller poly-structured data   is data for which   dynamic schema capabilities are important, but which doesn’t rise to “big data” volume. The point being that instead of thinking of all these in the same space, different combinatio...

There is a case-study for mongoDB at foursquare up on 10gen's site.  Mainly the usual stuff, viz. AutoSharding, GeoIndexing, Replica Sets, and Document Model. Nothing in the case study actually screams out "Yes!  This is why they should have used MongoDB "(over BigCouch for example), which is pretty funny, since one of MongoDB's big differentiators (high write throughput) would have been a key feature. Note :  FWIW, the high write throughput comes with the potential for data loss, but thats not really an issue for foursquare (if your checkin doesnt work, just do it again...). Another Note : The idea that a database doesn't guarantee the writes and can occasionally lose data is, of course, interesting  :-)

Paris Tech Review is running a fairly long (and somewhat content free) history and survey of BigData, from the growth of metadata, through NoSQL, analytics, and Business Intelligence.  Its a good summary if you haven't seen any of this before, otherwise don't bother. That said, the one relevant line follows We no longer witness the confrontation of theory and reality but an entirely new process: the pattern, if there is one, emerges from bottom-up processes of handling data through statistics. I'm not too sure that I'm willing to give up on the deductive process quite yet, but I will throw my hat in the ring for the value of serendepity, and unexpected discoveries. (and sorry for the image from one of the worser John Cusack movies...)

Kresten Krab Thorup presented Riak Mobile at Erlang Factory (London) .   The consistency model used by Riak is intended to continue operating well even while nodes in your Riak cluster are down or unreachable, a property which also makes it a excellent model for mobile data. [...] Typical usages for Riak Mobile is as a mobile content distribution platform using one-way sync; or with two-way sync to also push updated data back to your Riak cluster when the network is "eventually" available.  ... The client-part of Riak Mobile does not need an Erlang VM.  Rather, it comes as either a Java or an Objective-C component using SQLite for on-device storage, and thus integrates well into the native development environments... Sounds like Riak might actually actually do what CouchDB promised...

Data Analytics is in all our futures .  Mind you, when I say Data Analytics , I'm not referring to complex Business Intelligence systems in the High Temples of Enterprises . I am referring to tools that make our lives radically simpler, tools like Siri, which have successfully used Analytics to make a ridiculously complex task even more ridiculously simple (*). Enter Kaggle -  a company dedicated to the proposition that Data Analytics can be a sport (albeit one for the brainy amongst us).  You give them access to your data, the problem, and a prize.  They (Kaggle), outsource the problem to their gaggle of smart people around the world, who compete with each other on finding the best solution to the problem.  The winner gets the prize. It hits all the right marks - competitive natures (FTW sucker!), prizes (money!), and the spirit of exploration.  Also, for those companies that don't want to give up sensitive information, Kaggle also has a serv...

BigData and Rents Janko at GigaOm had an article about Ethics and BigData ( Drawing an ethical line in the bigdata sand ).  The basic point is well-framed - in a world of increasing complexity, it is getting easier to hide societally harmful behavior behind algorithmic complexity. All companies want to maximize internal efficiency to save on power bills and production processes. Even non-profit and government organizations  want to eliminate costly defrauding of their systems  and figure out the best ways to spend their limited resources. But is there a line we shouldn’t cross when it comes to using analytics to squeeze every last dollar out of an operation?  I think there is, and I think it’s at the point of transition between things we  want  and things we  need . Case in point: On Tuesday, the  New York Times  published an article  describing how specialized software is letting landlords maximize rents  by analyzing supp...

As members of our society, we generate a ton of data.  Some of this ends up as governmental  (tax information, census), some of it ends up as administrative  (land records, traffic accidents...  yes, a fine distinction, but still a relevant one), and some of it is, well, just data (rainfall). Over the last few years or so, there have been a number of attempts to make this data public - typically under some form of open data  initiative. The general idea, of course, is that once people can start messing with this data, they can come up with all sorts of relevant and fun uses.  For an example, check out what the good people in California have done , and in particularly, the remarkably cool Explore California mashup ...  While their heart may be in the right place, politics usually ends up causing chaos - and when I say politics , I actually mean national security .  You'd be surprised at how much stuff is considered 'dangerous'. Rai...