BigData and Complexity

Derrick Harris writes in GigaOm

When it comes time to undertake a big data strategy that requires turning advanced algorithms on potentially massive data sets, many fast realize they don’t have, or have nearly enough of, the necessary skills internally... 

...In an era of webscale computing and large clusters running big-data workloads, we’ll also need more people who can apply mathematics to data with the goal of automating and troubleshooting distributed systems...

...The problem, as (Google) authors Chris Lewis and Rong Ou present it, is that with such a large, growing and increasingly complex code base — and thousands of developers working on it — it becomes nearly impossible for code reviewers to identify “hot spots.”

To summarize its not enough that we can now consume and process vast quantities of data - extracting meaning from this data is becoming increasingly complex.  In the DIKW hierarchy, we're barely at the Information stage, and extracting Knowledge is, well, advanced math.  And in the immortal words of Barbie, Math is Hard (*).

In short, Complexity never goes away, it just moves up the food chain.  We need to start getting used to the fact that math skillz are very much going to be in vogue...


* -->  Yeah, I know, its actually Math Class Is Tough.  Deal with it.