BigData and Ethics

BigData and Rents
Janko at GigaOm had an article about Ethics and BigData (Drawing an ethical line in the bigdata sand).  The basic point is well-framed - in a world of increasing complexity, it is getting easier to hide societally harmful behavior behind algorithmic complexity.

All companies want to maximize internal efficiency to save on power bills and production processes. Even non-profit and government organizations want to eliminate costly defrauding of their systems and figure out the best ways to spend their limited resources. But is there a line we shouldn’t cross when it comes to using analytics to squeeze every last dollar out of an operation? 

I think there is, and I think it’s at the point of transition between things we want and things we need. Case in point: On Tuesday, the New York Times published an article describing how specialized software is letting landlords maximize rents by analyzing supply, demand and market prices to suggest the highest-possible price to charge for a particular apartment. It sounds logical enough if you’re in the property business, only peoples’ homes aren’t plane tickets, hotel rooms or rental cars.

So fine, you, the business person, can use BigData to figure out the maximum amount that you can suck out of a situation.
The problem is that you can also use BigData to avoid the regulatory impact associated with most of your activities!
Huh?
You basically need to think of this as the modern day equivalent of tax-dodging.  In the depths of history (i.e., the Prohibition era) you, the aspiring tax cheat had your book-keeper bury the shady stuff in  reams and reams of double-entry book-keeping a-la Al Capone.

You were pretty safe, till the Kevin Costner and Sean Connery decided to throw a ton of manpower at the problem (MapReduce, FBI style!), and eventually uncovered your scam.
With computerization - and QuickBooks - it is now actually easier to bury the data - think of it as increasing the size of the haystack in Ye Olde Needle And Hayestack.  Consequently, the rise (and fall) of LTCM, Enron, Lehman, Madoff, etc., etc.
Hence - Regulation.  With the exponential increase in the societal costs associated with collapses, it behooves us to either prevent it in the first place (if possible!), or if not, channel the collapse in such a form as to prevent The End Of The World As We Know It(*).

Regulation? Or Ethics? Or Both?

To bring this back to BigData - it is just another tool. However, it is also a tool that can be used to make sense of data in new and complex ways that are truly not apparent and/or available to "mere mortals".  I'm fairly certain that I don't buy the point made in the article about 'pure profit'

higher rent just because someone will pay it is pure profit.(**)

However, this is exactly the type of situation where societal policy and/or regulation and/or ethical standards need to stay in front of the issue.  BigData can be used for rent-extraction (hah! pun!) causing tremendous societal damage and human misery in a very short time.  Given the marginalized state of our current governing systems, I'm not too sure that there is a solution though, other than appealing to people's best interests, and that tends to rarely work all that well...

(*) For Financial Crises, there is always Bagehot's rule - short and long descriptions thereof

(**) Karl Smith on profit - There is no “enough profit.” More is always better. More profit yesterday. More profit today and still more profit for breakfast tomorrow.    Ok, maybe that is a bit over the top, but the point remains.  And yes, the trick is in reading the term profit.  You do need to factor in all the costs, as compared to omitting the societal/environmental ones as we tend to be so happy to do...