What gets lost in the search for patterns? Is it You?

Part of the promise of Big Data is how it can be used to uncover patterns and trends, especially when artificial intelligence is applied. As with statistics, the higher the number of samples (data), the more likely the pattern can be used to make accurate predictions.

For example, to quote Eric Siegel:

“Computers have gotten a lot faster … and we have a lot more data. As a result, banks, retailers, political campaigns, doctors and hospitals, and many more organizations have been quite successful of late at predicting the behaviour of particular humans. Their efforts have been helpful at winning customers, elections, and battles with disease.”

There’s more than just applying Big Data to human behaviour, of course. It can be used to analyze and make predictions about drugs, the weather, astronomy … whenever large quantities of data exist, Big Data techniques can be used.

It sounds wonderful, but I have a concern.

First, patterns and trends often relate to majorities — what most of the data indicates. What if you are in the minority? What if you cannot be pigeon-holed into a group/category? Will you miss out on the benefits of Big Data?

As an example, consider Big Data as used for marketing . If you don’t fit the pattern, do you miss out on seeing advertising for products or services that you would be interested in buying?  Are you subjected to prompts to buy what you have no interest in because you (seem to) fit some other pattern? (Actually, based on the laughable, sometimes bewildering FaceBook and Google ads that are served up to me, I think that is what is actually happening).

I am making it sound like a problem, but it is actually an opportunity. If there were a way for individuals to give their feedback back into Big Data processes, to provide even more data to guide the pattern recognition, it would increase its accuracy.

For example, if there were some way I could tell Google that I was only looking up hotels in Florida because I want to visit a sick relative, that laying on a beach doing nothing is about the worst vacation I could possibly imagine, and that they should stop sending me daily ads for beachfront hotels? That would be welcome!

This is not technologically impossible. Something like Solid (see last posting) might be a platform to gather that feedback, a way that you could inform Big Data about yourself and where you might fit in.

And, because Big Data is so big, it should be able to handle situations where  you don’t fit a pattern at all. Where you are sui generis, one of a kind. Where you are your own category and therefore unpredictable. Big Data should be able to handle even that datum, even if it’s only to slot you into the “Unique” category.

 

 

#THINK 2013: the ORION conference on Extreme Data (part one)

Big Data Week 2013 took place April 22-28.  Fortunately, Toronto was one of 25 participating cities, and I had the opportunity to go to the ORION THINK 2013 conference. My report will have to be  brief because it was a day jam-packed with overwhelming data in its own right. I can only give you the highlights focusing on the you-and-me perspective of this blog.  Even so, I think that I’ll have to do several posts.

The theme of THINK 2013 was “Extreme Data”  —  Big Data as the mind-boggling, ever-increasing amounts of data available AND the great potential that comes from those huge amounts of data, IF you use the right technology and the right approach. Privacy is a good example. Big Data is ever-more pervasive and Extreme, so it is essential that your privacy is protected. As Ann Cavoukian, Information and Privacy Commissioner of  Ontario, persuasively argued,  this can not only be done without getting in the way of Big Data, but it can enhance it by making Big Data become Smart Data. Privacy is really about control of your personal data as determined by you. Using artificial intelligence technology, that control can be embedded into the data itself so it can’t be misused, then your privacy is protected automatically without you or anyone else having to act as a watchdog.

Smart Data would know more than just how to take care of itself. Smart Data could be really smart. For example, you may want only a few people to have access to your medical information, and a different group of people to have access to your employment history. With Smart Data you could tailor the levels of access to your data very specifically.

If that sounds far-fetched, know that researchers at the University of Toronto are already working on Smart Data in partnership with the Commissioner’s Office.