The So-Far Wasted Potential of Big Data

Big data has been a very hot topic in both technology and behavior science for some time now. People quickly grasp the possibilities big data opens. Theoretically, we can combine all sorts of rich information about individuals, from their shopping habits to their web browsing to their health records to their traffic violations, and use it to engage them in a product, sell them a service, or guide them to better health (I had to add at least one truly prosocial use there). 

Realistically, though, what we are seeing is similar to the paradox of choice. There is just so much data and no really clear method of organizing it so it can be used. When there is too much data, it becomes confusing and muddies the interaction with users.

What anyone looking to use big data needs is a clear objective of what they want to achieve with the data, a strong technological infrastructure to tap into the right data at the right time, and a plan to translate the outcome of the big data analysis into something for the user.  Some companies have done this well on a smaller scale (Disney comes to mind), but I’m not sure I’ve seen it done in health care and I certainly haven’t seen it done in government. That’s too bad, because big data done right could help craft a motivating consumer experience by supporting the fundamental needs that underlie engagement:

• Autonomy. By tapping into a large dataset of similar consumers (or a deep dataset on an individual consumer), it is possible to design an experience that is well-aligned with a person’s interests and values.  Theoretically, this is some of what Facebook’s news feed feature is designed to support, by showing stories that are most relevant to the user and cutting through the clutter of less relevant items. As readers of my blog know, I find this mechanism backfires for users like myself with under 1000 friends, and would prefer the autonomy support of being able to permanently set “most recent” stories as my display.

• Competence. Big data provides a mechanism to level-set user tasks for optimal challenge. By comparing each user to the broader population, big data can help companies predict the logical next step for an individual user. What have other people like this person done, and how well have they succeeded at it? Big data provides insight into the next task in the sequence for an individual user based on behavior patterns in the greater population. An example that uses big data in this way (although one could argue it’s not a very engaging experience!) is standardized tests such as the SAT, which take users down a particular path based on which questions they’ve gotten right.

• Relatedness. If big data is cleverly used, it can help make a user feel understood. I like to use the example of how Amazon’s suggestions for me go beyond what I might anticipate. It’s not surprising that Amazon has figured out what types of books I read or that I typically order certain household cleaning supplies. It does surprise me that Amazon has realized my kitchen décor is red. Aside from the potential creep factor, Amazon’s astute assessment of my purchasing preferences does make me feel like the site knows me. On the flip side, a poor use of big data can violate a sense of connection. I am always amazed when a website is smart enough to showcase a product I’ve been admiring in an advertisement—but too dumb to realize I already bought the shoes.

It will be interesting to see in the next 5-10 years which companies figure out how to engage users with big data and which don’t. My prediction is that we’ll see two successful groups emerge: The giant companies, like IBM and Amazon, who have the firepower to develop an infrastructure that manages and sorts big data, and the smaller more nimble startups that figure out how to consume that data from other sources. Time will tell.