November 11, 2016
Big Data, Polls and Elections
This presidential election year was unlike any before it, the year of big data. We had data, lots of data, not just Big Data, but Huge Data (yuge data?). With AI and machine learning being used on a scale that has never been seen before, massive amounts of polling, data scientists and analysts working overtime to predict the outcome. Everybody knew the outcome before it happened. But what happened?
They were wrong.
How could they be so wrong? Thousands upon thousands of companies are using these same techniques to increase revenue on a daily basis, yet many of them cannot say whether they actually are or not. Harvard Business Review says “The biggest reason that investments in big data fail to pay off, though, is that most companies don’t do a good job with the information they already have.” Econsultancy states “It’s the smaller packages of data that can help you achieve your eCommerce goals, not the sheer volume of it.”
But there is one more thing that was forgotten in all this data, people. People are not data. Humanity is not data. Psychohistory may purport to be able to predict the actions of masses of humanity, but it’s just fiction (for now). We cannot predict human behavior. We sometimes can predict what the reaction to a change on our website will be, but on a large scale we are very bad at understanding, never mind predicting, human behavior.
All this is to say, if you are not testing on a regular basis, and then putting those tests into action by personalization, then you are making as big a mistake as all the pollsters and data scientists did this election year.