Exploring the Long Tail with Analytics and Machine Learning


Computers and Chess

The game of chess is a big factor in the development of Advanced Analytics and Machine Learning (ML). I recently heard a story on NPR about chess and computers that explained that these days computers regularly beat their grandmaster human competitors at chess. Grandmaster, author and chess columnist Andrew Soltis says, “Right now, there’s just no competition. The computers are just much too good.” Murray Campbell of IBM’s Deep Blue project was also interviewed. Campbell says, “[When playing chess computers] play what they think is the objectively best move in any position, even if it looks absurd, and they can play any move no matter how ugly it is.”

Chess

There are 400 possible configurations of the pieces in chess after both players have made their first move. It gets very complex very quickly. After 6 moves there are over 9 million possible configurations (1). Grandmasters can plan ahead about 10-15 moves. They can also anticipate the most probable 3-4 countermoves their opponent might play for each move. Magnus Carlsen, current chess World Champion, was interviewed by Time Magazine in 2009. Carlsen explained that in some situations he can plan 15-20 moves ahead. Humans are pretty incredible.

But on a given turn in chess a computer can simulate every possible play 30 moves into the future and also simulate all of its opponent’s potential counter-moves – hundreds of thousands. The computer can also “remember” previous games it has played or observed or just been told about (millions of games). From those simulations and “memories” the computer can calculate the probability successful outcomes for all the scenarios. Then they simply play the best move… and then do it again on the next turn.

Boring, Mundane and Tedious Work

This sheds some light on how computers can tackle all sorts of things humans don’t do well or don’t want to do. Computers don’t get bored with repetitive, mundane activities. They don’t care about how complex a problem is or how much work is required to tackle it. In the end they don’t judge how inelegant the solution may be. Computers are used to apply Advanced Analytics and Machine Learning in the same way.

When companies engage in process improvement and quality control activities, constraints like timelines, money and manpower limit the options that humans might consider exploring in these endeavors. We are forced to prioritize and choose things that we think are most likely to bear fruit. Furthermore, humans deal with things like ego, exhaustion, anxiety, and hope. These influence how we prioritize our options and potentially obstruct the pathways to important insights. We never have unlimited resources so we bat around clichés like “focus on the low hanging fruit” and “diminishing returns start to apply.”

The Long Tail

Below is a graph of customer service complaints. Over 40% of the complaints are concentrated among 5 major categories. In order to improve overall customer experience, we, as humans, might think to focus efforts there – it’s the “low hanging fruit.” It could become tedious to delve into the lower frequency categories.

Long Tail - Top 5

As long as you can provide it the data, a computer isn’t constrained the same way humans may be. It doesn’t care if you ask it to explore the “long tail.” A computer will not complain nor get tired when asked to pursue seemingly impractical, tedious or obscure paths that on the surface seem unlikely to yield value. Frequently, the results from such exploration do in fact validate human presumptions of lack of value. Very valuable insight is often gained that would have otherwise been left undiscovered.

Long Tail

It’s easy to focus on the left side of the graph. But let’s not overlook nearly 60% of all complaints are to the right in the “long tail.”

The Long Tale

Certainly some categories from the “long tail” might be combined. Advanced Analytics and Machine Learning can be employed to find similarities among all complaints and group them into a smaller set of categories and subcategories. Some categories might be predictors of other categories. Advanced Analytics and Machine Learning can find root causes that might appear in the “long tail” that are actually driving a significant volume of complaints. Some of the root cause correlations might very difficult to evaluate. We could easily sub-optimize or miss really important opportunities if the long tail is ignored. It may be impractical for humans to dig through the long tail. Computers can delve into mounds of data and can help complete the tale. And as with chess, computers won’t complain when asked to do the work.

If you’re interested in seeing how to bring machine learning and analytics into your business, please feel free to reach out to me directly or download our Quick Start Workshop information sheet here.

Joe Guy

Joe Guy

COO & Co-founder at Mariner
Joe is passionate about helping companies leverage their information assets using IoT and Advanced Analytics to gain strategic business insight to improve revenue and profitability. Joe is involved with business development and strategy and leads the operations and human resources areas of Mariner, ensuring that our customers achieve their goals while our employees have a blast helping them do it.
Joe Guy

Please share your thoughts on this post:

Your email address will not be published. Required fields are marked *

About Mariner

Call Us Email Us

How Can We Help You?

Mariner © 2018. All Rights Reserved.