What is the biggest bottleneck for the Internet of Things (IoT) to be successful?
Clue: It’s no different than the biggest bottleneck for “Big Data.”
And no, it’s not the infrastructure challenges. Any other guesses?
Well, let’s reframe the question. What is the biggest bottleneck in the process of converting raw data to actionable decisions?
Before we give you the answer, let’s consider the life cycle of data.
There are roughly 4 phases, called “the 4 A’s of data” from Jeffrey Stanton’s Introduction to Data Science:
While each phase has its challenges, the problems associated with Acquisition, Architecture (integrating & shaping) and Archival have been (or soon will be) solved by technology. That just leaves Analysis unsolved.
If it wasn’t obvious, the biggest bottleneck is the human ability to make sense of data. Our bandwidth for Analysis is limited and is not keeping pace with the explosion of data.
The Internet of Things will further exacerbate this problem. Too many sensors and devices generating way too much data for any human to meaningfully act upon. Solving this problem is a significant piece of the Digital Business puzzle.
There isn’t a silver bullet, but the solution lies somewhere between automating even more of the mundane and giving machines “intelligence” so they can make their own operational decisions. Sounds daunting but thanks to Microsoft Azure, Sparkling Logic and some secret Mariner sauce – we have a solution!
From Data to Decisions: The Smart Way
Our solution is a heavily modified version of a system described by Alan K Fish, in his book Knowledge Automation. The technologies used in our example include Microsoft Azure Machine Learning (ML), Azure Intelligent Systems Service (ISS), HDInsight, and Sparkling Logic’s SMARTS Decision Management service.
We’ve applied our expertise in predictive maintenance (PdM), decision management, analytics, machine learning and cloud to create the conceptual architecture show below.
At the cost of oversimplification, here is the flow:
- Data is generated by various devices and sensors.
- Azure ISS is used to provide a secure connection and help with the management of devices and collection of data. Once the data is in the cloud we store it in a “data lake” built using HDInsight.
- Next we apply Azure ML’s machine learning algorithms to uncover patterns and gain predictive insight.
- Finally, the enriched data set can be fed to a decision management service, like SMARTS, which fires off decisions.
In short, the architecture above siphons data in, makes sense of it and churns decisions out – with limited human involvement! While many details are missing, conceptually this represents a predictive, “decisioning,” cloud service that can be re-used for multiple scenarios.
Using the pattern described above, or something similar, we can help your business become a more intelligent and scalable digital business.
Shash Hegde, Technology Solutions Specialist at Microsoft, contributed to this post