IoT’s big challenge: Managing billions of devices
Source: https://twitter.com/KirkDBorne/status/1252075819914137600
The breadth of IoT’s distribution over the coming years will frustrate efforts to harness it as a unified resource. IoT application developers are embedding algorithmic capabilities in resource-constrained endpoints such as mobile phones, business machines, and consumer products of every type. More IoT edges are acquiring the ability to make decisions and take actions autonomously. More IoT devices are embedding machine learning and other sophisticated analytic algorithms.
Though they’re growing more powerful, IoT endpoints will continue to rely on resources located elsewhere in the sprawl, such as on adjacent devices, nearby gateways, and always-on public clouds. We’re moving into a world that’s growing more federated among the disparate IoT domains that must somehow interoperate for mutual advantage.
Every IoT endpoint, device, gateway, platform, cloud, and service is a potential domain unto itself. Each domain will have its own requirements, policies, rules, and constraints with which IoT must ensure compliance. If you want a good graphic overview of a federated IoT infrastructure, check out the diagram in Ajit Jaokar’s recent Data Science Central blog.
As IoT’s disparate nodes flow into and out of myriad federation relationships, they will attempt to achieve mutually beneficial cross-domain terms of engagement. Some of these cross-domain IoT federation agreements will be of long duration, while others will be purely evanescent, on the fly, and in the moment. A great many federation arrangements will focus on managing the end-to-end data/analytics resource requirements of IoT endpoints and other connected resources that process and act on sensor data.
Casting a wide IoT net
How will we ensure that increasingly federated IoT clouds enable dynamic end-to-end management of disparate data/analytics workloads? In this recent DZone/IoTZone article, Denis Canty discusses how IoT’s federation structure is likely to evolve in this new era. He proposes a model in which each IoT cloud, edge, and device will have a corresponding domain that is managed through its own “data management controller (DMC).” According to Canty’s model, federated DMCs could play the pivotal role in dynamically discovering, aggregating, and brokering all the resources — processing, memory, storage, bandwidth, and so on — that each IoT needs.
A federated IoT interoperability model such this will prove absolutely essential for many real-world applications running on mobile phones, embedded sensors, and other resource-constrained endpoints. Many IoT endpoints may not have sufficient local resources for storing all data and executing all algorithms necessary to do their jobs effectively. By the same token, these endpoints may incur a significant performance hit when accessing complementary resources from adjacent nodes and/or IoT clouds.
Here are a few of the complex IoT scenarios where federated management of data analytics workloads may prove essential:
- Facilitate embedding of algorithmic capabilities so that IoT devices and apps can adapt continuously and react locally to their environments and, as needed, to metrics and commands from neighboring devices
- Enable developers to rapidly access, configure, and tweak any algorithm, including those discovered on the fly in neighboring nodes, that is suited to the IoT analytics challenges they face
- Execute algorithms on any size device at the edge or at gateways in containers on widely supported processing frameworks, including Spark and Hadoop, within a distributed IoT fabric
- Analyze data as it streams at the device level and move it rapidly to federated cloud-based platforms for storage, thereby eliminating the need to retain it locally
- Support execution of selected IoT analytics workloads locally, reducing or eliminating the need to round-trip many capabilities back to federated computing clusters in the cloud
- Perform spatio-temporal sensor-data analysis and summarization at the IoT endpoint, and send the rest to a federated IoT data lake or log database for further analysis, storage, and archiving
- Cleanse incoming sensor data at the endpoints, such as by imputing missing values, and forwarding to a federated IoT gateway for cross-endpoint normalization, aggregation, inference, and analysis
- Access data from any streaming data platform, as well as from any RDBMS, hub, repository, file system, or other store in any cloud, on-premise, or other at-rest data platform
- Tap into the massively parallel computing power of the cloud computing clusters at the heart of distributed IoT environments
- Scale IoT applications to support any size, volume, and speed of data from any federated IoT device anywhere on the planet
- Source data from any federated device, sensor, database, stream, and middleware fabric in the IoT
- Correlate events at the application level across the vast span of the IoT
- Distribute execution of analytic algorithms dynamically out to disparate IoT edge devices and gateways in order to maximize end-to-end application speed, throughput, and agility
- Enable robust, secure, and efficient performance, job, state, and health management under centralized administration across heterogeneous IoTs, clouds, devices, and apps
- Expose a functional API that simplifies development, testing, and deployment of algorithms and other complex application logic artifacts that are being deployed to the edges of a complex IoT application
- Aggregate a library of prebuilt IoT algorithms, maintained across federated repositories, to speed developer productivity in the building and maintain edge applications
Obviously, IoT federation structures could easily grow as complex and unwieldy as the distributed analytics apps they aim to support. The cross-domain messaging overhead needed to coordinate federated management of distributed IoT data-analytic workloads could prove prohibitive. And let’s not forget the equally important messaging overhead associated with managing other federated IoT services, such as reliable messaging, transactional rollback, long-running orchestration, and quality-of-service policies.
Regarding the middleware fabric for federated IoT, I agree with Canty’s suggestion that blockchain might prove optimal. Blockchain’s inherently open, distributed, agile, and transactional architecture would fit any cross-domain interoperability scenario such as IoT where secure data interchange is essential.
I also found Canty’s discussion of IoT app containerization in a federated interoperability context to be worth investigating. It would be good if developers could rely on containerized environments, such as Docker, for ensuring that IoT apps tap into the federated resources sufficient to execute data/analytic applications efficiently and scalably on any edge platform. Here’s my discussion of IoT app containerization from several months ago.
Without a comprehensive federation architecture, it’s not clear how distributed IoT applications will feed continuously optimized algorithmic intelligence to every edge at all hours of the day and night. When it’s an autonomous endpoint such as a self-driving car, an IoT analytics federation framework would ensure that the algorithms driving it all are continuously updated in conformance of the requirements of all stakeholders, such as manufacturers, dealers, ride-sharing providers, passengers, government regulators, insurers, and service organizations.
IoT federation will help society as a whole to optimize this sprawling fog for every conceivable application — even for future applications we haven’t begun to conceive.
