NNStreamer: Efficient and Agile Development of On-Device AI Systems

Abstract

We propose NNStreamer, a software system that handles neural networks as filters of stream pipelines, applying the stream processing paradigm to deep neural network applications. A new trend with the wide-spread of deep neural network applications is on-device AI. It is to process neural networks on mobile devices or edge/IoT devices instead of cloud servers. Emerging privacy issues, data transmission costs, and operational costs signify the need for on-device AI, especially if we deploy a massive number of devices. NNStreamer efficiently handles neural networks with complex data stream pipelines on devices, significantly improving the overall performance with minimal effort. Besides, NNStreamer simplifies implementations and allows reusing off-the-shelf media filters directly, which reduces developmental costs significantly. We are already deploying NNStreamer for a wide range of products and platforms, including the Galaxy series and various consumer electronic devices. The experimental results suggest a reduction in developmental costs and enhanced performance of pipeline architectures and NNStreamer. It is an open-source project incubated by Linux Foundation AI & Data, available to the public and applicable to various hardware and software platforms.

Toward Among-Device AI from On-Device AI with Stream Pipelines

“…Recent releases of DeepStream [15] propose edge-to-cloud AI pipelines. In addition to the issues of not treating tensors as 1st class citizens of stream data [8], it is proprietary software dedicated to NVidia hardware, which cannot meet R5, R6, and R7. Google has proposed MediaPipe AI pipeline framework [12], which is now targeting on-device AI applications of IoT devices in addition to cloud AI services.…”

“…Our approach extends the on-device AI pipeline platform, NN-Streamer, to the among-device AI pipeline platform. The on-device AI capability inherited from [8] R7 is preserved while new requirements (R1 to R6) are satisfied by this work. To satisfy the new requirements, we first extend the AI stream data type ("other/tensors" MIME) to support tensors with flexible dimensions and schemaless data streams.…”

“…The previous work [8] is an on-device AI pipeline framework enabling developers to describe and execute AI systems as stream pipelines. We have deployed it to recent devices of the affiliation and a few software platforms.…”

“…We have deployed it to recent devices of the affiliation and a few software platforms. Implementing on-device AI systems as stream pipelines with NNStreamer has satisfied the related requirements mentioned in [8], has improved the overall performance, and has saved the developmental costs. After its initial deployments, we have observed new types of on-device AI applications pursued: distributed on-device AI systems where inputs, outputs, and inferences might happen on different devices.…”

“…• Based on users' feedback, including AI algorithm researchers, application and OS developers, and device vendors, we identify requirements, propose a corresponding design, and deploy the implementation. • Extending the previous work [8], we provide mechanisms reflecting the lessons learned by deploying products. • We propose and deploy stream transmission protocols for among-device AI systems tested for products.…”