Banner -- An Image Sensor Reconfiguration Framework for Seamless Resolution-based Tradeoffs (video)

Hu, Jinhan; Shearer, Alexander; Rajagopalan, Saranya; LiKamWa, Robert

doi:10.1145/3307334.3328594

Cited by 6 publications

(4 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Apple integrated an NPU inside their recent line of iPhones, which is used in conjunction with builtin 3-D sensors, primarily for the FaceID use case, i.e., unlocking the phone using the user's face. 1,2 Recently, NPUs have been more broadly used to run other vision tasks, such as semantic segmentation. Along similar lines, Google integrated an edge tensor processing unit (TPU) inside their recent Pixel smartphone, which can be programmed to execute a wide variety of vision tasks.…”

Section: A Hardware Accelerationmentioning

confidence: 99%

“…This is because the existing pipeline needs to undergo several sequential steps, such as flushing the existing frames, allocating memory to new frames, and starting the pipeline afresh. The Banner framework [1] avoids these time-consuming steps by allocating memory for the highest possible resolution so that any resolution image can be stored in that location. On top of that, Banner modifies the existing V4L2 framework to let it read only the actual resolution size number of pixels from memory whenever it needs to service request from a vision application.…”

Section: B Os/runtime Configurationsmentioning

confidence: 99%

See 1 more Smart Citation

Software-Defined Imaging: A Survey

et al. 2023

View full text Add to dashboard Cite

show abstract

Section: A Hardware Accelerationmentioning

confidence: 99%

Section: B Os/runtime Configurationsmentioning

confidence: 99%

Software-Defined Imaging: A Survey

et al. 2023

View full text Add to dashboard Cite

show abstract

“…The system design of Xihe is largely inspired by our empirical study and tackles problems that are common to edge-based AR systems [16,18,19]. Specifically, when designing Xihe, we focus on minimizing the reliance on mobile resources to avoid excessive power consumption [2,3,12]; we also minimize the network communication to the edge server by only issuing requests that are likely to improve the lighting estimation accuracy, e.g., when the lighting condition changes or when Xihe has more updated environment information [4,17]. Our work differs from existing work on edge-based AR systems in addressing the lighting estimationspecific requirements and 3D vision-based opportunities.…”

Section: Related Workmentioning

confidence: 99%

Xihe

Zhao

Guo

2021

Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services

View full text Add to dashboard Cite

Omnidirectional lighting provides the foundation for achieving spatially-variant photorealistic 3D rendering, a desirable property for mobile augmented reality applications. However, in practice, estimating omnidirectional lighting can be challenging due to limitations such as partial panoramas of the rendering positions, and the inherent environment lighting and mobile user dynamics. A new opportunity arises recently with the advancements in mobile 3D vision, including built-in high-accuracy depth sensors and deep learning-powered algorithms, which provide the means to better sense and understand the physical surroundings. Centering the key idea of 3D vision, in this work, we design an edge-assisted framework called Xihe to provide mobile AR applications the ability to obtain accurate omnidirectional lighting estimation in real time.Specifically, we develop a novel sampling technique that efficiently compresses the raw point cloud input generated at the mobile device. This technique is derived based on our empirical analysis of a recent 3D indoor dataset and plays a key role in our 3D vision-based lighting estimator pipeline design. To achieve the realtime goal, we develop a tailored GPU pipeline for on-device point cloud processing and use an encoding technique that reduces network transmitted bytes. Finally, we present an adaptive triggering strategy that allows Xihe to skip unnecessary lighting estimations and a practical way to provide temporal coherent rendering integration with the mobile AR ecosystem. We evaluate both the lighting estimation accuracy and time of Xihe using a reference mobile application developed with Xihe's APIs. Our results show that Xihe takes as fast as 20.67ms per lighting estimation and achieves 9.4% better estimation accuracy than a state-of-the-art neural network. CCS CONCEPTS• Computing methodologies → Mixed / augmented reality; • Human-centered computing → Ubiquitous and mobile computing systems and tools; • Computer systems organization → Distributed architectures.

show abstract

“…Object tracking can target operations on bounding boxes that follow detected objects to reduce computational workload. 4 Image-target-based tracking for augmented reality can use smaller frames when targets are nearby, 3 as shown in Figure 1. Temporal redundancy can be exploited to reuse previously detected visual features for object tracking.…”

mentioning

confidence: 99%