Rhythmic pixel regions: multi-resolution visual sensing system towards high-precision visual computing at low power

Kodukula, Venkatesh; Shearer, Alexander; Nguyen, Van Sinh; Lingutla, Srinivas; Liu, Yifei; LiKamWa, Robert

doi:10.1145/3445814.3446737

Cited by 13 publications

(10 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, the table tennis coach focuses on the wrist action, while the golf coach focuses on weight transfer. This finding motivates us to integrate an adaptive rendering policy [2] based on the viewer's attention.…”

Section: Introductionmentioning

confidence: 94%

Adaptive 5G systems for interactive volumetric sports analysis in augmented reality

Wen

Gold

et al. 2022

Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services

Self Cite

View full text Add to dashboard Cite

Remote coaching for sports is challenged by the lack of 3D spatial communication. While athletes send live or recorded videos to their coaches, these 2D representations fail to capture the spatial relationships of the body, limiting the ability to understand timing, weight distribution, and smoothness in an athletic movement. This demonstration presents Augmented Coach, an AR sports coaching platform for coaches to remotely view, manipulate, and annotate athletic movements in 3D augmented space. Also, this demonstration provides an adaptive platform to study real-time efficient volumetric data transmission between remotely connected devices, including over 5G cellular networks. CCS CONCEPTS• Human-centered computing → Ubiquitous and mobile computing systems and tools.

show abstract

Section: Introductionmentioning

confidence: 94%

Adaptive 5G systems for interactive volumetric sports analysis in augmented reality

Wen

Gold

et al. 2022

Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services

Self Cite

View full text Add to dashboard Cite

show abstract

“…ROIs have been exploited for image compression [41], [42] in the past. Recent studies have been diving into how ROI can be leveraged for speed benefits [43]- [45]. Other works have discussed how we can power-gate image sensor resolution for high energy savings [46].…”

Section: Related Workmentioning

confidence: 99%

“…However, we were not able to find a camera that was able to dynamically adjust its ROI while being compatible with our FPGA evaluation board. Given this, we chose to digitally simulate ROI for demonstration purposes as performed in other papers in the literature [94]. The image capture, preprocessing, digital ROI, Kalman filter update and prediction steps, and any other postprocessing required by each specific tracker is done in the processing system of the FPGA board.…”

Section: A Hardware Implementationmentioning

confidence: 99%

Adaptive Subsampling for ROI-Based Visual Tracking: Algorithms and FPGA Implementation

et al. 2022

View full text Add to dashboard Cite

There is tremendous scope for improving the energy efficiency of embedded vision systems by incorporating programmable region-of-interest (ROI) readout in the image sensor design. In this work, we study how ROI programmability can be leveraged for vision applications by anticipating where the ROI will be located in future frames and switching pixels off outside of this region. We refer to this process of ROI prediction and corresponding sensor configuration as adaptive subsampling. Our adaptive subsampling algorithms comprise an object detector and an ROI predictor (Kalman filter) which operate in conjunction to optimize the energy efficiency of the vision pipeline with the end task being object tracking. To further facilitate the implementation of our adaptive algorithms in real systems, we select a candidate algorithm and map it onto an FPGA. Leveraging Xilinx Vitis AI tools, we designed and accelerated a YOLO object detector-based adaptive subsampling algorithm. In order to further improve the algorithm post-deployment, we evaluated several competing baselines on the OTB100 and LaSOT datasets. We found that coupling the ECO tracker with the Kalman filter has a competitive AUC score of 0.4568 and 0.3471 on the OTB100 and LaSOT datasets respectively. Further, the power efficiency of this algorithm is on par with, and in a couple of instances superior to, the other baselines. The ECO-based algorithm incurs a power consumption of approximately 4 W averaged across both datasets while the YOLO-based approach requires power consumption of approximately 6 W (as per our power consumption model). In terms of accuracy-latency tradeoff, the ECO-based algorithm provides near-real-time performance (19.23 FPS) while managing to attain competitive tracking precision.INDEX TERMS Adaptive subsampling, embedded computer vision, FPGA acceleration, hardware/software co-design, single object tracking, vision applications

show abstract

“…ROIs have been exploited for image compression [42], [43] in the past. Recent studies have been diving into how ROI can be leveraged for speed benefits [44], [45], [46]. Other works have discussed how we can power-gate image sensor resolution for high energy savings [47].…”

Section: Related Workmentioning

confidence: 99%

“…However, we were not able to find a camera that was able to dynamically adjust its ROI while being compatible with our FPGA evaluation board. Given this, we chose to digitally simulate ROI for demonstration purposes as performed in other papers in the literature [75]. The image capture, preprocessing, digital ROI, Kalman filter update and prediction steps, and any other postprocessing required by each specific tracker is done in the processing system of the FPGA board.…”

Section: A Hardware Implementationmentioning

confidence: 99%

Adaptive Subsampling for ROI-based Visual Tracking: Algorithms and FPGA Implementation

Iqbal¹,

Muro²,

Katoch³

et al. 2021

Preprint

View full text Add to dashboard Cite

There is tremendous scope for improving the energy efficiency of embedded vision systems by incorporating programmable region-of-interest (ROI) readout in the image sensor design. In this work, we study how ROI programmability can be leveraged for tracking applications by anticipating where the ROI will be located in future frames and switching pixels off outside of this region. We refer to this process of ROI prediction and corresponding sensor configuration as adaptive subsampling. Our adaptive subsampling algorithms comprise an object detector and an ROI predictor (Kalman filter) which operate in conjunction to optimize the energy efficiency of the vision pipeline with the end task being object tracking. To further facilitate the implementation of our adaptive algorithms in real life, we select a candidate algorithm and map it onto an FPGA. Leveraging Xilinx Vitis AI tools, we designed and accelerated a YOLO object detector-based adaptive subsampling algorithm. In order to further improve the algorithm post-deployment, we evaluated several competing baselines on the OTB100 and LaSOT datasets. We found that coupling the ECO tracker with the Kalman filter has a competitive AUC score of 0.4568 and 0.3471 on the OTB100 and LaSOT datasets respectively. Further, the power efficiency of this algorithm is on par with, and in a couple of instances superior to, the other baselines. The ECO-based algorithm incurs a power consumption of approximately 4 W averaged across both datasets while the YOLO-based approach requires power consumption of approximately 6 W (as per our power consumption model). In terms of accuracy-latency tradeoff, the ECO-based algorithm provides near-real-time performance (19.23 FPS) while managing to attain competitive tracking precision.

show abstract

Rhythmic pixel regions: multi-resolution visual sensing system towards high-precision visual computing at low power

Cited by 13 publications

References 18 publications

Adaptive 5G systems for interactive volumetric sports analysis in augmented reality

Adaptive 5G systems for interactive volumetric sports analysis in augmented reality

Adaptive Subsampling for ROI-Based Visual Tracking: Algorithms and FPGA Implementation

Adaptive Subsampling for ROI-based Visual Tracking: Algorithms and FPGA Implementation

Contact Info

Product

Resources

About