Crowd-AI Camera Sensing in the Real World

Abstract

Smart appliances with built-in cameras, such as the Nest Cam and Amazon Echo Look, are becoming pervasive. They hold the promise of bringing high fidelity, contextually rich sensing into our homes, workplaces and other environments. Despite recent and impressive advances, computer vision systems are still limited in the types of sensing questions they can answer, and more importantly, do not easily generalize across diverse human environments. In response, researchers have investigated hybrid crowd- and AI-powered methods that collect human labels to bootstrap automatic processes. However, deployments have been small and mostly confined to institutional settings, leaving open questions about the scalability and generality of the approach. In this work, we describe our iterative development of Zensors++, a full-stack crowd-AI camera-based sensing system that moves significantly beyond prior work in terms of scale, question diversity, accuracy, latency, and economic feasibility. We deployed Zensors++ in the wild, with real users, over many months and environments, generating 1.6 million answers for nearly 200 questions created by our participants, costing roughly 6/10ths of a cent per answer delivered. We share lessons learned, insights gleaned, and implications for future crowd-AI vision systems.

Sense and Accessibility

“…For those aspects of our participants' challenges that can be addressed by improved technology, we see promise in emerging approaches such as general-purpose, personalizable sensor models like Zensors++ [28,40], though the setup and training processes may need simplifcation for widespread deployment to nontechnical end-users. Personalized ML approaches, such as Project Euphonia [27] (which explores making voice-activated systems work better for people with disabilities that impact their speech), also show promise, though such eforts are in early stages and the trade-ofs of personalized vs. universal approaches must be considered carefully.…”

Integrating Artificial Intelligence with Camera Systems for Automated Surveillance and Analysis

“… Evaluation Metrics:-Mean Average Precision (chart) generally used metric to assess object perfection-recall trade-offs across different IOU thresholds. Crossroad over Union (IOU) Measures the discovery performance, considering imbrication between prognosticated and ground-verity bounding boxes, pivotal for localization delicacy evaluation [6].…”

“It’s Just Part of Me:” Understanding Avatar Diversity and Self-presentation of People with Disabilities in Social Virtual Reality

“…Moreover, how to suitably label and describe human appearance, especially the marginalized groups, is another important question to consider [4,33,40]. Besides refining the algorithms and datasets from the computer vision perspective [77], HCI solutions could also be adopted, for example, indicating the potential inaccuracy of the recognized results to users [54,91], or leveraging human-AI collaboration to achieve more reliable results [32,51].…”