Deep learning based face recognition system with smart glasses

Daescu, Ovidiu; Huang, Hongyao; Weinzierl, Maxwell A.

doi:10.1145/3316782.3316795

Cited by 5 publications

(6 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One of the most important and significant tasks for BVI people is to recognize the face and identity information of relatives and friends. Daescu et al [13] created a face recognition system that can receive facial images captured via the camera of smart glass based on commands from the user, process the result on the server, and thereafter return the result via audio. The system is designed as a client-server architecture, with a pair of cellphones, smart glasses, and a back-end server employed to implement face recognition using deep CNN models such as FaceNet and Inception-ResNet.…”

Section: Smart Glass System For Bvi Peoplementioning

confidence: 99%

See 1 more Smart Citation

Smart Glass System Using Deep Learning for the Blind and Visually Impaired

Mukhiddinov

2021

Electronics

View full text Add to dashboard Cite

Individuals suffering from visual impairments and blindness encounter difficulties in moving independently and overcoming various problems in their routine lives. As a solution, artificial intelligence and computer vision approaches facilitate blind and visually impaired (BVI) people in fulfilling their primary activities without much dependency on other people. Smart glasses are a potential assistive technology for BVI people to aid in individual travel and provide social comfort and safety. However, practically, the BVI are unable move alone, particularly in dark scenes and at night. In this study we propose a smart glass system for BVI people, employing computer vision techniques and deep learning models, audio feedback, and tactile graphics to facilitate independent movement in a night-time environment. The system is divided into four models: a low-light image enhancement model, an object recognition and audio feedback model, a salient object detection model, and a text-to-speech and tactile graphics generation model. Thus, this system was developed to assist in the following manner: (1) enhancing the contrast of images under low-light conditions employing a two-branch exposure-fusion network; (2) guiding users with audio feedback using a transformer encoder–decoder object detection model that can recognize 133 categories of sound, such as people, animals, cars, etc., and (3) accessing visual information using salient object extraction, text recognition, and refreshable tactile display. We evaluated the performance of the system and achieved competitive performance on the challenging Low-Light and ExDark datasets.

show abstract

Section: Smart Glass System For Bvi Peoplementioning

confidence: 99%

“…The main advantages of the proposed system over other existing systems is supporting tactile graphics generation and walking in night-time environment. Note that other existing works [13,40,45] also used a client-server architecture and increased smart glass's battery life and decreased data processing time.…”

Section: Smart Glass System For Bvi Peoplementioning

confidence: 99%

Smart Glass System Using Deep Learning for the Blind and Visually Impaired

Mukhiddinov

2021

Electronics

View full text Add to dashboard Cite

show abstract

“…There are deep learningbased applications using ambient cameras, e.g., for fall detection [9], activity recognition [46], and tracking museum visitors [30]. Wearable egocentric cameras have also used deep learning models, e.g., for predicting daily activities [3], visual assistance [31,33], visual guides [40], and face recognition [5]. There are also nonegocentric wearable cameras using deep learning, e.g., for emotion recognition [43] and eating recognition [2].…”

Section: Camera-based Deep Learning Systemsmentioning

confidence: 99%

“…We also added other sensors to PAL's platform and open-sourced PAL as a modular platform for behavior change applications [16]. Users want behavior 4 Weight Imprinting [36], 5 Pre-trained on ImageNet change support and tracking in visual contexts [18], and we envision that PAL can be used for self-tracking and context-aware behavior change applications, e.g., Just-in-time Adaptive Interventions (JITAIs) [32]. PAL's egocentric visual context detection can also be further used for other applications, e.g., memory support for people with Alzheimer's or their caretakers.…”

Section: Applicationsmentioning

confidence: 99%

Wearable System for Personalized and Privacy-preserving Egocentric Visual Context Detection using On-device Deep Learning

Khan

Fernandes

Vaish

et al. 2021

Adjunct Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization

View full text Add to dashboard Cite

Wearable egocentric visual context detection raises privacy concerns and is rarely personalized or on-device. We created a wearable system, called PAL, with on-device deep learning so that the user images do not have to be sent to the cloud for processing, and can be processed on-device in a real-time, offline, and privacy-preserving manner. PAL enables human-in-the-loop context labeling using wearable audio input/output and a mobile/web application. PAL uses on-device deep learning models for object and face detection, low-shot custom face recognition (~1 training image per person), low-shot custom context recognition (e.g., brushing teeth, ~10 training images per context), and custom context clustering for active learning. We tested PAL with 4 participants, 2 days each, and obtained ~1000 in-the-wild images. The participants found PAL easy-to-use and each model had дt80% accuracy. Thus, PAL supports wearable, personalized, and privacy-preserving egocentric visual context detection using human-in-the-loop, low-shot, and on-device deep learning. CCS CONCEPTS• Human-centered computing → Human computer interaction (HCI); Ubiquitous and mobile computing; Ubiquitous and mobile computing systems and tools.

show abstract

“…Wearable cameras are commonly used for intelligence augmentation applications [31,8] and even combined with physiological sensors [11]. Deep learning has also been used with egocentric cameras, e.g., for predicting daily activities [3], eating recognition [2], visual assistance [26,27], visual guides [30], and face recognition [5]). However, none use on-device deep learning, especially for personalized and privacy-preserving egocentric visual contexts.…”

Section: Wearable Cameras and Deep Learningmentioning

confidence: 99%

PAL: Intelligence Augmentation using Egocentric Visual Context Detection

Khan¹,

Maes²

2021

Preprint

View full text Add to dashboard Cite

Egocentric visual context detection can support intelligence augmentation applications. We created a wearable system, called PAL, for wearable, personalized, and privacy-preserving egocentric visual context detection. PAL has a wearable device with a camera, heart-rate sensor, ondevice deep learning, and audio input/output. PAL also has a mobile/web application for personalized context labeling. We used on-device deep learning models for generic object and face detection, low-shot custom face and context recognition (e.g., activities like brushing teeth), and custom context clustering (e.g., indoor locations). The models had over 80% accuracy in in-the-wild contexts (~1000 images) and we tested PAL for intelligence augmentation applications like behavior change. We have made PAL is open-source to further support intelligence augmentation using personalized and privacy-preserving egocentric visual contexts.

show abstract

Deep learning based face recognition system with smart glasses

Cited by 5 publications

References 9 publications

Smart Glass System Using Deep Learning for the Blind and Visually Impaired

Smart Glass System Using Deep Learning for the Blind and Visually Impaired

Wearable System for Personalized and Privacy-preserving Egocentric Visual Context Detection using On-device Deep Learning

PAL: Intelligence Augmentation using Egocentric Visual Context Detection

Contact Info

Product

Resources

About