Understanding Face and Eye Visibility in Front-Facing Cameras of Smartphones used in the Wild

Abstract

Figure 1. In our dataset of photos taken from front-facing cameras of smartphones used in the wild, the face is visible in only 29% of the cases. However, the eyes, and not the whole face, are visible 48% of the time. We derive multiple implications for face and eye detection on mobile devices. For example, our analysis suggests that gaze estimation on mobile devices should rely less on full-face images but estimate gaze based on the eyes only instead.

The Role of Eye Gaze in Security and Privacy Applications: Survey and Future HCI Research Directions

“…Explicit gaze-based authentication was mainly investigated in the lab. As eye tracking technologies become cheaper [68], and soon to become ubiquitous on handheld mobile devices [73,75], an emerging research opportunity is to evaluate authentication schemes in the wild. This would allow studying learnability to investigate if users' performance improve after continued daily usage, as well as social implications when using the schemes in public, such as social embarrassment, or unintentionally looking at bystanders when performing gaze input.…”

Headbang: Using Head Gestures to Trigger Discrete Actions on Mobile Devices

“…A limitation of Headbang could be that according to Khamis et. al [18] the user's head is not always visible to the front camera of the device while it is used. This could lead to the problem that the device cannot recognize the user's head gestures.…”

EyeSpot: Leveraging Gaze to Protect Private Text Content on Mobile Devices from Shoulder Surfing

“…Brudy et al used a Kinect to estimate the gaze direction of passers-by in front of a large public display and visualized it to users of the display to make them aware of shoulder surfers [13]. While this approach works well for public displays, detecting the gaze direction of bystanders with the front-facing camera of a user's mobile device is often unfeasible due to its narrow-angle lens [3,34].…”

“…While eye tracking has become feasible using front-facing cameras of mobile devices, there are still open challenges [30,34]. Therefore, we opted for using a mobile eye tracker and markers on the corners of the phone's screen.…”

“…Although we used an external eye tracker, advances in computing power and front-facing cameras (e.g., depth camera in iPhone X) promise highly accurate gaze estimation on commodity mobile devices [30]. A challenge that was identified in prior work is that users do not always hold smartphones in a way such that the front-facing camera can detect their faces [34]. However, this is expected to improve over time, as more gaze estimation methods continue to develop that do not rely on an entirely visible face, but rather only visible eyes [54].…”