Single-stroke language-agnostic keylogging using stereo-microphones and domain specific machine learning

Abstract

Mobile phones are equipped with an increasingly large number of precise and sophisticated sensors. This raises the risk of direct and indirect privacy breaches. In this paper, we investigate the feasibility of keystroke inference when user taps on a soft keyboard are captured by the stereoscopic microphones on an Android smartphone. We developed algorithms for sensor-signals processing and domain specific machine learning to infer key taps using a combination of stereo-microphones and gyroscopes. We implemented and evaluated the performance of our system on two popular mobile phones and a tablet: Samsung S2, Samsung Tab 8 and HTC One. Based on our experiments, and to the best of our knowledge, our system (1) is the first to exceed 90% accuracy requiring a single attempt, (2) operates on the standard Android QWERTY and number keyboards, and (3) is language agnostic. We show that stereo-microphones are a much more effective side channel as compared to the gyroscope, however, their data can be combined to boost the accuracy of prediction. While previous studies focused on larger key sizes and repetitive attempts, we show that by focusing on the specifics of the keyboard and creating machine learning models and algorithms based on keyboard areas combined with adequate filtering, we can achieve an accuracy of 90% -94% for much smaller key sizes in a single attempt. We also demonstrate how such attacks can be instrumentalized by a malicious application to log the keystrokes of other sensitive applications. Finally, we describe some techniques to mitigate these attacks.

Stealing PINs via mobile sensors: actual risk versus user perception

“…We do not have such an assumption as the training data are obtained from all users in the experiment. In terms of accuracy, with the exception of [12], PINlogger.js generally outperforms other works with an identification rate of 74% in the first attempt. This is a significant success rate (despite that the sampling rate in-browser is much lower than that available in-app) and confirms that the described attack imposes a serious threat to the users' security and privacy.…”

“…We consider (digit-only) PINs since they are popular credentials used by users for many purposes such as unlocking phone, SIM PIN, NFC payments, bank cards, other banking services, gaming, and other personalized applications such as health care and insurance. Unlike similar works which have to gain the access through an installed app [7][8][9][10][11][12][13][14][15][16], our attack does not require any user permission. Instead, we assume that the user has loaded the malicious web content in the form of an iframe, or another tab while working with the mobile browser as shown in Fig.…”

“…In an extended attack scenario, a more complex segmentation process would be needed to identify the start and end of a touch action. This could be achieved by measuring the peak amplitudes of a signal, as done in [12].…”

“…Spreitzer suggests another PIN Skimming attack [15] and steals a user's PIN based on the measurements from the smartphone's ambient light sensor. Narain et al introduce another attack [12] on smartphone numerical and alphabetical keyboards and the user's PINs and credit card numbers by using the smartphone microphone. TapLogger by Xu et al [16] is another attack on the smartphone numpad which outputs the pressed digits and PINs based on accelerometer and orientation sensor data.…”

“…restricting the sensor to one app, reducing the sampling rate, temporal pause of the sensor on sensitive entries such as keyboard, rearranging keyboard for password entrance, asking for explicit permission from the user, ranking apps based on their similarities to malware, and obfuscating anomalies in sensor data [7,[10][11][12][13][14][15][16]31,32]. However, after many years of research on showing the serious security risks of sensors such as accelerometer and gyroscope, none of the major mobile platforms have revised their in-app access policy.…”