A Systematic Literature Review of Empirical Methods and Risk Representation in Usable Privacy and Security Research

Abstract

Usable privacy and security researchers have developed a variety of approaches to represent risk to research participants. To understand how these approaches are used and when each might be most appropriate, we conducted a systematic literature review of methods used in security and privacy studies with human participants. From a sample of 633 papers published at five top conferences between 2014 and 2018 that included keywords related to both security/privacy and usability, we systematically selected and analyzed 284 full-length papers that included human subjects studies. Our analysis focused on study methods; risk representation; the use of prototypes, scenarios, and educational intervention; the use of deception to simulate risk; and types of participants. We discuss benefits and shortcomings of the methods, and identify key methodological, ethical, and research challenges when representing and assessing security and privacy risk. We also provide guidelines for the reporting of user studies in security and privacy.

Stuck in the Permissions With You: Developer & End-User Perspectives on App Permissions & Their Privacy Ramifications

“…3.2.1 End-User Participant Recruitment. We recruited 309 3 enduser participants using Prolific [63], a typical crowdsourcing platform for recruiting participants for empirical privacy studies [20]. We decided to recruit participants residing in the U.K. because (1) privacy is a cultural and contextual topic, and people's interpretation of what privacy means can vary based on where they are located [14,69,70,97]; 4 (2) Prolific offers representative samples only for two countries, the U.S. and the U.K. [64]; (3) we had limited resources and budget; and (4) a simple translation from English into other languages could have caused different interpretations of our questions.…”

Measurement Practices in UX Research: A Systematic Quantitative Literature Review

“…However, findings from a review conducted in 2011 on UX research from 2005 to 2009 show that such practice is uncommon, with 51% of studies using self-developed scales not reporting any items (Bargas-Avila & Hornbaek, 2011). A recent literature review by Distler et al (2021) within the context of usable privacy and security research also showed that papers sometimes contain insufficient details on the study procedure and methods, including how tasks and questions were framed when presented to the participants of a study. The authors suggested that such reporting standards need to be improved, and publishers should request or require such information to increase both how understandable and how replicable research is (Distler et al, 2021).…”

“…A recent literature review by Distler et al (2021) within the context of usable privacy and security research also showed that papers sometimes contain insufficient details on the study procedure and methods, including how tasks and questions were framed when presented to the participants of a study. The authors suggested that such reporting standards need to be improved, and publishers should request or require such information to increase both how understandable and how replicable research is (Distler et al, 2021). Given the importance of sharing items (Flake & Fried, 2020) and the rising interest in fostering replication in research (Wacharamanotham et al, 2020), we want to investigate if the practice of sharing items has become more common in recent years.…”

Investigation of Information Privacy in Employment: Fundamental Knowledge and Practical Solutions for the Human-Centered Design of Measures to Preserve the Right to Informational Self-Determination in Employment

“…The goal of our literature review was to identify relevant work that has (1) empirically examined information privacy in the employer-employee relationship, (2) significantly contributed to the theoretical foundation of empirical research, and (3) applied methods of privacy engineering to the employment context. Although similar literature reviews have been conducted before, they are either focusing on non-employment contexts [112,113,114,115,116,160,161,162,163,168,268], are outdated [69], or the underlying methodology is unclear and not comprehensible [25].…”

“…To include articles from engineering research areas on the topic of (usable) privacy, we used the following repositories based on previous literature reviews [163,268]: The ACM Digital Library, the IEEE Xplore Digital Library, the USENIX Papers Search, the Sciendo search, the Springer Link repository, and the Web of Science. 1 However, because technical research areas tend to conceptualize "information privacy" simply as "privacy", the articles had to meet the following basic requirements to be considered: article ∋ (employee privacy ∧ (employee ∨ worker) ∧ (workplace ∨ employer))…”