The Use of Mobile Devices in High‐stakes Remotely Delivered Assessments and Testing

Abstract

With Internet access no longer restricted to desktop and laptop computers, job applicants now have the opportunity to complete remotely delivered assessments on mobile, handheld small screen devices such as smartphones, and personal digital assistants. In this study, a large dataset is used to investigate demographic and score differences between job applicants who completed a remotely delivered high-stakes assessment on a mobile device and those who completed it on a nonmobile device. Based on a sample of 3,575,207 job applicants who completed an unproctored Internet-based assessment between January 2011 and April 2012, the percentage of applicants completing the assessment on a mobile device was small, 1.93%, but nevertheless represented more than 69,000 people. Overall, there were small test-taker demographic differences in the use of mobile devices versus nonmobile devices in that mobile devices were slightly more likely to be used by women, AfricanAmericans and Hispanics, and younger applicants. Scores on a personality measure were similar for mobile and nonmobile devices but scores on a general mental ability test were substantially lower for mobile devices. Tests of measurement invariance also indicated equivalence across the mobile and nonmobile samples. Test taker and organizational implications for completing remotely delivered high-stakes noncognitive and cognitive assessments on mobile versus nonmobile devices are discussed.

The comparison of mobile devices to computers for web-based assessments

“…This is an important difference since prior research has shown that the user interface can have an impact on computer and web-based test scores (Bridgeman, Lennon, & Jackenthal, 2003;Huff, Cline, & Guynes, 2012;Květon, Jelínek, Vobořil, & Klimusová, 2007;Schroeders and Wilhelm, 2010;Wästlund et al's, 2008). More specifically to this research, very recent research have found differences in mean test scores between mobile devices and traditional computers (Arthur et al, 2014;Illingworth et al, 2014;Morelli et al, 2014;Sanchez & Branaghan, 2011;Sanchez & Goolsbee, 2010). However, the effect sizes of those differences have traditionally been small and very little is known as to why the differences exist at all.…”

“…What research that does exists (Arthur, Doverspike, Muñoz, Taylor, & Carr, 2014;Illingworth, Morelli, Scott, & Boyd, 2014;Morelli, Mahan, & Illingworth, 2014;Sanchez & Branaghan, 2011;Sanchez & Goolsbee, 2010) demonstrates that assessments can be impacted by the type of device that was used to completed it and that this impact is not necessarily consistent across different types of assessments. What little we know about the differences that exist is limited by the lack of experiments investigating issues that may cause differences on assessments across devices.…”

“…These numbers both increased from the prior year, which were 7% and 9% respectively (Fallaw, Kantrowitz, & Dawson, 2012). Research by Arthur et al (2014) revealed that 1.9% (almost 70,000 individuals) of a sample of more than 3.5 million completed unproctored assessments on a mobile device. Similarly, research by Illingworth et al (2014) revealed that just over .8%, or about 8000 applicants of their sample of over 900,000 completed pre-employment assessments on a mobile device whereas Morelli et al (2014) reported that .3% and .5% of their samples of over 209,000 and 375,000, respectively.…”

“…If a user is able to effectively complete the assessment, then there should be an absence of errors on the questionnaire. Prior research has found some mean score differences between mobile and traditional computers for a variety of assessments (Arthur et al, 2014;Illingworth et al, 2014;Morelli et al, 2014) for questionnaires with unknown levels of optimization. However most of these studies reported small to negligible effect sizes.…”

“…If the two devices are equal in usability, then there would be no differences in the time it would take to complete the questionnaires. However, since computers and mobile devices are different in the amount and/or quality of information that can be displayed on the screen (Sanchez & Branaghan, 2011;Sanchez & Goolsbee, 2010) and differences in screen size have been associated with differences in time to complete the assessments on optimized web-based assessments (Arthur et al, 2014;Sanchez & Branaghan, 2011), then differences in time to complete the assessment would be expected in this situation as well. This leads to the second hypothesis:…”