An Information-Processing-Based Conceptual Framework of the Effects of Unproctored Internet-Based Testing Devices on Scores on Employment-Related Assessments and Tests

Arthur, Winfred; Keiser, Nathanael L.; Doverspike, Dennis

doi:10.1080/08959285.2017.1403441

Cited by 25 publications

(35 citation statements)

References 69 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Morelli, Potosky, Arthur, and Tippins (2017) note that "reactive equivalence" studies comparing assessment modes are not theoretically (or even practically) informative as they do not address why mode differences occur or the reasons for the construct non-equivalence. Frameworks such as Potosky's (2008) and Arthur, Keiser, and Doverspike's (2018) indicate ways in which different assessment delivery devices might differ; these need to be expanded and tested to more systematically understand whether and why more efficient delivery of assessments might be a more rather than less accurate way to measure (see Apers & Derous, 2017, for an example in the context of resume screening). Morelli et al (2017) make a strong case for better theory-based predictions that go beyond considering variance associated with technology use as "construct irrelevant."…”

Section: Effects Of Efficiencymentioning

confidence: 99%

The Unrealized Potential of Technology in Selection Assessment

Ryan

Derous

2019

Revista De Psicología Del Trabajo Y De Las Organizaciones

View full text Add to dashboard Cite

The Unrealized Potential of Technology in Selection Assessment Over the past two decades, assessment for employee selection has undergone radical changes due to technological advances (see Tippins, 2015, for a review). Examples include changes in assessment delivery (computerized, online, mobile, use of adaptive testing), changes in assessment content (greater use of video and audio, graphics, gamification), changes in interactivity (recording of applicants, video interviewing), changes in scoring and reporting (quicker, internet delivered), and changes in many other aspects (e.g., use of algorithms and data mining). The purpose of this paper is not to provide another review of these advances as quality treatments exist (Scott, Bartram, & Reynolds, 2017; Tippins, 2015) but to discuss 3 ways incorporation and use of technological advances in assessment practice and research could be strengthened. First, we discuss the need to think more creatively regarding what is measured. That is, despite the promise of technology as a means of measuring new KSAOs (i.e., knowledge, skills, abilities, and other constructs like motivation) and richer content, many of our assessments are just "transported" traditional tools. As an example, gamified assessments (Bhatia & Ryan, 2018) most often are measuring traditional constructs in environments with game elements layered on.

show abstract

Section: Effects Of Efficiencymentioning

confidence: 99%

The Unrealized Potential of Technology in Selection Assessment

Ryan

Derous

2019

Revista De Psicología Del Trabajo Y De Las Organizaciones

View full text Add to dashboard Cite

show abstract

“…Whereas Potosky (2008) framed the technological attributes that affect the communication quality between two parties, the structural characteristics and information processing (SCIP) framework (Arthur, Keiser, & Doverspike, 2017) is a conceptualization of technology that assumes structural attributes affect the assessee's cognition. Specifically, this model explains how an unproctored Internet test's (UIT) device type introduces construct-irrelevant variance by affecting the respondents' cognitive load.…”

Section: Arthur Keiser and Doverspike's (2017) Scip Frameworkmentioning

confidence: 99%

“…Specifically, this model explains how an unproctored Internet test's (UIT) device type introduces construct-irrelevant variance by affecting the respondents' cognitive load. Arthur, Keiser, et al (2017) preface the development of their model with a literature review of the 23 published and unpublished studies on UIT device types. The review's summary of the literature's findings indicated that although mobile and nonmobile cognitive and noncognitive assessments do not differ in terms of psychometric properties, such as factor structure, the reliability of scores, and differential item functioning, among others, cognitive assessments on mobile devices typically result in lower scores than those on nonmobile devices.…”

Section: Arthur Keiser and Doverspike's (2017) Scip Frameworkmentioning

confidence: 99%

“…A typical approach to understanding mobile-delivered tests has been to administer a test via smartphone and examine all the device's properties one by one (e.g., mobility, the operating system used, the screen size). However, Arthur, Keiser et al (2017) asked important research questions that arise from this approach: Where is the point on the device structural continuum that score differences manifest? Why?…”

Section: Where Do We Go From Here?mentioning

confidence: 99%

See 1 more Smart Citation

A Call for Conceptual Models of Technology in I-O Psychology: An Example From Technology-Based Talent Assessment

Morelli¹,

Potosky

Arthur

et al. 2017

Ind. Organ. Psychol.

Self Cite

View full text Add to dashboard Cite

The rate of technological change is quickly outpacing today's methods for understanding how new advancements are applied within industrial-organizational (I-O) psychology. To further complicate matters, specific attempts to explain observed differences or measurement equivalence across devices are often atheoretical or fail to explain why a technology should (or should not) affect the measured construct. As a typical example, understanding how technology influences construct measurement in personnel testing and assessment is critical for explaining or predicting other practical issues such as accessibility, security, and scoring. Therefore, theory development is needed to guide research hypotheses, manage expectations, and address these issues at this intersection of technology and I-O psychology. This article is an extension of a Society for Industrial and Organizational Psychology (SIOP) 2016 panel session, which (re)introduces conceptual frameworks that can help explain how and why measurement equivalence or nonequivalence is observed in the context of selection and assessment. We outline three potential conceptual frameworks as candidates for further research, evaluation, and application, and argue for a similar conceptual approach for explaining how technology may influence other psychological phenomena.

show abstract

“…Industrial-organizational psychology has an opportunity to provide some important theoretical research and empirical data to identify the best practices for companies considering utilizing these technologies, particularly VR technology. The structural characteristics and information processing (SCIP) framework may be particularly useful in this endeavor (Arthur, Keiser, & Doverspike, 2017). There appears to be a wealth of empirical data being generated in other fields (e.g., education, instructional design) that could aid in development of additional theoretical models that could be applied to understanding the role of technology in developing effective training programs.…”

mentioning

confidence: 99%

The Need for Conceptual Models of Technology in Training and Development: How Immersive Does Training Need to Be?

Cox

House

Lopez

et al. 2017

Ind. Organ. Psychol.

View full text Add to dashboard Cite

Morelli, Potosky, Arthur, and Tippins (2017) articulate a strong need for industrial and organizational (I-O) psychologists to develop a more theory-based understanding of the role of technology in employee selection and assessment. We agree with their concerns but argue that this issue should include examination of how technology impacts training also. Researchers have noted that training is increasingly important for firms, and technology-enhanced training can improve learning and transfer (Ford & Meyer, 2013). However, the arguments that the authors make about the need for a theory-driven approach for examining the impact of technology on selection applies to training outcomes as well. Although considerable evidence exists that workplace training is effective and that technology can impact the success of training, there has been less theory-driven research exploring how technology can enhance or detract from training success. Researchers have already identified several variables related to technology that promote learning, but one variable that remains consistently unexplored in the organizational literature is immersion. This research is particularly important given how increasingly accessible immersive technology, such as virtual reality (VR), is becoming. Thus, we argue that as virtual training environments become more widely available, the variable of “immersion” in training environments is a particularly important one that warrants additional research.

show abstract

An Information-Processing-Based Conceptual Framework of the Effects of Unproctored Internet-Based Testing Devices on Scores on Employment-Related Assessments and Tests

Cited by 25 publications

References 69 publications

The Unrealized Potential of Technology in Selection Assessment

The Unrealized Potential of Technology in Selection Assessment

A Call for Conceptual Models of Technology in I-O Psychology: An Example From Technology-Based Talent Assessment

The Need for Conceptual Models of Technology in Training and Development: How Immersive Does Training Need to Be?

Contact Info

Product

Resources

About