How do you Behave as a Psychometrician? Research Conduct in the Context of Psychometric Research

Abstract

The identification of fraudulent and questionable research conduct is not something new. However, in the last 12 years the aim has been to identify specific problems and concrete solutions applicable to each area of knowledge. For example, previous work has focused on questionable and responsible research conducts associated with clinical assessment, measurement practices in psychology and related sciences; or applicable to specific areas of study, such as suicidology. One area of study that merits further study of questionable and responsible research behaviors is psychometrics. Focusing on psychometric research is important and necessary, as without adequate evidence of construct validity the overall validity of the research is at least debatable. Our interest here is to (a) identifying questionable research conduct specifically linked to psychometric studies; and (b) promoting greater awareness and widespread application of responsible research conduct in psychometrics research. We believe that the identification and recognition of these conducts is important and will help us to improve our daily work as psychometricians.

The State of Open Science Practices in Psychometric Studies of Suicide: A Systematic Review

“…Firstly, the majority of manuscripts were published under restricted access, limiting the availability of research findings to a broader audience (Table 2). Concerningly, none of the included manuscripts utilized preregistration, an essential tool in countering questionable research practices and particularly relevant in psychometric research (Christensen et al, 2019;Gomes et al, 2022;Nosek et al, 2015;Tackett et al, 2019;Flores-Kanter & Mosquera, 2023).…”

“…Within psychometric studies, recent research has identified questionable research conduct that can potentially threaten the validity and reliability of psychometric research results (Manapat et al, 2023). To ensure more rigorous and reliable psychometric research, good practices associated with transparency and open science have been recommended (Flake & Fried, 2020;Flores-Kanter & Mosquera, 2023). These include the adoption of open science practices such as preregistration, pre-printing, and the sharing of study materials, code, and data.…”

“…OPEN SCIENCE IN PSYCHOMETRIC SUICIDE RESEARCH In the past decade, there has been increasing emphasis on adopting open science practices as effective means to counter questionable research practices or behaviors (Banks et al, 2019;Munafò et al, 2017;Segerstrom et al, 2023;Wigboldus & Dotsch, 2016). This call has been made in general and has been particularly highlighted in specific fields such as psychometrics (Flake & Fried, 2020;Flores-Kanter & Mosquera, 2023;Manapat et al, 2023) and suicide research (Carpenter & Law, 2021;Kirtley et al, 2022). For instance, preregistration of hypotheses and analytic plans before data collection and open sharing of data can help reduce publication bias, which is characterized by selectively publishing positive results that reach statistical significance.…”

“…Preregistration also serves as a safeguard against post hoc hypotheses and analyses presented as if they were planned a priori, a practice known as HARKing (Hypothesizing After the Results are Known) and p-hacking. In the domain of psychometric studies, preregistration of research projects/plans is proposed as a valuable approach to address specific forms of selective information, such as fit-hacking (employing different strategies to achieve an acceptable or optimal model fit) and model-HARKing (constructing the entire document to align with the model that exhibits the best fit) (Flores-Kanter & Mosquera, 2023). By pre-registering research projects/plans, psychometric researchers can clearly outline essential aspects, including the measurement and structural models to be estimated, as well as the fit indicators to be considered.…”

“…It is important to highlight that open science practices are not solely advocated to reduce questionable research practices but also to foster a better research culture and enhance the openness and comprehensibility of the scientific process underlying research findings (Banks et al, 2018;Chin et al, 2021). This principle also applies to the domain of psychometric studies (Flake & Fried, 2020;Flores-Kanter & Mosquera, 2023). Open access options ensure that all relevant information and materials necessary to fully understand the research process (e.g., design OPEN SCIENCE IN PSYCHOMETRIC SUICIDE RESEARCH protocols, measures, analytic scripts) and the achieved results (e.g., data) are openly accessible.…”