Assessing the Degree of Ecological Validity of Your Study: Introducing the Multidimensional Assessment of Research in Context (MARC) Tool

Abstract

In cognitive neurosciences, fundamental principles of mental processes and functional brain organization have been established with highly controlled tasks and testing environments. Recent technical advances allowed the investigation of these functions and their brain mechanisms in naturalistic settings. The diversity in those approaches have been recently (Matusz, P. J., Dikker, S., Huth, A. G., & Perrodin, C. (2019). Are we ready for real‐world neuroscience? Journal of Cognitive Neuroscience, 31(3), 327–338. doi:10.1162/jocn_e_01276) classified via a three‐category cycle, including controlled laboratory, partially naturalistic laboratory, and naturalistic real‐world research. Based on this model, we developed the Multidimensional Assessment of Research in Context (MARC) tool to easily delineate the approach researchers have taken in their study. MARC provides means to describe the degree of ecological validity for each component of a study (e.g., sample, stimuli, measures, etc.), and the study's location on the cycle. The tool comprises seven questions concerning a study's characteristics. It outputs a summary of those and a compass plot, which can be used for presentations, preregistration, grant proposals, and papers. It aims to improve drawing conclusions across studies and raise awareness about the generalizability of research findings.

Toward an explicit technology of ecological validity

“…Notwithstanding this important limitation, the development of a tool to assess ecological validity addresses a call by researchers conducting reviews of the topic in behavior analysis (e.g., Chezan et al, 2022; Ledford et al, 2016) and aligns with recent innovations in other fields (e.g., cognitive neuroscience; Naumann et al, 2022). One unique feature of our tool is the consideration of informational sources (e.g., past research, stakeholders, participants) that commonly influence researcher behavior.…”

“…Thus, we encourage the use of this tool across a broad range of applications in behavior analysis, some of which may not have been adequately represented in our review. The best way to accomplish this would be to develop an online and open‐sourced version of the tool, like that achieved in cognitive neuroscience by Naumann et al (2022), and to update the tool as needed. Additional research on the use of the tool (e.g., its iterative use in designing research, evaluative use in the editorial process, and comparative use in published reviews) and its influence on researcher behavior is warranted.…”

Socioeconomic Status and Reading Development: Moving from “Deficit” to “Adaptation” in Neurobiological Models of Experience‐Dependent Learning

“…Thus, the decontextualized nature of these tasks may help to explain the presence of observed SES disparities, highlighting the importance of using context‐rich measures for maximum predictive validity and suggesting a deeper consideration of measures selection during experimental design (Doebel, 2020). Certain tools, such as Naumann and colleagues' Multidimensional Assessment of Research in Context (MARC), may help developmental scientists evaluate the ecological validity of measures throughout the cycle of a given research study (Naumann et al, 2022).…”

Evaluating What Works in the Classroom - How French Primary School Teachers Apply the Science of Learning

“…quantitative, qualitative, art-based) and protocols (i.e. design-based research, case studies) might be kept in mind to creatively address educational needs, while checking the degree of ecological validity of the proposed studies (Naumann et al, 2022).…”