Investigation of evidence for the internal structure of a modified science motivation questionnaire II (mSMQ II): a failed attempt to improve instrument functioning across course, subject, and wording variants

Abstract

<p>The Science Motivation Questionnaire II (SMQ II) was developed to measure aspects of student motivation in college-level science courses. Items on the SMQ II are structured such that the word ‘science’ can be replaced with any discipline title (<italic>e.g.</italic>, chemistry) to produce a discipline-specific measure of student motivation. Since its original development as the Science Motivation Questionnaire and subsequent refinement, the SMQ II and its discipline-specific variants have been used in a number of science education studies. However, many studies have failed to produce acceptable validity evidence for their data based on the proposed internal structure of the instrument. This study investigated if modifications could be made to the SMQ II such that it produces consistent structural evidence across its use in various forms. A modified SMQ II (mSMQ II) was tested with wording variants (‘science’ and ‘biology’ or ‘chemistry’) in general biology and in preparatory and general chemistry courses at several institutions. Exploratory and confirmatory factor analysis were used to cull problematic items and evaluate the structure of the data based on the relations posited by the SMQ II developers. While extensive revisions resulted in acceptable data model fit for the five-factor structural models in most course and wording conditions, significant issues arose for the single-factor scales. Therefore, potential users are cautioned about the utility of the SMQ II or its variants to support the evaluation of classroom practices. A reflective review of the theoretical underpinnings of the SMQ II scales call into question the original framing of the scales and suggests potential alternatives for consideration.</p>

Multi-institutional Study of Self-Efficacy within Flipped Chemistry Courses

“…Results indicated no discernible pattern between first-term grades and motivation, with a pattern arising at the end of the second-term. The SMQ-II is based upon Social-Cognitive Theory (SCT), , however, the subscales draw upon multiple theoretical frameworks of motivation while also seeking an overall motivational composite score, which has resulted in complications for measurement and scale adaptability. , Given this limited number of motivation-based studies of flipped learning environments, there is still a need for the use of sound motivational theories and frameworks in investigating their impacts.…”

“…If structural validity of the data from the population under investigation is supported, then evidence is provided that the data maps onto the latent construct. However, if the structural validity of the data is not supported, investigations of the response process and/or content validity may need to be conducted. , Furthermore, if the measured data will be used to compare groups on the latent construct, evidence of consequential validity needs to be established. For self-reported quantitative data, this level of validity can be supported through measurement invariance to determine if group-bias is present in the data structure .…”

Response Process Validity Evidence in Chemistry Education Research

“…The final rubric had an acceptable Cohen's kappa value of 0.8 and was used by author Hosbein to re-code the remaining transcripts." 53 Research on the range of analytical methods employed has identified benefits and challenges with both-less intensive analyses unearth fewer response problems and may be more prone to bias, 50,54 while more intensive analyses may not be feasible-one may simply lack the resources (e.g., time, funding) to engage in a full qualitative analysis of response process data, especially if instrument administration is timedependent. 36 In response to the ambiguity in the literature about what is "best" and to serve a range of research contexts, researchers have devised qualitative methods that seek to balance feasibility and reliability-we adapted one such approach for chemistry education context in Table 1.…”

“…For example, Hosbein and Barbera (2020) asked respondents not only to read the item aloud and describe their interpretation, but also to choose an answer and describe why they chose the answer they did . Of the 43 articles, 9 addressed the question “What information do you believe to be relevant to answering this question?” by probing during their analysis but do not give explicit information regarding how the probing was conducted. ,− Only two of the articles specifically addressed student challenges when answering questions, and only one explicitly asked students, “How difficult was this question to answer?” , …”

“…After discussing discrepancies in coding, the rubric was revised and additional transcripts were independently recoded. The final rubric had an acceptable Cohen’s kappa value of 0.8 and was used to recode the remaining transcripts.” Research on the range of analytical methods employed has identified benefits and challenges with each approach. Less intensive analyses unearth fewer response problems and may be more prone to bias, , while more intensive analyses may not be feasible.…”