Estimating standard errors of estimated variance components in generalizability theory using bootstrap procedures

Moore, Joann L.

doi:10.17077/etd.caonwmhv

Cited by 8 publications

(14 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All data and analysis scripts are available for download via Open Science Framework (osf.io/bh35a). Data were pre-processed and analyzed within the R Statistical Programming Language (R Core Team, 2016 ) environment, and using the gtheory (Moore, 2016 ), psych (Revelle, 2017 ), and lme4 (Bates et al, 2015 ) packages. Graphics were created using the ggplot2 (Wickham, 2009 ) package.…”

Section: Resultsmentioning

confidence: 99%

On the Dependability and Feasibility of Layperson Ratings of Divergent Thinking

2018

View full text Add to dashboard Cite

A new system for subjective rating of responses to divergent thinking tasks was tested using raters recruited from Amazon Mechanical Turk. The rationale for the study was to determine if such raters could provide reliable (aka generalizable) ratings from the perspective of generalizability theory. To promote reliability across the Alternative Uses and Consequence task prompts often used by researchers as measures of Divergent Thinking, two parallel scales were developed to facilitate feasibility and validity of ratings performed by laypeople. Generalizability and dependability studies were conducted separately for two scoring systems: the average-rating system and the snapshot system. Results showed that it is difficult to achieve adequate reliability using the snapshot system, while good reliability can be achieved on both task families using the average-rating system and a specific number of items and raters. Additionally, the construct validity of the average-rating system is generally good, with less validity for certain Consequences items. Recommendations for researchers wishing to adopt the new scales are discussed, along with broader issues of generalizability of subjective creativity ratings.

show abstract

Section: Resultsmentioning

confidence: 99%

On the Dependability and Feasibility of Layperson Ratings of Divergent Thinking

2018

View full text Add to dashboard Cite

show abstract

“…These measures were calculated using the “rel” package (Lo Martire, 2017) in R version 3.4.0 (R Development Core Team, 2008), and were averaged across lists. We also computed the Dependability coefficient using the “gtheory” package (Moore, 2016), both averaged across lists and based on the dataset as a whole. The ICC was calculated for the mean ratings with the parameters “two-way”, and “absolute agreement”, indicating random effects for participants and items.…”

Section: Methodsmentioning

confidence: 99%

Normative Data of Dutch Idiomatic Expressions: Subjective Judgments You Can Bank on

et al. 2019

View full text Add to dashboard Cite

The processing of idiomatic expressions is a topical issue in empirical research. Various factors have been found to influence idiom processing, such as idiom familiarity and idiom transparency. Information on these variables is usually obtained through norming studies. Studies investigating the effect of various properties on idiom processing have led to ambiguous results. This may be due to the variability of operationalizations of the idiom properties across norming studies, which in turn may affect the reliability of the subjective judgements. However, not all studies that collected normative data on idiomatic expressions investigated their reliability, and studies that did address the reliability of subjective ratings used various measures and produced mixed results. In this study, we investigated the reliability of subjective judgements, the relation between subjective and objective idiom frequency, and the impact of these dimensions on the participants’ idiom knowledge by collecting normative data of five subjective idiom properties (Frequency of Exposure, Meaning Familiarity, Frequency of Usage, Transparency, and Imageability) from 390 native speakers and objective corpus frequency for 374 Dutch idiomatic expressions. For reliability, we compared measures calculated in previous studies, with the D-coefficient, a metric taken from Generalizability Theory. High reliability was found for all subjective dimensions. One reliability metric, Krippendorff’s alpha, generally produced lower values, while similar values were obtained for three other measures (Cronbach’s alpha, Intraclass Correlation Coefficient, and the D-coefficient). Advantages of the D-coefficient are that it can be applied to unbalanced research designs, and to estimate the minimum number of raters required to obtain reliable ratings. Slightly higher coefficients were observed for so-called experience-based dimensions (Frequency of Exposure, Meaning Familiarity, and Frequency of Usage) than for content-based dimensions (Transparency and Imageability). In addition, fewer raters were required to obtain reliable ratings for the experience-based dimensions. Subjective and objective frequency appeared to be poorly correlated, while all subjective idiom properties and objective frequency turned out to affect idiom knowledge. Meaning Familiarity, Subjective and Objective Frequency of Exposure, Frequency of Usage, and Transparency positively contributed to idiom knowledge, while a negative effect was found for Imageability. We discuss these relationships in more detail, and give methodological recommendations with respect to the procedures and the measure to calculate reliability.

show abstract

“…For example, Moore (2010) compared the bias and standard error of estimation of traditional methods and bootstrap method for estimating the standard error of the estimated variance components in several test design including single‐facet crossed design. Moore (2010) questioned the adequacy of the traditional method in estimating the standard errors of variance components and found that bootstrap method performed better for normal data and less for non‐normal data. In addition, Moore (2010) found the adequacy of the traditional method was more questionable with smaller sample size.…”

Section: Statistic Estimators Traditional ε̂2mentioning

confidence: 99%

“…Moore (2010) questioned the adequacy of the traditional method in estimating the standard errors of variance components and found that bootstrap method performed better for normal data and less for non‐normal data. In addition, Moore (2010) found the adequacy of the traditional method was more questionable with smaller sample size.…”

Section: Statistic Estimators Traditional ε̂2mentioning

confidence: 99%

“…Several studies investigated the performance of the traditional method for estimating the standard errors of the estimated variance components (Moore, 2010; Gao & Brennan, 2001; Hayashi & Kamata, 2005). For example, Moore (2010) compared the bias and standard error of estimation of traditional methods and bootstrap method for estimating the standard error of the estimated variance components in several test design including single‐facet crossed design. Moore (2010) questioned the adequacy of the traditional method in estimating the standard errors of variance components and found that bootstrap method performed better for normal data and less for non‐normal data.…”

Section: Statistic Estimators Traditional ε̂2mentioning

confidence: 99%

See 1 more Smart Citation

Standard Errors of Variance Components, Measurement Errors and Generalizability Coefficients for Crossed Designs

Almehrizi

2020

J Educational Measurement

View full text Add to dashboard Cite

Estimates of various variance components, universe score variance, measurement error variances, and generalizability coefficients, like all statistics, are subject to sampling variability, particularly in small samples. Such variability is quantified traditionally through estimated standard errors and/or confidence intervals. The paper derived new standard errors for all estimated statistics for two crossed designs (single‐facet design and two‐facet design) in generalizability theory. The derivation was based on the assumption of multivariate normal distribution for observation scores using delta method. The derivation was differentiating between fixed and random facets. The adequacy of the derived standard errors was examined using Mont Carlo simulation for the two designs under several test conditions and compared to the traditional existing methods as well as implementation on real data. Results showed that the derived standard errors for all estimators are converging to the empirical standard errors for all simulation conditions with the two designs for both normal and non‐normal continuous data.

show abstract

Estimating standard errors of estimated variance components in generalizability theory using bootstrap procedures

Cited by 8 publications

References 21 publications

On the Dependability and Feasibility of Layperson Ratings of Divergent Thinking

On the Dependability and Feasibility of Layperson Ratings of Divergent Thinking

Normative Data of Dutch Idiomatic Expressions: Subjective Judgments You Can Bank on

Standard Errors of Variance Components, Measurement Errors and Generalizability Coefficients for Crossed Designs

Contact Info

Product

Resources

About