Estimating Reliability Under a Generalizability Theory Model for Test Scores Composed of Testlets

Lee, Guemin; Frisbie, David A.

doi:10.1207/s15324818ame1203_2

Cited by 39 publications

(48 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yet, the G and Phi coefficients estimated by keeping the number of items constant and by increasing the number of testlets rose more than the G and the Phi coefficients estimated by keeping the number items constant and increasing the number of items in each testlet. This was an important finding parallel to results obtained found in the literature (Hendrickson, 2001;Lee & Frisbie, 1999). That is to say, the increase in the number of testlets contributed more to reliability than the increase in the number of items in the testlets did.…”

Section: The D Study Findingssupporting

confidence: 86%

“…The DIF analyses performed in these studies are considered at item level Wainer & Thissen, 1996). Other studies calculating the sums or averages of the items constituting testlets, and then obtaining the scores at the testlet level are also available in the literature (Lee, Dunbar, & Frisbie, 2001;Lee & Frisbie, 1999;Sireci et al, 1991).…”

Section: Differential Item Functioning (Dif)mentioning

confidence: 99%

See 1 more Smart Citation

The Effects of Testlets on Reliability and Differential Item Functioning

2015

EDUC SCI-THEOR PRACT

View full text Add to dashboard Cite

Reliability and differential item functioning (DIF) analyses were conducted on testlets displaying local item dependence in this study. The data set employed in the research was obtained from the answers given by 1500 students to the 20 items included in six testlets given in English Proficiency Exam by the School of Foreign Languages of a state University in Turkey. One of the purposes of this study was to determine the influences of the tests composed of testlets on reliability, so the reliability coefficients obtained for cases where the influences of testlets were considered and those for cases where the testlet influences were not considered were compared. In consequence of the G theory analyses conducted in this context, it was found that the G and Phi coefficients estimated by not considering the testlet effects were higher than those estimated by considering the testlet effects. It was concluded that the reliability was estimated to be relatively higher when the influences of the testlet were not considered. Two methods were used in this study so as to determine the effects of testlets on differential item functioning and the results were compared. In the DIF-determining method considering the testlet effect, both the number of items displaying DIF at the significant and estimated levels of DIF were found to be higher than in the method not considering the testlet effect.

show abstract

Section: The D Study Findingssupporting

confidence: 86%

Section: Differential Item Functioning (Dif)mentioning

confidence: 99%

The Effects of Testlets on Reliability and Differential Item Functioning

2015

EDUC SCI-THEOR PRACT

View full text Add to dashboard Cite

show abstract

“…The variance components of the score effects were estimated, and reliability estimates were obtained and compared for each method. Methods based on mean-squares are applied to the GE-NOVA and urGENOVA programs in estimating variance components for the generalizability theory models (Lee 2002;Lee and Frisbie 1999). We used default estimation methods of HLM 6.0 in this study: restricted maximum likelihood method (REML) for two-level models and full information maximum likelihood method (FIML) for three-level models.…”

Section: Analysesmentioning

confidence: 99%

Estimating reliability of school-level scores using multilevel and generalizability theory models

Jeon

Lee

Hwang

et al. 2009

Asia Pacific Educ. Rev.

Self Cite

View full text Add to dashboard Cite

The purpose of this study was to investigate the methods of estimating the reliability of school-level scores using generalizability theory and multilevel models. Two approaches, 'student within schools' and 'students within schools and subject areas,' were conceptualized and implemented in this study. Four methods resulting from the combination of these two approaches with generalizability theory and multilevel models were compared for both balanced and unbalanced data. The generalizability theory and multilevel models for the 'students within schools' approach produced the same variance components and reliability estimates for the balanced data, while failing to do so for the unbalanced data. The different results from the two models can be explained by the fact that they administer different procedures in estimating the variance components used, in turn, to estimate reliability. Among the estimation methods investigated in this study, the generalizability theory model with the 'students nested within schools crossed with subject areas' design produced the lowest reliability estimates. Fully nested designs such as (students:schools) or (subject areas:students:schools) would not have any significant impact on reliability estimates of school-level scores. Both methods provide very similar reliability estimates of school-level scores.

show abstract

“…Although there are only relative decisions made in CTT, there are two different types of decisions as relative and absolute since there are two different types of error variance as relative and absolute in G theory (Yin and Shavelson, 2008;Brennan, 2001;Brennan, 1992;Shavelson & Webb, 1991). The relative error variance of G-theory, which is used in relative decisions, can be thought of as an analog to the error variance of CTT (Lee & Frisbie, 1999).…”

Section: Introductionmentioning

confidence: 99%

Using generalizability theory to investigate the reliability of peer assessment

2016

View full text Add to dashboard Cite

In this study, the effectiveness of peer assessment, which has an important role in measurement and evaluation, was attempted to be defined. For this purpose, performance task, which is one of the alternative assessment techniques, was evaluated with the help of a scoring rubric prepared by the researchers. As a basic research, the working group was 41 sophomore students and their instructor. Three of 41 students were acted as rater and they rated their 38 peers' performances with the instructor. The analysis of the data was carried out by using fully crossed two-facet design (sxtxr) of generalizability theory in three steps: G-studies for peer and peers-instructor ratings and D-study for peer ratings. According to the results of the G studies, the reliability coefficient obtained from the peer ratings and peer-instructor ratings were quite high (0.86 and 0.82 respectively). According to the result of the D study of peer ratings, just two peer raters are enough for getting high reliability coefficient. With the help of the gained results, it is suggested that peer assessment, which is effective on learning and decision making processes of students, should be used more often in education systems.

show abstract

Estimating Reliability Under a Generalizability Theory Model for Test Scores Composed of Testlets

Cited by 39 publications

References 15 publications

The Effects of Testlets on Reliability and Differential Item Functioning

The Effects of Testlets on Reliability and Differential Item Functioning

Estimating reliability of school-level scores using multilevel and generalizability theory models

Using generalizability theory to investigate the reliability of peer assessment

Contact Info

Product

Resources

About