Comparability of Bilingual Versions of Assessments: Sources of Incomparability of English and French Versions of Canada's National Achievement Tests

Ercikan, Kadriye; Gierl, Mark J.; McCreith, Tanya; Puhan, Gautam; Koh, Kim

doi:10.1207/s15324818ame1703_4

Cited by 49 publications

(31 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Think-aloud protocols were conducted using a total of 40 science items from the English and French versions of SAIP SOURCES OF DIFFERENTIAL ITEM FUNCTIONING 7 originally investigated by another study (Ercikan et al, 2004). The SAIP assessment had 144 science items in total.…”

Section: Methodsmentioning

confidence: 99%

“…The method that had been used in the original study for identifying DIF (Ercikan et al, 2004) is based on item response theory. For each group in a comparison, the difference between predicted and observed mean score is computed for each decile for each item.…”

Section: Think-aloud Protocolsmentioning

confidence: 99%

“…The experts were presented with the same set of written instructions that had been adopted and modified from another study (Ercikan et al, 2004); the instructions were printed on a 8" x 11" sheet of white paper in landscape format and 14 point Times Roman font. The experts were asked to read aloud both English and French versions and then rate each item according to a four-point scale: 0 = No difference between the two versions, 1 = Minimal differences between the two versions, 2 = Clear differences between the two versions, which are NOT expected to cause different performances of two groups of examinees, and 3 = Clear differences between the two versions, which ARE expected to cause different performances of two groups of examinees.…”

Section: Procedures -Think-aloud Protocolsmentioning

confidence: 99%

See 2 more Smart Citations

Investigating Linguistic Sources of Differential Item Functioning Using Expert Think-Aloud Protocols in Science Achievement Tests

Roth

Oliveri

Sandilands

et al. 2013

International Journal of Science Education

Self Cite

View full text Add to dashboard Cite

Even if national and international assessments are designed to be comparable, subsequent psychometric analyses often reveal differential item functioning (DIF). Central to achieving comparability is to examine the presence of DIF, and if DIF is found, to investigate its sources to ensure differentially functioning items do not lead to bias. In this study, sources of DIF were examined using think-aloud protocols. The think-aloud protocols of expert reviewers

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Think-aloud Protocolsmentioning

confidence: 99%

Section: Procedures -Think-aloud Protocolsmentioning

confidence: 99%

See 1 more Smart Citation

Investigating Linguistic Sources of Differential Item Functioning Using Expert Think-Aloud Protocols in Science Achievement Tests

Roth

Oliveri

Sandilands

et al. 2013

International Journal of Science Education

Self Cite

View full text Add to dashboard Cite

show abstract

“…A verbal or numerical rating scale could be developed for this purpose (cf. [4,10,36,48]). Such ratings can be a great help to the Project Centre, which cannot have expertise in all languages and is therefore largely dependent on the national reviews, when making the final judgmental decisions.…”

Section: Developing Judgmental Reviewsmentioning

confidence: 99%

“…In international education studies, an important potential source of DIF and bias are language-related differences (other sources including, e.g., cultural and curricular differences, often overlapping with language-related differences; for cultural and curricular differences, see e.g., [4][5][6]). Language-related differences refer to differences between languages and translation errors and can be roughly classified as in Table 1 (cf.…”

Section: Introductionmentioning

confidence: 99%

International Education Studies: Increasing Their Linguistic Comparability by Developing Judgmental Reviews

Arffman

2012

ISRN Education

View full text Add to dashboard Cite

In international education studies, the different-language test versions need to be equally difficult to read and answer for the test to be valid. To ensure comparability, several quality control procedures have been developed. Among these, surprisingly little attention has been paid to judgmental reviews and their ability to identify language-related sources of bias. Also, the reviews have often failed in identifying biases. This paper explored whether it is possible to improve the ability of judgmental reviews to identify language-related sources of bias. A new review was made of two Finnish items which in the PISA (Programme for International Student Assessment) 2000 reading test showed differential item functioning but for which no clear language-related explanations were found in the review in 2000. The items were compared systematically, at all linguistic levels, to the corresponding items in the English and French source versions, at the same time taking into account the cognitive processes required to answer them and students' written responses to them. Language-related explanations were found for both items which may have led to differences in performance, suggesting that it is possible to make judgmental reviews better able to identify language-related bias. Suggestions are given on how to do this.

show abstract

AssessingHawaiian

Wilson

2013

The Companion to Language Assessment

View full text Add to dashboard Cite

Over a period of 30 years, Hawaiian has moved from being a nearly extinct and forgotten language receiving almost no attention to being the single non‐English language most embedded as an official language of education and academic assessments in any of the 50 states within the USA. This change is highly related to the post‐World War II movement for equality for previously colonized peoples. It is also built upon inertia from government language use patterns and community identity factors in the small 19th‐century multiracial kingdom of Hawaii. Hawaii is the only one of the 50 states in the USA with two official languages recognized in its constitution. The gradual expansion of use of Hawaiian by the state of Hawaii has become a role model for other indigenous governments of the USA including American Indian tribes, Alaska Native villages, and Pacific Island territories. Assessments of Hawaiian language fluency and of academic knowledge through the Hawaiian language range from the private sector through to graduate school. While locally distinct historical factors have played a major role in the current attention to the use of Hawaiian, Hawaiian has also been weakened by shared American historical factors relative to restrictions of language diversity, and on indigenous languages in particular. Hawaiian was nearly exterminated by the elimination of public Hawaiian medium education in 1896, and a subsequent ban on its use as a medium of education in schools that only ended in 1986. Continuing into the present are perspectives on Hawaiian that undervalue high fluency in it compared to foreign languages and which run counter to other trends that place high value on Hawaiian for symbolic purposes.

show abstract

Comparability of Bilingual Versions of Assessments: Sources of Incomparability of English and French Versions of Canada's National Achievement Tests

Cited by 49 publications

References 14 publications

Investigating Linguistic Sources of Differential Item Functioning Using Expert Think-Aloud Protocols in Science Achievement Tests

Investigating Linguistic Sources of Differential Item Functioning Using Expert Think-Aloud Protocols in Science Achievement Tests

International Education Studies: Increasing Their Linguistic Comparability by Developing Judgmental Reviews

AssessingHawaiian

Contact Info

Product

Resources

About