Applying Item Responses Theory for Measuring Student’s Ability in Academic Speaking

Abstract

Purpose of the study: Every human being is required to be skilled at communicating, skilled at expressing thoughts, ideas, and feelings. This research aims to explain student’s ability in academic speaking based on the framework of item responses theory.
 Methodology: This research uses a mix method research approach; qualitative and quantitative approaches are used together to answer the formulated problems. A qualitative approach is used for digging up information about the needs to develop the speaking test. The try out subjects was 25 university student taking Bahasa Indonesia subject; while 125 students were used as the measurement subject. The data were students’ responses to the speaking test which was scored by two people (rater). The reliability of the instrument was estimated by the Generalizability theory. Rasch model analysis was used to estimate the item parameter; while the Maximum Likelihood was used to estimate the students’ ability.
 Main Findings: The value of 𝜎 ̂(𝑜) s influenced by the similarity to the average mean score observed in academic speaking. The value of 𝜎 ̂(𝑝) and 𝜎 ̂(𝑜) suggests that the distribution of variability in person and item is the same and high. A sufficiently large value of 𝜎 ̂(𝑝i) implies the fact that the value involves all residual sources for a variance. Balance alternatives, weigh consequently and decide rationally.
 Applications of this study: Based on the design proposed for the Indonesian Language Proficiency assessment, other generalizability design (G-Design) used is a cross design because each student (p) becomes the object of observation of two observers (r) who both assess four aspects of observation/indicator. It is used to determine the reliability (i.e., reproducibility) of measurements under specific conditions in academic speaking.
 Novelty/Originality of this study: Some rater qualifications that must be met include the process of gathering and using the appropriateness of background information before assessing. Measuring the ability of students in academic speaking by applying G theory and conducting an IRT analysis approach needs.