This study explores the utility of analytic scoring for the TOEFL® Academic Speaking Test (TAST) in providing useful and reliable diagnostic information in three aspects of candidates' performance: delivery, language use, and topic development.G studies were used to investigate the dependability of the analytic scores, the distinctness of the analytic dimensions, and the variability of analytic score profiles. Raters' perceptions of dimension separability were also obtained.Based on the phi coefficients and standard errors of measurement (SEMs), the dependability of analytic scores averaged across six tasks and double ratings was acceptable for both operational and practice settings. However, scores averaged across two tasks and double ratings were not reliable enough for operational use.Correlations among the analytic scores by task were high, but those between delivery and topic development were lower, and these results were corroborated by raters' perceptions. When averaged across tasks or task types (two or more tasks), correlations among the analytic scores were very high, and the profiles of scores were flat.The utility of analytic scoring is discussed, and both score dependability and whether analytic scores provide diagnostic information beyond that provided by holistic scores are considered.Key words: TOEFL iBT speaking, analytic scoring, score dependability, dimension distinctness, score profile, G theory i ii ETS administers the TOEFL program under the general direction of a policy board that was established by, and is affiliated with, the sponsoring organizations. Members of the TOEFL Board (previously the Policy Council) represent the College Board, the GRE Board, and such institutions and agencies as graduate schools of business, junior and community colleges, nonprofit educational exchange agencies, and agencies of the United States government.Since its inception in 1963, the TOEFL has evolved from a paper-based test to a computer-based test and, in 2005, to an Internet-based test, TOEFL iBT. One constant throughout this evolution has been a continuing program of research related to the TOEFL test. From 1977 to 2005, nearly 100 research and technical reports on the early versions of TOEFL were published. In 1997, a monograph series that laid the groundwork for the development of TOEFL iBT was launched. With the release of TOEFL iBT, a TOEFL iBT report series has been introduced.Currently this research is carried out in consultation with the TOEFL Committee of Examiners. Its members include representatives of the TOEFL Board and distinguished English as a second language specialists from the academic community. The Committee advises the TOEFL program about research needs and, through the research subcommittee, solicits, reviews, and approves proposals for funding and reports for publication. Members of the Committee of Examiners serve four-year terms at the invitation of the Board; the chair of the committee serves on the Board. Current (2005Current ( -2006 members of the TOEFL Committee of...

This study investigated the scoring of the Test of English as a Foreign Language TM Internet-based Test (TOEFL iBT™) Speaking section by bilingual or multilingual speakers of English and 1 or more Indian languages. We explored the extent to which raters from India, after being trained and certified, were able to score the Speaking section for TOEFL iBT examinees with mixed first language (L1) backgrounds, especially those speaking an Indian language, accurately and consistently. The effectiveness of a special training package designed for scoring Indian examinees was examined as well. A total of 26 trained and certified raters from India were randomly divided into 2 groups and participated in 2 on-site scoring sessions in Mumbai. In the first session, both groups received regular training for scoring the TOEFL iBT Speaking section, which was largely similar to that received by raters in North America. In the second scoring session, 1 group continued to receive the regular training while the second group was trained using a special training package. Rater feedback surveys were also given to the raters. It was found that with training similar to that which operational U.S.-based raters receive, the raters from India performed as well as the operational raters in scoring both Indian and non-Indian examinees. In addition, the special training helped the raters score Indian examinees more consistently, leading to increased score reliability estimates. It also boosted raters' levels of confidence in scoring Indian examinees.Key words: TOEFL iBT, speaking, speech scoring, rater background characteristics, non-native raters, rater biasii To obtain more information about the TOEFL programs and services, use one of the following:

We investigated the scoring of the Speaking section of the Test of English as a Foreign Language TM Internet-based (TOEFL iBT R ) test by speakers of English and one or more Indian languages. We explored the extent to which raters from India, after being trained and certified, were able to score the TOEFL examinees with mixed first languages accurately and consistently. The effectiveness of a special training package designed for scoring Indian examinees was examined as well. We found that the raters from India performed as well as the operational U.S.-based raters in scoring both Indian and nonIndian examinees. The special training also helped the raters score Indian examinees more consistently and confidently.