Pilot studies are often recommended by scholars and consultants to address a variety of issues, including preliminary scale or instrument development. Specific concerns such as item difficulty, item discrimination, internal consistency, response rates, and parameter estimation in general are all relevant. Unfortunately, there is little discussion in the extant literature of how to determine appropriate sample sizes for these types of pilot studies. This article investigates the choice of sample size for pilot studies from a perspective particularly related to instrument development. Specific recommendations are made for researchers regarding how many participants they should use in a pilot study for initial scale development.

No abstract

Adequate sample sizes for omnibus ANOVA tests do not necessarily provide sufficient statistical power for post hoc multiple comparisons typically performed following a significant omnibus F test. Results reported support a comparison-of-most-interest approach for sample size determination in ANOVA based on effect sizes for multiple comparisons.

A Monte Carlo simulation study was conducted to examine outliers' influence on Type I error rates in ANOVA and Welch tests, and the effectiveness of two outlier accommodation methods: nonparametric rank based method and Winsorizing. Recommendations are given regarding outlier handling with different sample sizes and number of outliers.

In 2009, DeMars stated that when impact exists there will be Type I error inflation, especially with larger sample sizes and larger discrimination parameters for items. One purpose of this study is to present the patterns of Type I error rates using Mantel–Haenszel (MH) and logistic regression (LR) procedures when the mean ability between the focal and reference groups varies from zero to one standard deviation. The findings can be used as guides for alpha adjustment when using MH or LR methods when impact exists. A second purpose is to better understand the conditions that cause Type I error rates to inflate. The results indicate that inflation can be controlled even in the presence of large ability differences and with large samples.

No abstract

Courses in introductory educational measurement are often hampered by the lack of computer programs by which to analyze test data. To be sure, computer software exists (e.g., SPSS, SAS, Iteman) that performs these analyses; however, these programs come at a high cost and are not designed for instructional use. As a result, many practicing teachers who take the introductory educational measurement course learn about reliability and item analysis but are not able to continue to use these skills after the course ends. The Test Analysis Program (TAP), written in Borland Delphi Professional Version 6.0, performs classical test and item analyses under Windows 9x/NT/XP. In addition to performing test analyses, the TAP software includes certain features that will assist instructors of educational measurement in the classroom.

Developing psychometrically sound instruments can be difficult, especially if little is known about the constructs of interest. When constructs of interest are unclear, a mixed methods approach can be useful. Qualitative inquiry can be used to explore a construct's meaning in a way that informs item writing and allows the strengths of one analysis method to compensate for the weaknesses of the other. Mixing method applications can be complex, however, there are few examples within the literature pertaining to the mix of interviews, Rasch modeling, and classical test theory. This article demonstrates how to mix qualitative inquiry with Rasch modeling (and classical test theory) in order to develop an instrument that measures a complex construct: patient trust.