Practical Strategies for Dealing with Unreliability in Competency Assessments

The Effects of Initially Misclassified Data on the Effectiveness of Discriminant Function Analysis and Finite Mixture Modeling

“…Very few instruments can reliably distinguish between cases with adjacent scores (Dwyer, 1996). Thus in these contexts cases nearer to the cut-point are more likely to be misclassified than cases lying further away (Lathrop, 1986;Dwyer, 1996). To serve as an analog to these situations, study 2 simulated data such that cases with a low probability of belonging to their parent distribution were more likely to be misclassified than cases with a high probability of belonging to their parent distribution.…”

Supervised classification in the presence of misclassified training data: a Monte Carlo simulation study in the three group case

“…In all of these instances, classifications are frequently made using cut-points on a continuous variable (e.g., achievement test score, intelligence test, anxiety inventory, college entrance examination). However, it is well known that using such cut-point methods for this purpose is likely to result in an initial misclassification of group membership (Lathrop, 1986 ; Dwyer, 1996 ). Thus, if this initial grouping is to be used for creating a prediction algorithm for accurately classifying future individuals, such training group misclassification can be particularly problematic.…”

“…In order to generate the data, true group membership was first assigned to each simulated subject. Then, cases nearer to the predetermined cut-points on the predictor variable were simulated as more likely to be misclassified than cases lying further away (Lathrop, 1986 ; Dwyer, 1996 ). In other words, cases with a relatively low probability of belonging to their initially assigned group were more likely to be misclassified than were those with a higher probability of their initial group membership.…”

“…From a statistical perspective, this occurs when the probability of a case being misclassified depends on that case's relative position in the distribution of the variable used to classify. For example, when an artificial cut-point is used with a measure in order to create groups using a continuous variable, individuals with scores closer to the cut-point are more likely to be misclassified whereas individuals with values further from the cut-point are less likely to be misclassified (Lathrop, 1986 ; Dwyer, 1996 ). Thus, if a specific score on a test is used to identify students at risk for academic problems, some individuals will very likely be misclassified (DiStefano and Morgan, 2011 ), with those having scores just above or just below the cut value most likely to be so.…”