The objectives of this study were to assess the reliability and accuracy of visual methods used to quantify the severity of bacterial spot (Xanthomonas arboricola pv. pruni) symptoms and evaluate the effects of rater experience on the quality of disease estimates. Three cohorts of raters differing in experience with disease assessment rated three sets of peach or nectarine leaves (n ‡ 103; disease severity levels from 0% to 100%) by direct estimation of percentage leaf area with symptoms. Four of the experienced raters also rated the leaves using a 1-7 interval scale. Actual disease severity on the leaves was obtained with the APS ASSESS image analysis software. Equivalence tests based on a bootstrap analysis were used to compare the rating scale and direct estimation methods, and to evaluate the effects of rater experience, computer training and human instruction on accuracy and reliability of disease estimates. In concordance analysis of continuous variables, with data from scale converted to percentage, the direct estimation method resulted in more accurate and reliable estimates than the interval scale. Analysing the scale data without conversion to percentage improved the concordance statistics for the scale but not sufficiently to match the direct estimation method. Accuracy was affected more by rater experience and intrinsic ability than reliability. Instruction on disease symptoms resulted in the largest improvement in estimates from inexperienced raters. Accurate and reliable direct estimation of bacterial spot severity on peach and nectarine can be made by raters with varying levels of experience provided they receive sufficient instruction.