Our rating scales are reliable tools for valid and reproducible assessment of the aging process in lower face areas.

Except for brow positioning, the upper face rating scales are reliable tools for valid and reproducible assessment of the aging process.

The 5-point photonumeric scale generated spans the severity of the type of crow's feet for which patients most commonly seek correction. The scale is well stratified for consistent rating.

The mid face scales are reliable tools for valid and reproducible assessment of age-related mid face changes.

The 5-point photonumeric scale generated spans the severity of marionette lines for which patients commonly seek correction. The scale is well stratified for consistent rating.

The 5-point photonumeric scale generated spans the severity of forehead lines for which patients commonly seek correction. The scale is well stratified for consistent rating.

The 5-point photonumeric scale generated spans the fullness of the upper and lower lip for which patients commonly seek correction. This scale is well stratified, with low intra- and interrater variability.

Many published scale validation studies determine inter‐rater reliability using the intra‐class correlation coefficient (ICC). However, the use of this statistic must consider its advantages, limitations, and applicability. This paper evaluates how interaction of subject distribution, sample size, and levels of rater disagreement affects ICC and provides an approach for obtaining relevant ICC estimates under suboptimal conditions. Simulation results suggest that for a fixed number of subjects, ICC from the convex distribution is smaller than ICC for the uniform distribution, which in turn is smaller than ICC for the concave distribution. The variance component estimates also show that the dissimilarity of ICC among distributions is attributed to the study design (ie, distribution of subjects) component of subject variability and not the scale quality component of rater error variability. The dependency of ICC on the distribution of subjects makes it difficult to compare results across reliability studies. Hence, it is proposed that reliability studies should be designed using a uniform distribution of subjects because of the standardization it provides for representing objective disagreement. In the absence of uniform distribution, a sampling method is proposed to reduce the non‐uniformity. In addition, as expected, high levels of disagreement result in low ICC, and when the type of distribution is fixed, any increase in the number of subjects beyond a moderately large specification such as n = 80 does not have a major impact on ICC.