Type I error in multiple comparison tests in analysis of variance

Abstract

In a hypothesis test, a researcher initially fixes a type I error rate, that is, the probability of rejecting the null hypothesis given that it is true. In the case of means tests, it is important to present a type I error that is equal to the nominal pre-fixed level, such that this error remains unchanged across various scenarios, including the number of treatments, number of repetitions, and coefficient of variation. The purpose of this study is to analyse and compare the following multiple comparison tests for the control of both conditional and unconditional type I error rates, depending on a significant F-test in the analysis of variance: Tukey, Duncan, Fisher’s least significant difference, Student–Newman–Keuls (SNK), and Scheffé. As an application, we present a motivation study and develop a simulation study using the Monte Carlo method for a total of 64 scenarios. In each simulated scenario, we estimate the comparison-wise and experiment-wise error rates, conditional and unconditional on a significant result of the overall F-test of analysis of variance for each of the five multiple comparison tests evaluated. The results indicate that the application of the means tests based only on the significance of the F-test should be considered when determining the error rates, as this can change them. In addition, we find that Fisher’s test controls for the comparison-wise error rate, the Tukey and SNK tests control for the experiment-wise error rate, and the Duncan and Fisher tests control for the conditional experiment-wise error rate. Scheffé’s test does not control for any of the error rates considered.

Effects of Combined Application of Biological Agent and Fertilizer on Fungal Community Structure in Rhizosphere Soil of Panax notoginseng

“…The soil's physical and chemical properties were processed by Microsoft Excel 2016 and SPSS 22 to use in the statistical analysis via the Duncan method [44]. The vegan package in R (Version 3.5.1) was used to draw heat maps and the composition of fungal communities for principal coordinate analysis (PCoA).…”

The dHOTLearn Model to Improve Critical Thinking Skills of Physics Education Program Undergraduate

“…Further study is performed for assessing the resulting GOAL model's quality in terms of the dHOTLearn model's validity and reliability study using single measures interrater coefficient correlation (ICC) and Cronbach's coefficient alpha. ICC was developed based on analysis of variance but in certain cases the results are similar to the alpha coefficient (Rodrigues et al, 2023). Cronbach's alpha calculates the lowest possible limit of a construct's reliability value (Lubiano et al, 2022;Zakariya, 2022).…”