Methodological aspects of prognostic factor studies: some caveats

Abstract

Methodological aspects of prognostic factor studies: some caveats To the Editor: In this comment some methodological aspects of prognostic factor studies are discussed. Categorization of continuous variables should be avoided and when necessary must be done before applying a test. Confidence intervals give more precise and clinically relevant information than p-values. Statistical tests providing a maximum of test power should be chosen. Repeated applications of the Cox model after bootstrap resampling permit an estimation of the stability of the model. Recently Souto and co-workers 1 created a new scoring system for patients with myelodysplastic syndromes.We would like to discuss some methodological items of this study. Continuous variables were categorized by trial and error until p-values were found close to 5%. It may be useful to divide a continuous variable into categories, but this is dangerous because categorizing means to a certain degree "throwing away information", or, if done by a datadependent approach, may increase substantially the type I error rate. A better approach would be the use of univariate Cox models for the continuous variables. 2 The authors stress the importance of the p-value, e.g. using the phrase " the most significant prognostic factor" P-values are only used to test whether the null hypothesis should be rejected, thus indicating whether there is any difference at all, but are never an estimation of the amount of the difference. A minimal difference may be statistically significant but clinically without any importance. Therefore confidence intervals of beta values of the prognostic factors would have provided more precise information than p-values. According to Sanz's work, 3 the survival curves of the variable hemoglobin were divided into three groups, but no significant difference could be found, whereas this was shown after dichotomization. We should not forget that Sanz et al. 3 analyzed a total of 370 patients, which gives much more statistical power for detecting real differences between the groups. Furthermore, when comparing the survival curves of ordered groups (here: hemoglobin), trend tests must be applied. Otherwise the statistical power would be reduced, such that real associations could be missed. The final Cox model depends on the variables initially offered, the rules determining the inclusion and LETTERS

Pitfalls in prognostic factor studies

“…The categorization of variables that are initially measured on a continuum is a frequently used procedure in surgical pathology, especially for the documentation of immunohistochemistry, as recent issues of this journal demonstrate 2–7 . Although this procedure may be useful for clinical decision making, or illustration purposes, it may negatively affect statistical evaluations 8–13 . Usually there is no standardization for the cutoff values of immunohistochemical data 7,13 .…”

Is There a Reliable Minimum Number of Lymph Nodes for T1 and T2 Colon Cancer?

“…We think that prognostic factor studies with surprising results should always be checked for pitfalls [2][3][4][5]. Dichotomizing a continuous or scaled variable causes several important problems.…”

Comments on the paper "c-erbB-2 expression and nuclear pleomorphism in canine mammary tumors"

“…This may create scientifically unrealistic models because this procedure discards potentially important information since all values between the cutoff levels have an equal effect on the model. Therefore, categorization considerably lowers the statistical test-power and can be seen as an introduction of measurement errors (8)(9)(10)(11)(12)(13)(14).…”