Annual Research Review: Threats to the validity of child psychiatry and psychology

Abstract

The two major threats to validity concern the inability to examine brain tissues in life and the evidence that there is a high overlap among disorders. We emphasize the need to follow published guidelines on preplanned analyses and we note the dangers associated with unregulated flexibility in data analysis. We note the very important clinical and developmental findings that have been ignored, perhaps partly because of an excessive focus on technologies. Nevertheless, we are positive about both the accomplishments and the ways in which challenges are being met.

Pharmacoepidemiology: Using randomised control trials and observational studies in clinical decision‐making

“…It may also be the case that multi‐arm trials recruit more effectively than 2‐arm trials, possibly since the multiple arms, with different inclusion criteria, mean more patients are eligible, and well‐designed multi‐arm studies may provide significant patient benefit compared to multiple 2‐armed trials Natural experiments Alternatives to RCTs that utilise naturally occurring circumstances to separate variables that usually associate together in a before and after cohort study Open‐label RCT An RCT where allocation concealment is undertaken but the study is not blinded and may increase the risk of observer bias.…”

“…18,62 However, systematic reviews have shown that reported sub-group analyses are seldom prespecified and there is a recognition that uncontrolled flexibility in the analysis of data carries a real risk of false positive findings. [62][63][64][65] If multiple assumptions are tested in subgroup analysis the likelihood of a falsely significant result by chance alone increases. 66 Although unscheduled sub-group analysis, labelled as such, may have a role in hypothesis-generation for subsequent trials, statistically inferential approaches to sub-group analysis should be limited to small numbers of pre-specified sub-groups underpinned by sound biological evidence to limit the reporting of false positive effects.…”

Editorial: Ingenious designs and causal inference in child psychology and psychiatry

“…Study of moderated mediation is one of a range of recent developments in design and analytic methods that have addressed threats to causal inference, particularly in the context of intervention trials (Rutter & Pickles, 2016). A useful method, borrowed from econometrics, involves identification of 'instrumental variables' (Dunn et al, 2015).…”

Editorial: Science unskewed – acknowledging and reducing ‘risk of bias’ in parenting research

“… There is a growing awareness that ‘risk of bias’ distorts the process of generating and interpreting evidence and threatens the validity of psychological and psychiatric research at a number of different levels (Rutter & Pickles, ). Such threats are likely to be greatest in fields of study focused on socially/politically contentious issues where beliefs and values are strongly held – especially where there is a lack of methodological rigour. …”