Our understanding of physical systems generally depends on our ability to match complex computational modelling with measured experimental outcomes. However, simulations with large parameter spaces suffer from inverse problem instabilities, where similar simulated outputs can map back to very different sets of input parameters. While of fundamental importance, such instabilities are seldom resolved due to the intractably large number of simulations required to comprehensively explore parameter space. Here we show how Bayesian machine learning can be used to address inverse problem instabilities, and apply it to two popular experimental diagnostics in plasma physics. We find that the extraction of information from measurements simply on the basis of agreement with simulations is unreliable, and leads to a significant underestimation of uncertainties. We describe how to statistically quantify the effect of unstable inverse models, and describe an approach to experimental design that mitigates its impact.