Scientific communication relies on evidence that cannot be entirely included in publications, but the rise of computational science has added a new layer of inaccessibility. Although it is now accepted that data should be made available on request, the current regulations regarding the availability of software are inconsistent. We argue that, with some exceptions, anything less than the release of source programs is intolerable for results that depend on computation. The vagaries of hardware, software and natural language will always ensure that exact reproducibility remains uncertain, but withholding code increases the chances that efforts to reproduce results will fail.T he rise of computational science has led to unprecedented opportunities for scientific advance. Ever more powerful computers enable theories to be investigated that were thought almost intractable a decade ago, robust hardware technologies allow data collection in the most inhospitable environments, more data are collected, and an increasingly rich set of software tools are now available with which to analyse computer-generated data.However, there is the difficulty of reproducibility, by which we mean the reproduction of a scientific paper's central finding, rather than exact replication of each specific numerical result down to several decimal places. We examine the problem of reproducibility (for an early attempt at solving it, see ref. 1) in the context of openly available computer programs, or code. Our view is that we have reached the point that, with some exceptions, anything less than release of actual source code is an indefensible approach for any scientific results that depend on computation, because not releasing such code raises needless, and needlessly confusing, roadblocks to reproducibility.At present, debate rages on the need to release computer programs associated with scientific experiments 2-4 , with policies still ranging from mandatory total release to the release only of natural language descriptions, that is, written descriptions of computer program algorithms. Some journals have already changed their policies on computer program openness; Science, for example, now includes code in the list of items that should be supplied by an author 5 . Other journals promoting code availability include Geoscientific Model Development, which is devoted, at least in part, to model description and code publication, and Biostatistics, which has appointed an editor to assess the reproducibility of the software and data associated with an article 6 .In contrast, less stringent policies are exemplified by statements such as 7 ''Nature does not require authors to make code available, but we do expect a description detailed enough to allow others to write their own code to do similar analysis.'' Although Nature's broader policy states that ''...authors are required to make materials, data and associated protocols promptly available to readers...'', and editors and referees are fully empowered to demand and evaluate any specific code, we belie...

We designed a system to enable the signature of an air gun array to be calculated at any point in the water from a number of simultaneous independent measurements of the near‐field pressure field [subject of a patent application]. The number of these measurements must not be less than the number of guns in the array. The underlying assumption in our method is that the oscillating bubble produced by an air gun is small compared with the wavelengths of seismic interest. Each bubble thus behaves as a point source, both in the generation of seismic waves and in its response to incident seismic radiation produced by other nearby bubbles. It follows that the interaction effects between the bubbles may be described in terms of spherical waves. The array of interacting guns is equivalent to a notional array of noninteracting guns whose combined seismic radiation is identical. The seismic signatures of the equivalent independent elements of this notional array can be determined from the near‐field measurements. The seismic radiation pattern emitted by the whole array can be computed from these signatures by linear superposition, with a spherical correction applied. The method is tested by comparing far‐field signatures computed in this way with field measurements made in deep water. The computed and measured signatures match each other very closely. By comparison, signatures computed neglecting this interaction are a poor match to the measurements.

No abstract

Conventional wisdom, that smaller components contain relatively fewer faults, may be wrong. This author found that medium-sized components were proportionately more reliable than small or large ones. Moreover, he says, there may be limits on the fault density we can achieve.

This paper covers two very large experiments carried out concurrently between 1990 and 1994, together known as the T-experiments. Experiment T1 had the objective of measuring the consistency of several million lines of scientific software written in C and Fortran 77 by static deep-flow analysis across many different industries and application areas, and experiment T2 had the objective of measuring the level of dynamic disagreement between independent implementations of the same algorithms acting on the same input data with the same parameters in just one of these industrial application areas. Experiment T1 showed that C and Fortran are riddled with statically detectable inconsistencies independent of the application area. For example, interface inconsistencies occur at the rate of one in every 7 interfaces on average in Fortran, and one in every 37 interfaces in C. They also show that Fortran components are typically 2.5 times bigger than C components, and that roughly 30% of the Fortran population and 10% of the C population would be deemed untestable by any standards.Experiment T2 was even more disturbing. Whereas scientists like to think that their results are accurate to the precision of the arithmetic used, in this study, the degree of agreement gradually degenerated from 6 significant figures to 1 significant figure during the computation.The reasons for this disagreement are laid squarely at the door of software failure, as other possible causes are considered and rejected.Version 2nd Apr 97, for CS & E Taken with other evidence, these two experiments suggest that the results of scientific calculations involving significant amounts of software should be treated with the same measure of disbelief as an unconfirmed physical experiment.