Reproducibility

The basic principle is that, for any research program, an independent researcher should be able to replicate the experiment, under the same conditions, and achieve the same results.

This gives a good guide to whether there were any inherent flaws within the experiment and ensures that the researcher paid due diligence to the process of experimental design.

A replication study ensures that the researcher constructs a valid and reliable methodology and analysis.

Reproducibility vs. Repeatability

Reproducibility is different to repeatability, where the researchers repeat their experiment to test and verify their results.

Reproducibility is tested by a replication study, which must be completely independent and generate identical findings known as commensurate results. Ideally, the replication study should utilize slightly different instruments and approaches, to ensure that there was no equipment malfunction.

If a type of measuring device has a design flaw, then it is likely that this artefact will be apparent in all models.

The Process of Replicating Research

For most of the physical sciences, reproducibility is a simple process and it is easy to replicate methods and equipment.

An astronomer measuring the spectrum of a star notes down the instruments and methodology used, and an independent researcher should be able to achieve exactly the same results, Even in biochemistry, where naturally variable living organisms are used, good research shows remarkably little variation.

However, the social sciences, ecology and environmental science are a much more difficult case. Organisms can show a huge amount of variation, making it difficult to replicate research exactly and so reproducibility is a process of attempting to make the experiment as reproducible as possible, ensuring that the researcher can defend their position.

In addition, these sciences have to make much more use of statistics to dampen down experimental noise caused by physiological and psychological differences between the subjects.

This is one of the reasons why most social sciences accept a 95% probability level, which is a contrast to the 99% confidence required by most physical sciences.

Reproducibility and Generalization - A Cautious Approach

Observing due caution with the process of generalizing results helps to strengthen the case for experimental reproducibility.

In any study, there is a far smaller chance of finding confounding evidence if the claims are narrowly defined than if they are sweeping generalizations.

For example, a psychologist who found that aggression in children under the age of five increased if they watched violent TV, could generalize that all children under five would display the same condition.

Extending this to all children means that the experiment is prone to replication issues - A researcher finding that aggression did not increase in nine year old children would invalidate the entire premise by questioning the reproducibility.

Reproducibility is not Essential

It is important to understand that creating replicable research is not essential for validity, although it does help. Sometimes, due to sheer impracticality, temporal difficulties, or expense, this is not possible.

The Framingham Heart Study, an experiment testing three generations of nurses for cardiac issues, has been going on for over 60 years and nobody is seriously expected to replicate it.

Instead, results from other studies around the world are used to build up a database of statistical evidence supporting the findings.

Reproducibility - An Impossible Ideal?

Many scientists argue that reproducibility is not an important factor for many sciences observing natural phenomena, such as astronomy, geology and, notoriously, evolution.

The rise of the Intelligent Design movement has seen evolutionary science under attack, because creationists claim that evolution is not reproducible and, therefore, it is not valid. This has opened up an intense debate about the role of replication study as, for example, a geologist cannot very well recreate conditions found on the primordial earth and observe rocks metamorphosing.

However, creationists misunderstand the idea of reproducibility and assume that it applies to an entire theory. In fact, this is incorrect and it is a manipulation of scientific practices; replicating research only applies to a specific experiment or observation.

Reproducibility and Specificity - A Geological Example

If I go into the Greek Mountains and observe trilobite fossils lying above ammonite fossils, I assume that trilobites came later than ammonites.

However, a more talented geologist than I later travels to exactly the same place, and points out that the rocks there are deformed and twisted 180 degrees, so my observations were the wrong way around. My field study was reproducible in that another researcher could come and try to replicate my observations.

Looking at the process from the other angle, imagine that an astronomer discovers a planet circling around a distant star. Nobody is suggesting that he builds a gaseous cloud and waits a few billion years for matter to accrete and an identical solar system to form, because that would be absurd.

Performing a replication study would involve other astronomers observing the star to try to find the planets, showing that there really are planets and that the original astronomer had no equipment malfunction.

Reproducibility and Archaeology - The Absurdity of Creationism

When Arthur Evans discovered Knossos, on Crete, and proposed that there was an ancient, advanced Minoan civilization, nobody suggested that he should recreate such a civilization and see if they built an identical city. Absurd as it may seem, this is the type of assumption that proponents of Creationism make.

Looking at this process in reverse, if a team of builders builds an exact replica of Knossos, it does not prove that such a civilization existed, although it would be a useful exercise in looking at some of the techniques used by ancient builders, allowing archaeologists to refine their ideas. To suggest otherwise really is a deliberate misunderstanding and warping of the scientific method.

Ultimately, if Creationists use the argument that evolution is wrong because it is not reproducible, then they destroy their own argument. If evolutionary processes cannot be subjected to replicable research, neither can Intelligent Design, so their argument founders on its own presumptions. Surely, proponents of ID need to recreate the six days of Genesis before their ideas can be accepted by science!

Bibliography

Rosen, J. (2008). Symmetry Rules: How Science and Nature are Founded on Symmetry. Berlin, Germany: Springer-Verlag

Pickering, A. (1995). Beyond Constraint: The Temporality of Practice and the Historicity of Knowledge. In J.Z. Buchwald (Ed.). Scientific Practice: Theories and Stories of Doing Physics. (42-86). Chicago, University of Chicago Press