Repeatable, Reproducible [and Replictable]

There appears to be great confusion in the scientific and social sciences communities on the meaning of words related to certain aspects of the scientific method. The ArXiv paper by Lorena Barba “Terminologies for Reproducible Research” highlights the confused state that has appeared over the last 20 years. The words in question include:

Repeatability
Reproducible
Replication

I will dispense with replication simply because it’s too hard to say quickly (especially replicability) but see below for a more serious reason. The contention appears in the meaning of reproducibility or to reproduce. As Barba points out there are at least three ‘camps’ in this community which she labels, A, B1, and B2. The A camp makes no distinction, so we’ll forget about those. To describe B1 and B2 we must look at two extreme scenarios with respect to an experiment:

1. An experiment is carried out and is done again by the same author, using the same equipment, same methods, basically the same everything.

2. The experiment is carried out by a third-party using different equipment, different methods, etc. Basically, everything is different

In between these two extremes are variants, For example, the third-party could use the same methods but implement them independently of the original author, usually by reading the description given in the original published paper.

Given these descriptions, the B1 group calls the first scenario, ‘to reproduce the experiment’ while the second group, B2, calls the first scenario, ‘to replicate the experiment’ and there lies the contention.

Personally, I don’t like either of these terms as used here. As I mentioned before, replicability is a hard word to say. But not only that, from a dictionary perspective, it means the same thing as reproducibility. The Oxford English Dictionary describes replicability as “The quality of being able to be exactly copied or reproduced.” So why use two words, for two quite different things, where the two words have essentially the meaning?

My personal choice are the following two words, Rather than use the word replicability which seems redundant, I choose repeatability, hence:

Repeatability: means ‘to repeat the experiment again’, the word implies that the experiment was done exactly as before – Scenario 1

Reproducibility: means: ‘to recreate the experiment anew; reproduce implies creating a new thing, independently of the old – Scenario 2.

Of course one can get much more fine-grained, especially when it comes to computational experiments. But the fine graining can be included as levels within the class reproducibility.

Other than a change in wording from replicability to reproducibility, I appear to belong to camp B2. I should list others in camp B2, these include FASEB, NIST, 6 sigma, ACM, and Wikipedia and this Wikpedia page and The Physiome Project. I am sure there are others. For example, ACM writes:

Repeatability (Same team, same experimental setup)

The measurement can be obtained with stated precision by the same team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same location on multiple trials. For computational experiments, this means that a researcher can reliably repeat her own computation.

Reproducibility (Different team, different experimental setup)

The measurement can be obtained with stated precision by a different team, a different measuring system, in a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using artifacts which they develop completely independently.

Essentially the same definitions I gave above.

The National Academies recently studied this issue closely and is soon coming out with a report. It is apparently in favor of the more confusing option.

This entry was posted in General Science Interest, Publishing. Bookmark the permalink.

Leave a Reply

Your email address will not be published.