Statistics and Reliability

In a number of experiments in engineering and physical sciences, reliability is high.

Example: High school experiments like finding out acceleration due to gravity by using a simple pendulum are easy to reproduce. In addition, the value is approximately the same whether it is done today or the next year and whether it is done in Norway or Nigeria. The factors that affect reliability are smaller concerns like elasticity of the string, humidity in the air, avoiding high amplitudes, etc.

Reliability is usually not so high because statistical studies can become hard to reproduce. There are a myriad of factors that affect experiments in social sciences. However, that cannot become an excuse for poor statistics.

Reliability vs Validity

The concept of validity is related but is different from statistical reliability. In statistics, reliability is all about being able to reproduce the same results whereas validity is all about coming as close to the true value as possible.

In our previous example, the closer the value is to 9.81m/s the better the reliability (there are small changes from place to place in this value). However, there can be a faulty experiment that might lead to a value of say 9.5m/s consistently. Then this experiment is high in reliability but low in validity. This can happen if, for example, a consistently high amplitude is given for the pendulum.

On the other hand, valid measurements must be reliable because we obtain the “right” answer every time we perform the experiment. However, statistically these concepts can be disparate and it is best to study them separately. Their mathematical treatment is also quite different and trying to combine them into one precision analysis can get complicated.

For precision, both statistical reliability and validity are important.

Reliability Analysis

There are many methods available to scientists to determine and improve the reliability of their experiment. For example, certain surveys might establish their reliability by asking the participants of the study the same or similar questions at two different times under similar conditions.

A reliability analysis can be done to establish that the experiment is reliable. This works only when the errors are uncorrelated. When errors are correlated, it might point to something deeper in the experiment which might be causing a problem. Such assumptions are important to note to fully appreciate the utility of statistical tools.