Significance Test (II)

This means, a significance test is used to determine if the difference between the assumed value in the null hypothesis and the value observed from experiment is big enough to reject the possibility that the result was a purely chance process.

A significance test is always accompanied by a value that specifies how much of this difference is enough to rule out a chance process and reject the null hypothesis. Common levels used in statistical analysis are 5% and 1%. This is called the significance level.

For example, suppose a researcher is studying the effect of breakfast in the morning to the performance in class of students. She will first select two random samples of students. One of these groups will have breakfast each day and the other will not, for a specified period of time.

At the end, she will compare the average grades of both these groups. Suppose that the group that had breakfast each day had an average grade of 4.32/5.00 and the group that did not have breakfast each day obtained an average grade of 4.05/5.00.

The question now is, can the researcher conclude that her research has shown that having breakfast in the morning increases the grades of students?

Concluding

There can be several reasons to explain the results of the experiment. It may happen that the students picked for the group that had breakfast each day simply were brighter students already. Or it may just be by chance that the group having breakfast each day just performed better than their counterparts on the examination days or were simply plain lucky.

For the above case, a significant test consists of determining whether the difference of 0.28/5.00 is good enough to conclude that students having breakfast indeed perform better than those that don’t.

The null hypothesis in this case is:

H₀: “Having breakfast in the morning has no effect on the grades of students”.

Depending on the distribution of students and their grades, the researcher must therefore determine whether the probability of obtaining this difference of 0.28/5.00 is significant low or high. She will then set a significance level (say 5%) for the case.

If the probability that the difference of 0.28/5.00 is lower than the significance level, then the significance test entails that the null hypothesis is rejected. This means that the experimenter can now conclude that having breakfast in the morning indeed has a positive effect on the grades of students.

On the other hand, if the difference of 0.28/5.00 comes out to be higher than the significance level, then using the significance test, the null hypothesis cannot be rejected. However, it is to be noted that in this case, we cannot conclude that the null hypothesis is true, i.e. we cannot accept the null hypothesis just because it is not rejected.