Significance tests play a key role in experiments: they allow researchers to determine whether their data supports or rejects the null hypothesis, and consequently whether they can accept their alternative hypothesis.
This article is a part of the guide:
Browse Full Outline
In everyday language, "significance" means that something is meaningful or important, but in statistical language, the definition is more precise. Furthermore, significance here does not imply theoretical, practical or research importance. A result can be statistically significant but a rather unimportant finding considering the bigger picture! A result is statistically significant if it satisfies certain statistical criteria.
The PValue and the Significance Level
Significance comes down to the relationship between two crucial quantities, the pvalue and the significance level (alpha). We can call a result statistically significant when P < alpha. Letâ€™s consider what each of these quantities represents.

pvalue: This is calculated after you obtain your results. It is the probability of observing an extreme effect even with the null hypothesis still being true. Importantly, it does not measure the size of an effect.

alpha: This is decided on before gathering data. It is the probability of the study rejecting the null hypothesis despite it being true (i.e. the chance of committing a Type 1 error). It is essential an error rate and usually set at or below 5%.
A Comfortable Confidence Level
Itâ€™s important to remember that there is nothing inherent about a 5% confidence level; it is merely a common convention. Where exactly the threshold is set is largely determined by the data in question and what the researchers are trying to achieve.
 Sciences where random error and natural variation are likely to play a part (for example investigative biology) will likely be content with alpha set to 5%.
 If you can expect a high level of precision and accuracy with the measurements and instruments employed, alpha can be set lower.
Pvalues are between 0 and 1. If P is less than the cutoff youâ€™ve prechosen, you should reject the null hypothesis in favor of the alternative. Alternatively, if P is greater than the cutoff, say 0.05, you should not reject the null.
A note about falsifiability: though you could be forgiven for thinking otherwise, any piece of research is technically setting out to prove or disprove the null hypothesis, and nothing more. The alternative hypothesis is correctly named â€“ it is only a position that is (provisionally) accepted as an alternative after the null hypothesis has been ruled out. All a significant result tells us is that there is â€œsomething going on" as opposed to nothing.
A Word of Caution
If you are studying statistics for a university course, the above may well be sufficient when it comes to writing up a term paper or understanding the general concepts behind statistical testing. However, the fact is that statistics is a complex and evolving science, and nowhere near the panacea that many students believe it to be.
Interestingly, the ASA (American Statistical Association) has published some guidelines about the proper use of the pvalue, which will be of interest to those publishing more serious research. Some of these recommendations are:
 pvalues do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
 Scientific conclusions and business or policy decisions should not be based only on whether a pvalue passes a specific threshold.
 A pvalue, or statistical significance, does not measure the size of an effect or the importance of a result.
 By itself, a pvalue does not provide a good measure of evidence regarding a model or hypothesis.
In other words, good research goes well beyond the simple yes/no mechanisms many students of statistics are first taught. A depth understanding of the limits of significance testing is beyond the scope of most studentsâ€™ curricula, however it does confirm the fact that research is seldom black and white!
Martyn Shuttleworth, Lyndsay T Wilson (Nov 26, 2008). Significance Test. Retrieved Nov 13, 2020 from Explorable.com: https://explorable.com/significancetest
You Are Allowed To Copy The Text
The text in this article is licensed under the Creative CommonsLicense Attribution 4.0 International (CC BY 4.0).
This means you're free to copy, share and adapt any parts (or all) of the text in the article, as long as you give appropriate credit and provide a link/reference to this page.
That is it. You don't need our permission to copy the article; just include a link/reference back to this page. You can use it freely (with some kind of link), and we're also okay with people reprinting in publications like books, blogs, newsletters, coursematerial, papers, wikipedia and presentations (with clear attribution).
Want to stay up to date? Follow us!
Footer bottom