Design of Experiment

A well-designed and constructed experiment will be robust under questioning, and will focus criticism on conclusions, rather than potential experimental errors. A sound experimental design should follow the established scientific protocols and generate good statistical data.

As an example, experiments on an industrial scale can cost millions of dollars. Repeating the experiment because it had poor control groups, or insufficient samples for a statistical analysis, is not an option. For this reason, the design phase is possibly the most crucial.

Design of Experiment Basics

With most true experiments, the researcher is trying to establish a causal relationship between variables, by manipulating an independent variable to assess the effect upon dependent variables.

In the simplest type of experiment, the researcher is trying to prove that if one event occurs, a certain outcome happens.

For example;

"If children eat fish, their IQ increases."

This is a good hypothesis and, at first glance, appears easily testable. The problem is that, in any solid experimental design, the opposite (contrapositive) should also be true. The design of experiment dictates that, if a certain event does not occur, the tested outcome will not happen, a subtle but crucial factor.

The reason for this is that it ensures that there is a genuine causal relationship between the independent and dependent variables.

Therefore, the following statement should also be true.

"If children do not eat fish, then their IQ will not increase."

The first statement is fairly easy to study, relying upon feeding children varying amounts of fish, and measuring their IQ.

However, it is much more difficult to test the second statement. The only way to test it properly is not to feed the children fish. It is impossible to use the same children, so a compromise must be reached, and the researcher must use two different groups of children.

The problem is that it is impossible to have two identical groups, and the Design of Experiment must take this into account. The researcher must understand that there are always going to be differences between the groups

This is why a solid experimental design should have extremely strong controls, and meticulous operationalization. Random groups are the best way of ensuring that the groups are as identical as possible.

In the fish example, all of the children could eat the same diet, but the tested group could be given extra fish supplements. Randomizing the groups tries to balance out the differences between individuals, and also removes any potential experimental bias.

Internal vs. External Validity

The second problem is that you have no idea whether other factors could influence the result.

Obviously, it is unethical to starve children, but other foods could have a significant influence upon IQ.

It is difficult to monitor what food the children are eating at home, leading to a potential confounding variable.

In addition, children from different schools may have a varying quality of teaching, potentially influencing the results.

These are just some of the factors potentially affecting the experiment, and any design of experiment must try to filter out the true results from the experimental 'noise'.

In an ideal 'True Experiment' situation, you would lock all of the children in a laboratory, subjecting them all to the same conditions. The researcher could then ensure that all variables are controlled, except for the independent variable, eating fish.

However, apart from being unethical, this places false restrictions upon the children. The researcher is trying to establish whether eating fish is beneficial to children's intelligence, so that they can advise parents and teachers about diet.

The real world is very different from the laboratory, and it would be dangerous to extrapolate the results from laboratory-based research to encompass all of the children in the world. The external validity would have been sacrificed for internal validity.

Design of Experiment, especially in the life sciences, usually involves finding the correct balance between internal and external validity, using judgment and experience.

Of course, complete perfection in an experiment is almost impossible, because time, resources and unknown factors will always play a significant role. The main point is that the experimental design should strive towards this goal.

The Design of Experiment is also influenced by the specific field of science. Physical sciences rarely have to consider ethics or random fluctuations; one lump of iron, for a chemistry experiment, is usually similar to another. Children, by contrast not only vary from each other but can rapidly change their behavior, in a few moments.

Physical Sciences vs. Life Sciences

Physics and chemistry, for example, are always going to facilitate more accurate designs than the life sciences. This is one of the reasons why there are two levels of significance; if p had to be < 0.01 (under 1% chance that the effect is due to coincidences), a biological experiment would never produce results.

To summarize, Design of Experiment is an ideal, a 'Gold Standard' towards which scientists should aspire, ensuring that any variations within an experiment are minimized.

With life and behavioral sciences, this is difficult to achieve, especially in artificial laboratory conditions, which may influence behavior and risk external validity. As long as a researcher justifies and assesses the effects of any deviation from the method, external and internal validity will not be compromised.

This difficulty is one of the reasons why behavioral sciences use quasi-experimental methods, and case studies, because Design of Experiment is all but impossible.