How to (not) run an experiment

Transcript

How you conduct an experiment influences the results you are getting from it and how much time, energy, effort, and money you have to spend to run the experiment itself.

There are three types of experiments.

The controlled experiment,
the natural experiment, and
the field experiment.

In a controlled experiment, you compare experimental samples against a control sample. These kinds of experiments are usually run inside a laboratory.

On the other side, natural experiments rely solely on observations of the variables of the system under study with no manipulation of those variables.

If those experiments are run in a natural setting, they are called field experiments.

The underlying principle of each experiment is the scientific method which consists of 6 steps.

In the first step, a question is defined.

If, for example, you are wondering which substrate you should use, you could state the question:

Does the substrate influence the mushroom yield?

You notice I did not ask, “Which substrate is the best?” While the last question is also valid, it
raises the question best compared to what?

Here, we could answer: In comparison to the mushroom yield.

Which brings us to the first question: Does the substrate influence the mushroom yield?

Video: How to (not) run an experiment

This question is more suitable for running an experiment because it contains a desired outcome – here the mushroom yield which we can measure.

In the second step, we start to research this topic. The more information we gather, the more we understand which factors influence the mushroom yield and which relationship between the substrate and the mushroom yield exist. Understanding the topic helps us formulate a hypothesis, a fancy word for a possible explanation of a phenomenon.

Formulating a hypothesis is the third step of the scientific method. In our example, such a hypothesis could be
“The C/N ratio of the substrate determines the mushroom yield.” An alternative would be that you divide the C/N ratio and formulate that “The carbon source and the nitrogen source itself determining the mushroom yield.”

We can now go to the drawing board to design the experiment with both hypotheses at hand.

For the first hypothesis, you would mix different substrates to match specific C/N ratios. For the second one, you would mix different carbon and nitrogen sources in different ratios to match specific C/N proportions.

In doing so, you end up with a list of substrate mixtures as the input and the mushroom yield as the output.

Figure 1: Input (subtsrate mixture) and outcomes (mushroom yield) of an experiment [1].

We then could go on to the fourth step and run the actual experiment.

Still, at this point, it would make sense to think a little bit more about other outputs of this experiment that would be valuable to us.

There would be

the height of the stem,
the thickness of the stem
the diameter of the cap
the texture of the fruitbody
the time for the spawn run
the time for the pinhead formation
the number of days to harvest
the time between to flushes
the number of flushes
the biological efficiency
the number of contaminated bags
and so on.

This list shows that while we only want in this experiment to test our hypothesis that the C/N ratio determines the mushroom yield, we could test other similar hypotheses in one go – meaning the influence of the C/N ratio on the other outcomes.

It is crucial that at this point that we write down how we measure these outcomes and if the needed tools are available to us.

Figure 2: Equipment needed to measure the outcomes [1].

There is another aspect I want to highlight when it comes to designing an experiment.

Assume we have two factors – humidity and temperature – Do we keep one fix and vary the other, or do we run both simultaneously? Both factors will influence the mushroom yield.

Therefore, we can draw one factor on the horizontal and the other on the vertical.

This graph gives us all the mushroom yields possible.

It is like a contour map with hills and values to explore.

Figure 3: Example of a contour map of an experiment [1].

If we keep one factor – the humidity fix at, for example, 80 %– and vary the temperature, we only move along this line.
By looking at this line, we will find the highest mushroom yield at 23°C.

If we switch to keep the temperature fixed at 23°C, we move onto this line. Here, we find at 80 % humidity the highest mushroom yield.

But is it really the highest?

Think for a second.

The map represents all mushroom yield possible.

The point we just found could be the highest, but there might be even higher ones. To find them, we have to vary both parameters simultaneously. Only then will we get an idea of the whole map.

As the following examples illustrate, we could vary one factor at a time, but this will take a lot of time, energy, effort, and money. Assume we want to vary the temperature between 10°C and 30°C and the humidity from 50% to 90% both in 5 increments.

Figure 4: No. of trials needed if variable are varied separately [1].

In this case, we have to run 45 trials. More importantly, we only get one results from each of the trials.
We, therefore, have to re-run each of them at least 3x – leading to 135 trials .

A better approach would be to vary both factors at the same time. In doing so, we first focus on the extreme values.
Remember, the idea is to get an understanding of the map. The minimum number of trials we have to run is 5 – with 3 replicas – we end up with 15.

Figure 5: No. of trials needed if variables are varied simultaneously – 1st iteration [1].

That is only 11 % of the first way.

If we have a first impression, we can add more trials one step at a time. Each time we learn more and more about the relationship between humidity and temperature and their influence on the mushroom yield.