The type of study we are most likely to recognise is the experimental study. Theses studies are designed to justify interventions. As such, they are in great demand by policy makers. How can we prove that this expensive simulator is preferable to a plastic cup and a drinking straw? The question is not just an academic one. Matsumoto et al (2002) demonstrated that the use of a surgical drain (urethra), an inverted plastic cup (bladder), and two straws (ureters) placed in a plastic case is just as effective in teaching novice urologists ureteroscopic skills as very expensive, high fidelity simulators. I suspect this study is being extensively critiqued as urologists around the world struggle to regain their educational funding for simulator models.
If you need a reminder of how an RCT really works, check this out
This was a study done during a talk to the public by Amanda Burls and Amy Price, entitled “Education by chocolate”. I bet it worked!! Their protocol is a great demonstration of RCT design. Find it at http://www.ithinkwell.org/education-by-chocolate-a-randomized-controlled-trial-by-thinkwell-2/
The two questions we ask in experimental studies are:
“Does it work?”
“Is this intervention better or more efficient than the other one?”
There are several difficulties with running controlled trials in education, though
It’s hard to standardise interventions (what if the presenter changes? Does ANY presenter present things the same way twice??)
- The study group is never homogenous (We often don’t teach the same way twice because our learners are different)
- The control group is never completely matched to the study group
- Measuring outcome is tough. (When do you do it? HOW do you do it? Do you measure knowledge, skills, attitudes, motivation etc etc)
- Pre-tests influence learning (they increase motivation to learn and stimulate recall of prior knowledge. In some learners, they can induce panic and cause them to look things up)
- How many participants do you need to involve to make a conclusion??
- If you do manage to find a homogenous sample of motivated learners in controlled conditions, is that EVER generalizable to real, messy learning environments???
All this being said, randomised controlled trials of one sort or another are still seen as the gold standard for research. And if we are trying to justify an expensive simulator, a good gold standard RCT can translate into educational gold.
Matsumoto ED, Hamstra SJ, Radomski SB, Cusimano MD: The effect of bench model ﬁdelity on endourological skills: a randomized, controlled study. J Urol 2002, 167:1243–1247.