Experimental studies: justifying medical education interventions

The type of study we are most likely to recognise is the experimental study. Theses studies are designed to justify interventions. As such, they are in great demand by policy makers. How can we prove that this expensive simulator is preferable to a plastic cup and a drinking straw?  The question is not just an academic one. Matsumoto et al (2002) demonstrated that the use of a surgical drain (urethra), an inverted plastic cup (bladder), and two straws (ureters) placed in a plastic case is just as effective in teaching novice urologists ureteroscopic skills as very expensive, high fidelity simulators. I suspect this study is being extensively critiqued as urologists around the world struggle to regain their educational funding for simulator models.

If you need a reminder of how an RCT really works, check this out

This was a study done during a talk to the public by Amanda Burls and Amy Price, entitled “Education by chocolate”. I bet it worked!! Their protocol is a great demonstration of RCT design. Find it at http://www.ithinkwell.org/education-by-chocolate-a-randomized-controlled-trial-by-thinkwell-2/

The two questions we ask in experimental studies are:

“Does it work?”

“Is this intervention better or more efficient than the other one?”

There are several difficulties with running controlled trials in education, though

  • It’s hard to standardise interventions (what if the presenter changes? Does ANY presenter present things the same way twice??)
  • The study group is never homogenous (We often don’t teach the same way twice because our learners are different)
  • The control group is never completely matched to the study group
  • Measuring outcome is tough. (When do you do it? HOW do you do it? Do you measure knowledge, skills, attitudes, motivation etc etc)
  • Pre-tests influence learning (they increase motivation to learn and stimulate recall of prior knowledge. In some learners, they can induce panic and cause them to look things up)
  • How many participants do you need to involve to make a conclusion??

And finally

  • If you do manage to find a homogenous sample of motivated learners in controlled conditions, is that EVER generalizable to real, messy learning environments???

All this being said, randomised controlled trials of one sort or another are still seen as the gold standard for research. And if we are trying to justify an expensive simulator, a good gold standard RCT can translate into educational gold.

Matsumoto ED, Hamstra SJ, Radomski SB, Cusimano MD: The effect of bench model fidelity on endourological skills: a randomized, controlled study. J Urol 2002, 167:1243–1247.


One thought on “Experimental studies: justifying medical education interventions

  1. Thanks for using our trial. The difficulties you state are relevant and apply just as easily to other trials, thinking the design through is critical, It is great that you are working through this with discussion that lead people to plan and think together!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s