One RCT Can’t Rule Them All

Last week I attended a digital health networking event. A friend and colleague, CEO of a medical software startup, expressed to me his frustration with the lack of evidence based medicine. In his experience, event when faced with a randomized controlled trial (RCT), clinicians remain "stubborn" and set in their ways and don't accept new technologies.

This is not the first time I hear this complaint from digital health and medical device makers. "The evidence is right in front of them, but clinicians refuse to look at it".

Indeed, it is not always obvious.

• RCTs are considered the most reliable type of clinical trial, so why does a RCT not suffice?
• RCTs are good enough for regulators, so why are they insufficient for clinicians?

What is going on here? Aren't clinicians supposed to use evidence to make decisions?

Like the ancient story of the elephant being described by blind men who touch only one part of the elephant each, and the disagreement that ensues, medical evidence has many aspects, each with its pros and cons, and not one of them is sufficient to change practice.

The most often forgotten aspect of evidence based medicine is that current practice is also based on some evidence. It is almost always more reliable than a single RCT. If all you have is one RCT, it doesn't disprove current practice.

All evidence relies on many assumptions. Some are not documented, some don't reflect a realistic setting, and some are not even sensible. If all you have is a single RCT it does not mean it is right.

Most reproductions of RCTs (i.e. a different research team repeating the trial on a different set of patients and getting the same result) fail to reproduce the original results. There are many reasons why this happens. If all you have is one RCT, you haven't shown a reliable prediction of treatment outcomes.

Last but not least, is the role of conflict of interest. Multiple changes to RCT design in recent years are designed to mitigate the obvious effect of financial incentive on bias towards positive results. There is also an equal number of ways to game the system (see "Lies, Damn Lies & Statistics"). If all you have is a single RCT and it was conducted by the manufacturer, then regardless of design, statistical significance or anything else you may have done right, clinicians should, and will, be sceptical of your results.

Further reading: Levels of Evidence