Why do we need so much evidence?

Evidnce based medicine, as the name implies, means that medical decisions need to be based on the best evidence we have about what is safe and effective to prescribe and do. What the name doesn't imply is the vast number of people who work behind the scenes to produce evidence for practitioners. While most people are familiar with clinical trials and that pharma and device makers spend millions of dollars on each one and produce enough evidence for a regulator to decide whether or not to approve the trialed product. But that's not where the evidence chain ends, this is where it begins.

Testing a new product to get it approved for clinical use starts with clinical trials that show the produce is not toxic (Phase I), effective (Phase II) and reliable (Phase III). If it passes all these tests then it usually gets approved for wide use. But this is not enough. Clinical trials are very important but they still suffer from:

• Only testing it when used by staff that are specifically trained on the product, but most staff would not have access to such training,
• Only testing it on carefully selected patients, that have no underlying or chronic conditions, and
• Only testing it on relatively small populations compared the number of patients who will end up using it. The number of people enrolled in a trial is the smallest number of people needed to show the product or procedure works, not necessarily to show that it is safe.

To see how the product or procedure works in the general population it needs to monitored for all patients who received it, to decide if it is as effective as was first thought and that it is safe. This is called post-market surveillance or sometimes Phase IV clinical trials and some drugs and devices have been de-regulated through this process but most just add warnings or are restricted for less volunerable patients.

The reason for this is simple: in a clinical trial (almost all of which are done on less than 2,000 patients) cannot find a side effect that occurs for one patient in 10,000, which is the standard for reporting side effects. Hundreds of thousands of patients are often needed for that and most innovations would be unaffordable if they needed to enroll that many patients in a clinical trial for approval.

The number of post market surveillance studies that are not done is hard to gauge but in interviews we conducted with many senior specialists, hospital admintrators and clinical researchers, we estimate that number to be about 100 times more than are actually done.

The only way to fill such a 10,000% demand for post market surveillance studies is using standardisation of data, research automation and fit-for-purpose analytic tools that are usable by doctors and nurses, not just business analysts and quants.