Mining clinical data

When you last visited a hospital or a family doctor you might have notices that everything they do is documented electronically: the creation of clinical data. Data being ones and zeroes in a computer means it is now easy to store it, retrieve it, search through it for patterns, and so on right? You might be right when dealing with a small number of patients. But when looking at thousands or millions of patients different arrangements of data make a big difference.

When a patient arrives at a consultation, the clinician needs the entire record for that patient. Because there are lots of patients and lots of records, the medical history of that patient, which may span years, needs to be stored in a way that makes it retrieved quickly. That means a particular arrangement of the data for quick storage and retrieval of many records as part of one history.

When looking for patterns in the data, such as what treatments work best for patients with a particular set of conditions, we don't need to look at the entire history of one patient at a time, but populations of patients with certain conditions. That means implies that data needs to be arranged in a way that is easy to retrieve groups of patients rather than individual histories.

The term "longitudinal study" hints at the difference in how we look at data for research vs. how we look at the same data for clinical care (i.e. lattitudinally). This kind of data transformation is perhaps unique to clinical data. This transformation of clinical data is part of Piano's AI-powered automation workflow. In addition for transforming data in this way, Piano performs all the other transformations needed to prepare the data for research, from anonymizing patients, to translation of text to standard concepts and harmonizing data from multiple sources.