The health record was invented to document each patient’s journey through the medical ecosystem. It weaves together the perspectives and verdicts of practitioners across different specialties who deliver care for a patient over the course of their life.

The clinical notes that constitute a major portion of the health record are intended for each physician to communicate the “state of the patient” to other physicians who may encounter that patient. The language used by physicians in these clinical notes is often a mix of specialized terminologies and common (“real-world”) phraseology. The entirety of clinical notes over a patient’s life span articulates the shifting state of their health — a longitudinal narrative of the patient’s family history, suspected conditions, confirmed diagnoses, anticipated prognosis, prescribed intervention, possible or observed side effects, regimen modification and various quality of life assessments. With the rapid adoption of electronic health records (EHRs), this collective clinical wisdom of physicians has been digitized into billions of context-rich clinical notes.

This digitization of health records presents an unprecedented opportunity for their systematic synthesis across tens of millions of patients over a privacy-preserving and HIPAA-compliant platform. As this grows to encompass diverse populations across health systems and countries, it will help quantify disease outcomes at increasingly granular resolutions, improve the value of care afforded to patients, provide an insightful window into the frontiers of clinical research, and bring the best of biomedical wisdom to bear for discovering novel drugs. Indeed, such a platform has the potential to serve as an up-to-date logbook of humanity’s health and will find numerous applications that bridge clinical observations (phenotypes) with deep molecular characteristics (genotypes).

However, even the smartest of silicon systems are unable to synthesize the wealth of unstructured text that dominates the clinical notes in EHRs. Hence, the curation of biomedical knowledge into actionable insights remains a manual endeavor.

So what exactly does curation of clinical text into decision support entail?

When a research request is made, a team of nurse practitioners, radiologists and other biomedical curators pore over diverse classes of clinical notes, images and various structured EHR databases. Their goal is to comb through the ocean of unstructured text and images toward structuring a spreadsheet with a schema specifically tailored to the question being posed. For instance, asking the seemingly innocuous question, “What are the typical treatment options and associated outcomes for BRAF V600E-negative melanoma patients who develop Keytruda resistance?” involves an incredible amount of curation effort. Even the first step of merely identifying the pertinent cohort (i.e., BRAF V600E-negative melanoma patients who develop Keytruda resistance) cannot be satiated with prestructured EHR or insurance claims databases alone. This necessitates capturing the salient context linguistically encoded within the clinical notes, while triangulating those insights with various other components of the EHR.

A typical workflow for getting at the relevant cohort in this example involves carving out the pool of patients with a confirmed melanoma diagnosis, determining the subset of those patients administered Keytruda (pembrolizumab) but subsequently switched to other therapies once their tumors stopped responding, and further constraining that subset to patients whose biopsies’ genetic sequencing confirmed the absence of BRAF V600E mutation.

While this process of accurately computing the relevant cohort can be a challenge for curators, the inherent handicaps are amplified while attempting the second portion of the query: typical treatment options and associated outcomes. This is a complex ask even for experienced curators and involves intense inquiry into reams of unstructured information that are rife with nuanced context. These clinical notes are often spread over several years of treatment history and disease progression per patient. It comes as no surprise, then, that the act of manually curating a handful of medical records — for what appears an ostensibly straightforward research request — evolves into an unwieldy exercise.

Even a modest modification made to the initial request often requires that the curation exercise be reconducted. The act of meticulous curation is also exhausting to human brains, exacting burnout and encouraging errors of enervation. Attempting to assuage these problems by calling on multiple curators can result in discordant conclusions, but often provide little to no insights into the underlying causes of discrepancies. While the research questions posed are often tenable at first sight, the initial estimates of pertinent patients generally turn out to be optimistic estimates, as the actual curation exercise reveals far fewer patients who fit the desired criteria. Hence, the highly laborious and human expertise-driven EHR curation process is badly broken, but the demand has never been greater.

Encouraging the development of workflow automation tools via thoughtfully architected software solutions can begin addressing some of the innate challenges of curation. Such progress requires research into how written/oral communication between human beings is encoded linguistically. This domain of natural language processing (NLP) has witnessed a recent renaissance. This has been powered by the dramatic progress of unsupervised neural networks that are requiring smaller and smaller training datasets. The convergence of these technologies with modern workflow automation software for clinical curation has been hindered by the inaccessibility of large-scale de-identified health record databases. This is because, in this age of rampant social media streams such as Facebook, Twitter and Instagram, HIPAA compliance alone is insufficient to mitigate re-identification risk. This is particularly true of the context-rich clinical notes that have proven extraordinarily challenging to de-identify automatically. Unsupervised neural networks are again revealing some initial promise in such automated de-identification.

It is astounding that taming the largely unstructured biomedical knowledge of health records continues to rely heavily on manual curation despite these many vagaries and fallacies. The current state of expert curation may be best described as the perilous pinnacles of human triangulation. With the rapid emergence of unsupervised neural networks that are beginning to hybridize human inquisitiveness and machine intelligence, there is ample scope for optimism. Thanks to this emerging augmented intelligence, the comprehensive chronicling of patient care is poised to have a far-reaching impact on drug discovery and development.