The Predictive Power of the Physician Narrative: Comparing the Abilities of a Free Text-Based Model and a Variable-Based Model to Predict Patient Outcomes

Introduction: In the era of big data and artificial intelligence, the primacy of the physician over the computer has been called into question. Natural language processing (NLP) allows researchers to leverage free text in the electronic medical record for outcomes modeling, quantifiably demonstrating the value within the physician narrative. Here, we use natural language processing (NLP) and ML to build a model that predicts non-home discharge after craniotomy for meningioma using only basic demographics data and the text of preoperative notes and radiology reports. This model outperforms a more traditional model built using a database of 52 clinical variables preselected by board-certified neurosurgeons for their potential relevance.

Methods: We performed a retrospective study of 598 adults age 18 years or older and surgically treated for intracranial meningioma at our institution from 1995 to 2015. Age, sex, prior operation, and text from the preoperative note and radiology report were collected for each patient. Text was represented via transverse frequency-inverse document frequency. 52 expert-chosen preoperative variables were collected in a second database.

Thirty-two machine learning algorithms were trained to predict non-home discharge from the NLP database or from the 52-variable database. Top performing algorithms for each were combined to form ensemble models. The ensembles were externally validated on data excluded from initial model training.

Area under the curve (AUC) was calculated for the NLP and 52-variable ensembles to compare discriminative ability. Word clouds were generated to visualize which words best predict non-home discharge.

Results: The NLP ensemble predicted non-home discharge with an AUC of 0.80 (95% CI = 0.74–0.86) and 0.76 on internal and external validation compared with an AUC of 0.77 (95% CI = 0.73–0.81) and 0.74 for the 52-variable ensemble.

Words including “progressive,” “large,” “decline,” “edema,” and “effacement” were predictive of non-home discharge.

Conclusion: A model built from free text better predicts non-home discharge for meningioma patients than a model trained on 52 discretely collected variables. These findings demonstrate the value of physician narratives for predicting patient outcomes. Further, this novel method of analysis may represent a less time- and resource-intensive method of generating clinical insights when compared with the traditional process of building and maintaining large databases for predictive modeling.

Fig. 1 Receiver operating characteristic curves and area under the curve (AUC) for the (A) NLP and (B) 52-variable ensemble models.

Fig. 2 Word clouds demonstrating the relative importance of specific words and phrases to the NLP model from the (A) preoperative note and (B) preoperative radiology report. Red font denotes words and phrases associated with non-home discharge, while blue font denotes association with home discharge. Font size is proportional to how influential the word or phrase is in either direction (e.g., a large, red word is highly associated with non-home discharge; a small blue phrase is associated with home discharge, though not as strongly).