Text mining of clinical outcomes for medical research: how accurate should it be?
Summary
In medicine, clinical prediction models are often developed to estimate future risk of pa-
tients regarding a certain health outcome (e.g., in-hospital mortality). To develop these
models, historic structured data is needed about patient characteristics and the relevant
health outcomes. Sometimes the to be predicted health outcome was not recorded in struc-
tured data but may be extracted from the textual notes by using text mining. If a text
mining model is developed to extract outcome variables from clinical notes, that model
can be used to generate the training data for the prediction model. Contemporary research
often applies text mining, but the impact of text mining quality on prediction model per-
formances in this setting remains unclear. We performed a simulation study that charted
this relationship in a case study of in-hospital mortality prediction in ICUs. We created
a logistic regression and neural network prediction model and trained it on data extrac-
ted by multiple text mining models with a wide range of performance. We varied the
performance of the text mining models by changing the size of the training data used to
develop them and by shifting the decision boundary. We found that analysis can be done
to determine whether the text mining model performs well enough, or whether more data
might be needed for text mining training purposes. We also concluded that shifting the
decision boundary of the text mining model can be a viable way to increase prediction
model performance, especially when a low amount of training data is used. The know-
ledge gained in this project may be used to create better performing prediction models
using text mining models when training data is limited.