Clinical trials provide the “gold standard” evidence for advancing the practice of medicine, even as they evolve to integrate real-world data sources. Modern clinical trials are increasingly incorporating real-world data sources – data not intended for research and often collected in free-living contexts. We refer to trials that incorporate real-world data sources as real-world trials. Such trials may have the potential to enhance the generalizability of findings, facilitate pragmatic study designs, and evaluate real-world effectiveness. However, key differences in the design, conduct, and implementation of real-world vs traditional trials have ramifications in data management that can threaten their desired rigor.

Three examples of real-world trials that leverage different types of data sources – wearables, medical devices, and electronic health records are described. Key insights applicable to all three trials in their relationship to Data and Safety Monitoring Boards (DSMBs) are derived.

Insight and recommendations are given on four topic areas: A. Charge of the DSMB; B. Composition of the DSMB; C. Pre-launch Activities; and D. Post-launch Activities. We recommend stronger and additional focus on data integrity.

Clinical trials can benefit from incorporating real-world data sources, potentially increasing the generalizability of findings and overall trial scale and efficiency. The data, however, present a level of informatic complexity that relies heavily on a robust data science infrastructure. The nature of monitoring the data and safety must evolve to adapt to new trial scenarios to protect the rigor of clinical trials.

EHRs contain a rich source of real-world data that can support evidence generation to better understand mental disorders and improve treatment outcomes. However, EHR datasets are complex and include unstructured free text data that are time consuming to manually review and analyse. We present NeuroBlu, a secure, cloud-based analytic tool that includes bespoke NLP software to enable users to analyse large volumes of EHR data to generate real-world evidence in mental healthcare.

(i) To assemble a large mental health EHR dataset in a secure, cloud-based environment.

(ii) To apply NLP software to extract data on clinical features as part of the Mental State Examination (MSE).

(iii) To analyse the distribution of NLP-derived MSE features by psychiatric diagnosis.

EHR data from 25 U.S. mental healthcare providers were de-identified and transformed into a common data model. NLP models were developed to extract 241 MSE features using a deep learning, long short-term memory (LSTM) approach. The NeuroBlu tool (https://www.neuroblu.ai/) was used to analyse the associations of MSE features in 543,849 patients.

The figure below illustrates the percentage of patients in each diagnostic category with at least one recorded MSE feature.

Delusions and hallucinations were more likely to be recorded in people with schizophrenia and schizoaffective disorder, and cognitive features were more likely to be recorded in people with dementia. However, mood symptoms were frequently recorded across all diagnoses illustrating their importance as a transdiagnostic clinical feature. NLP-derived clinical information could enhance the potential of EHR data to generate real-world evidence in mental healthcare.

This study was funded in full by Holmusk.