A systematic review and meta-analysis of deep learning approaches for clinical natural language processing: a hybrid transformer framework with prisma 2020 methodology

Abstract

Clinical documents, such as discharge summaries, radiology reports, clinical notes and pathology records are the most valuable source of patient health information, but information in them is largely untapped and unstructured for large-scale computational analysis. The use of accurate text extraction, classification and summarisation of clinical text could greatly shorten clinical decision support, pharmacovigilance, clinical trial recruitment and epidemiological surveillance. Transformer architectures and large biomedical corpora pre-trained models have led to significant improvements in clinical NLP benchmarks, like Bio BERT, Clinical BERT and PubMed BERT. Yet there is no systematic review in the field that is quantitative and follows the guidelines of PRISMA 2020 to analyses performance trends over architectures and tasks. In this study, we make three contributions: Firstly, conduct a PRISMA 2020 compliant systematic review and meta-analysis of 312 peer-reviewed studies from 2018 to 2025; Secondly, uncover architectural trends of the past eight years; and Thirdly, propose and empirically test a Hybrid Transformer architecture that uses Clinical BERT for encoding and a GPT-2 clinical decoder to be integrated through a multi-head cross-attention bridge. The results of this meta-analysis clearly show that models based on the RNN architecture (between 64.8% and 65.8% F1 in 2018–2019) are outperformed by those based on the BERT architecture (between 76.1% and 78.3% F1 in 2020–2022) and, in turn, by hybrid transformer models (from 87.8% to 89.7% F1 in 2023–2025). The proposed Hybrid Transformer performs at BLEU-4 = 51.8, ROUGE-1 = 68.4, and F1 = 91.2% for the clinical summarisation benchmark; all of which outperform all the baselines evaluated. The results of the risk of bias assessment by PROBAST showed that 26.9% of studies had a high risk of bias with the highest risk of bias being in the analysis domain. The results confirm the state-of-the-art of hybrid encoders-decoders for clinical NLP and inspire further research on multilingual pre-training and federated learning for privacy-preserving model development in the clinical domain.