Research synthesis as a strategy for advancing biomedical and health informatics knowledge

In my first year as Editor-in-Chief of the Journal of the American Medical Informatics Association (JAMIA), I published an editorial focused on advancing biomedical and health informatics knowledge through reviews of existing research.¹ In the 2019 editorial, I delineated the criteria and best practices for reviews in JAMIA including: (1) address a topic central to biomedical and health informatics and relevant to the JAMIA readership; (2) employ a formal search strategy of multiple databases; (3) depict the flow of information with a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) diagram; (4) document who (minimum of 2) was involved in the flow process, the level of agreement between them, how discrepancies were resolved, and how the process was managed; (5) if a systematic review, apply a method of quality assessment; (6) incorporate a synthesis approach (eg, narrative, tabular, graphical, meta-analysis); (7) analyze what is known, what remains unknown, uncertainty around findings, recommendations for future research, and where relevant recommendations for practice; and (8) register review protocol. In this editorial, I highlight 5 reviews; 4 focus on clinical decision support (CDS)^2–5 and the last on shared tasks for natural language processing (NLP) challenges that use electronic health record (EHR) data.⁶

Two of the manuscripts are systematic reviews with quantitative meta-analyses. Reese et al² addressed complex interventions implemented as CDS using native EHR functionality. They applied the PRISMA reporting tool for complex interventions (PRISMA-CI) to identify the proportion of randomized controlled trials with CDS interventions that were complex. They also described common gaps in the reporting of complexity in CDS research and determined the impact of increased complexity on CDS effectiveness. Seventy-six percent of the 21 studies reviewed evaluated a complex CDS intervention. They evaluated the effect of increased complexity using random-effects meta-analysis finding a small but positive effect in favor of increasing intervention complexity. The authors note the lack of use of analytic frameworks or causal pathways and minimal use of theory. Moreover, they conclude that there is a need for documentation and access to resources to enable replication and adaptation of complex CDS interventions.

In a second systematic review with meta-analysis, Chen et al³ evaluated the design, effectiveness, and economic outcomes of contemporary (2011–2021) chronic disease CDS systems. They registered the review protocol on PROSPERO. Of 80 studies in the review, 76 examined effectiveness outcomes and 9 had economic outcomes; 5 studies included both. Sixty-three percent of the effectiveness studies described an outcome that favored the CDS intervention group. However, the meta-analysis showed that the effect sizes were small and heterogeneous with limited clinical and statistical significance. The economic analysis estimated incremental cost-effective ratios from $2192 to $151 955 per quality-adjusted life year.

Xie et al⁴ conducted a systematic review of randomized controlled trials focused on the effectiveness of clinical dashboards as audit and feedback or CDS tools on medication use and test ordering. Of the 11 randomized controlled trials in the review, 8 studied clinical dashboards as standalone interventions and 3 as part of multicomponent interventions. The authors assessed the risk of bias, certainty of evidence, and synthesized the findings of the trials. The findings related to the standalone dashboards varied showing conflicting evidence across type of medication and clinical population. The 3 trials that investigated dashboards as part of multicomponent interventions showed promising findings including decreased use of opioids for low back pain, increased proportion of patients receiving cardiovascular risk screening, and reduced antibiotic prescribing for upper respiratory tract infections.

Van Dort et al⁵ conducted a meta-synthesis of qualitative studies that explored user acceptance of digital interventions for antimicrobial prescribing and/or monitoring in hospitals. They systematically classified the qualitative data from 15 studies using the unified theory of acceptance and use of technology (UTAUT) model. Thirteen studies focused on CDS for prescribing. The majority of perceptions were classified in the UTAUT performance expectancy domain in the perceived usefulness and relative advantage constructs. The second largest domain was facilitating conditions. These findings highlight the primacy of ensure that the utility of digital interventions for antimicrobial prescribing and/or monitoring meets user expectations and that adequate infrastructure is in place to support use.

Gao et al⁶ applied the PRISMA guideline extension for scoping reviews (PRISMA-ScR) to complete a scoping review of papers on clinical NLP shared tasks that use publicly available EHR data. They categorized 48 clinical tasks from 35 papers by the type of NLP problems addressed (eg, named entity recognition, summarization, document classification, question answering) and data source. They also identified whether the shared task originated from the NLP community within clinical informatics domain or from the general domain NLP community. The majority of tasks were related to named entity recognition followed by information extraction. Notably, almost a third of the corpora for the shared tasks was from the MIMIC cohort and almost all corpora were from single tertiary academic medical center. The authors note that the tasks differ between general NLP domain, which has contributed natural language understanding and generation tasks, while the NLP community within the clinical informatics domain has primarily focused on named entity recognition and documentation classification. The authors call for collaboration between the general NLP domain and clinical informatics domain, reporting transparency, and standardization in data preparation.

Reviews of existing research are critical to advance biomedical and health informatics knowledge. JAMIA remains a premier venue for dissemination of such reviews. I particularly encourage reviews that are at the intersection of informatics and health equity and apply the PRISMA-Equity 2012 Extension.⁷