iCARE study evaluates AI-assisted discharge summaries in a real-world clinical setting

AI

Large language model (LLM)-generated discharge summaries can meet, and in some respects exceed, current approaches. New research led by iCARE (Imperial Clinical Analytics, Research and Evaluation) found that AI-generated discharge summaries were more likely to meet the threshold for clinical acceptability and received higher ratings for accuracy, completeness, clarity and tone than summaries produced through usual clinical practice.

Discharge summaries give GPs and other healthcare professionals a concise account of a patient’s hospital admission, including diagnoses, investigations, treatments, medication changes and follow-up arrangements. They can also help patients and families understand what happened during a hospital stay and what should happen next. Producing a summary requires clinicians to bring together information from across an admission, often drawing on multiple clinical notes, investigations and specialist teams. This work is often completed alongside competing demands for direct patient care and under considerable time pressure. The task has become increasingly complex as patient records have grown in both volume and detail.

“There’s a constant tension between clinical tasks that push a patient’s care forward and administrative tasks, like discharge summaries, that move patients out of hospital.” Explains Dr Benjamin Atkins, Resident Doctor, Imperial College Healthcare NHS Trust, “Both must happen urgently, often for several patients at once and neither can wait. That simultaneous demand is exactly why automation of administrative tasks would be so powerful on the wards.”

Evaluating generative AI in clinical documentation

As healthcare records become increasingly complex, generative AI is attracting interest as a potential tool to support clinical documentation. The key question is not whether these systems can generate convincing text, but whether they can improve clinically important processes without compromising accuracy, safety or professional accountability.
To explore this, researchers conducted a retrospective evaluation using real-world data from the Imperial Secure Data Environment (SDE). They developed an application that used LLMs to generate draft discharge summaries from the unstructured electronic health record data associated with a patient’s hospital admission. This reflected the information-rich, multi-document process clinicians navigate when preparing discharge summaries.

Would you sign your name to this discharge summary?

Researchers assessed whether AI-generated discharge summaries met the standard required for routine clinical use. Twelve doctors from Imperial College Healthcare trust (ICHT) who routinely prepare discharge documentation independently reviewed 1,294 summaries in a blinded evaluation, assessing clinical acceptability, quality and potential for harm. 92 per cent of AI-generated summaries met the threshold for clinical acceptability, meaning they required either no edits or only minor revisions before sign-off, compared with 70 per cent of clinician-written summaries.

AI-generated summaries also received higher ratings for accuracy, completeness, clarity and tone, with reviewers identifying fewer potentially harmful omissions and inaccuracies. Together, these findings suggest that AI-assisted documentation could strengthen the quality of information transferred at discharge, an important component of safe and effective continuity of care.

Dr Fidan Damirova, Resident Doctor, Imperial College Healthcare NHS Trust highlighted that:
“The real promise of this work lies in using AI to transform fragmented clinical information into clear, clinically meaningful discharge communication, with the potential to support safer transitions of care while enabling clinicians to focus more time on what matters most: their patients.”

Human-in-the-loop

A key strength of the study was its clinical co-design approach. The application was developed and evaluated in partnership with resident doctors, including Dr Atkins and Dr Damirova, who regularly prepare discharge summaries at Imperial College Healthcare NHS Trust.

Involving end users from the outset enabled the research team to assess the technology against real clinical needs and workflows. It also ensured that accuracy, completeness, usability and patient safety informed the system’s development. This approach strengthens the relevance of the findings to practice and highlights the importance of clinician involvement in both the design and use of AI-enabled documentation.

For the research team, this collaborative approach was not simply a feature of the study design, but a prerequisite for developing clinically useful and trustworthy AI tools:
Kathleen Goldsmith, lead researcher and Senior Data scientist at iCARE, emphasises the importance of co-design: “To build transformative clinical AI we need to develop applications alongside the clinicians who will use them, and rigorously evaluate their safety, reliability and clinical value. This work is an important step towards making AI a meaningful and trusted part of clinical practice.”

Evidence for responsible adoption

Generative AI is often discussed in terms of future potential, but evidence on its performance in routine clinical practice remains limited. By evaluating AI-generated discharge summaries using real hospital records and blinded clinician assessment, this study moves the conversation from technical capability to practical application.
The findings suggest that discharge documentation may represent a realistic and valuable use case for generative AI in healthcare. More broadly, they demonstrate the importance of evaluating AI against clinical requirements and real-world workflows, providing a model for how emerging technologies can be assessed before wider implementation.
The approach may also have applications beyond discharge documentation. Erik Mayer, Director of iCARE, explains: “Whilst this work focused on discharge summaries, the approach taken is applicable to other use cases that will ultimately support time efficiency gains for healthcare workers and help them spend more time delivering patient-centred care.”

People
  • Erik Mayer
    Erik Mayer
    iCARE Director