{
  "abstract": "Background In healthcare, most existing uses of artificial intelligence (AI) focus on ‘narrow AI’. These systems are trained to provide binary outputs (i.e. yes/no) as to whether a finding is present or absent on an image. While useful, the real benefit for patients comes when AI is used not just for problem detection, but also to guide and shape the care pathway for patients with evidence-based management and treatment decisions. In this work, we illustrate a pipeline using a common example of bone fractures in children. Non-specialist healthcare practitioners (HCP) may miss up to 1 in 10 fractures on children’s X-rays. Commercial AI tools for this population are limited, and none currently provide information other than presence/absence of a fracture. Our study presents a multimodal AI framework that takes this further. It combines both image-based fracture detection (i.e. a computer vision algorithm) with text-based guidance drawn from national fracture management guidelines. The goal is not only to assist in fracture detection, but also to support frontline practitioners in deciding how best to manage it.Method We trained an AI model (an object detection model) using a multi-centric dataset of paediatric X-rays to detect the location and type of paediatric-specific fractures. The model’s output is then passed into a retrieval system (a RAG pipeline component) that searches clinical guidance literature to identify the optimal patient management. A concise summary of the recommended care plan is then generated.Conclusion For real transformational benefit and ease, AI systems must demonstrate value beyond narrow tasks. Our study illustrates one such approach – combining image-based fracture detection with guideline-based management advice. Although we focus on paediatric fractures, this framework can be adapted to other clinical tasks.Acknowledgements for funding or support CP is funded by the Great Ormond Street Hospital Children’s Charity (GOSHCC) (Award Number: VS0618)OJA is funded by an NIHR Career Development Fellowship (NIHR–CDF–2017–10–037)SCS is funded by an NIHR Advanced Fellowship Award (NIHR–301322)",
  "authors": [
    {
      "affiliations": [
        "UCL Great Ormond Street Institute of Child Health, UK"
      ],
      "name": "Cato Pauling"
    },
    {
      "affiliations": [
        "Great Ormond Street Hospital for Children NHS Foundation Trust, UK"
      ],
      "name": "Jaskaran Kawatra"
    },
    {
      "affiliations": [
        "Great Ormond Street Hospital for Children NHS Foundation Trust, UK"
      ],
      "name": "Pavithra Rajendran"
    },
    {
      "affiliations": [
        "Great Ormond Street Hospital for Children NHS Foundation Trust, UK"
      ],
      "name": "Alastair Hill"
    },
    {
      "affiliations": [
        "Great Ormond Street Hospital for Children NHS Foundation Trust, UK"
      ],
      "name": "Emily Ashworth"
    },
    {
      "affiliations": [
        "UCLH, UK"
      ],
      "name": "Emma Allan"
    },
    {
      "affiliations": [
        "Great Ormond Street Hospital for Children NHS Foundation Trust, UK"
      ],
      "name": "Daniel Key"
    },
    {
      "affiliations": [
        "Great Ormond Street Hospital for Children NHS Foundation Trust, UK"
      ],
      "name": "Alex Eze"
    },
    {
      "affiliations": [
        "Great Ormond Street Hospital for Children NHS Foundation Trust, UK"
      ],
      "name": "Shiren Patel"
    },
    {
      "affiliations": [
        "Great Ormond Street Hospital for Children NHS Foundation Trust, UK"
      ],
      "name": "Owen Arthurs"
    },
    {
      "affiliations": [
        "Great Ormond Street Hospital for Children NHS Foundation Trust, UK"
      ],
      "name": "Susan Shelmerdine"
    }
  ],
  "title": "141 Multimodal AI framework: End-to-end diagnostic pipeline enhanced with retrieval augmented generation",
  "uid": "702a794a-4c4c-56ff-87a6-a69c670fd757"
}