{
  "abstract": "Introduction Patient safety event reporting systems are widely used, yet organisations face challenges analysing the high volume of incident reports. While low-harm events represent the majority of submissions, they are rarely examined systematically due to the time and resources required to manually review the complex, lengthy, narrative data. Emerging technology like large language models (LLMs) offers new opportunities to address this gap. This study aimed to develop and evaluate an artificial intelligence-driven approach to identify patient safety issues, uncover system-level trends and assess readiness for implementation within a US healthcare system.Methods We quantitatively evaluated OpenAI’s GPT-4o model accuracy in analysing patient safety event reports and qualitatively assessed pre-implementation outcomes. The LLM extracted safety problems from 9357 free-text narratives and then generated a taxonomy of ‘parent’ and ‘child’ subcategories. The model labelled every report’s problem list using the taxonomy, and dashboards were developed to visualise trends. Patient safety experts reviewed two separate subsets of reports (n=100 and n=219) to validate the LLM’s accuracy. We conducted 10 stakeholder interviews to assess the dashboards’ acceptability, appropriateness and adoption.Results The LLM had scores of 94% mean agreement among reviewers in identifying problems, and 91.5% and 83.3% agreement in assigning ‘parent’ and ‘child’ category labels, respectively. The model identified previously hidden patterns of patient safety issues. Stakeholders described LLM-generated insights as clear, appropriate and valuable for their work. They perceived few barriers to adoption and believed the model could expedite manual report reviews and support system-level quality improvement.Conclusion LLMs offer an approach to capture and analyse multidimensional concepts in patient safety reports. By extracting problem summaries and categorising them into an accepted taxonomy, they can expose previously unidentified trends and provide visibility into system-level risks. LLMs could effectively augment and expedite manual reviews of safety events, while guiding prioritisation of quality and safety interventions.",
  "authors": [
    {
      "affiliations": [
        "Division of Pediatric Hospital Medicine, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, USA",
        "Division of Primary Care and Population Health, Department of Medicine, Stanford University School of Medicine, Stanford, California, USA"
      ],
      "name": "Kevin Chen"
    },
    {
      "affiliations": [
        "Stanford Medicine Children’s Health, Palo Alto, California, USA"
      ],
      "name": "Kiley Rogers"
    },
    {
      "affiliations": [
        "Stanford Medicine Children’s Health, Palo Alto, California, USA"
      ],
      "name": "William Haberkorn"
    },
    {
      "affiliations": [
        "Stanford Medicine Children’s Health, Palo Alto, California, USA"
      ],
      "name": "Matthew Lew"
    },
    {
      "affiliations": [
        "Stanford Medicine Children’s Health, Palo Alto, California, USA"
      ],
      "name": "Jeanmarie Kanegan"
    },
    {
      "affiliations": [
        "Stanford Medicine Children’s Health, Palo Alto, California, USA"
      ],
      "name": "Haemi Nam"
    },
    {
      "affiliations": [
        "Stanford Medicine Children’s Health, Palo Alto, California, USA"
      ],
      "name": "Jean Chantra"
    },
    {
      "affiliations": [
        "Division of Primary Care and Population Health, Department of Medicine, Stanford University School of Medicine, Stanford, California, USA",
        "Evaluation Sciences Unit, Division of Primary Care and Population Health, Department of Medicine, Stanford University School of Medicine, Stanford, California, USA"
      ],
      "name": "Steven M Asch"
    },
    {
      "affiliations": [
        "Stanford Medicine Children’s Health, Palo Alto, California, USA",
        "Division of Pediatric Infectious Diseases, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, USA"
      ],
      "name": "Grace M Lee"
    }
  ],
  "title": "AI-driven analysis of patient safety reports using large language models: an exploratory multiple methods study",
  "uid": "ea8a37e9-b7e7-561d-a8a5-ed937347ed87"
}