{
  "abstract": "Background Living systematic reviews (LSRs) maintain an updated summary of evidence by incorporating newly published research. While they improve review currency, repeated screening and selection of new references make them labourious and difficult to maintain. Large language models (LLMs) show promise in assisting with screening and data extraction, but more work is needed to achieve the high accuracy required for evidence that informs clinical and policy decisions.Objective The study evaluated the effectiveness of an LLM (GPT-4o) in title and abstract screening compared with human reviewers.Methods Human decisions from an LSR on prodopaminergic interventions for anhedonia served as the reference standard. The baseline search results were divided into a development and a test set. Prompts guiding the LLM’s eligibility assessments were refined using the development set and evaluated on the test set and two subsequent LSR updates. Consistency of the LLM outputs was also assessed.Results Prompt development required 1045 records. When applied to the remaining baseline 11 939 records and two updates, the refined prompts achieved 100% sensitivity for studies ultimately included in the review after full-text screening, though sensitivity for records included by humans at the title and abstract stage varied (58–100%) across updates. Simulated workload reductions of 65–85% were observed. Prompt decisions showed high consistency, with minimal false exclusions, satisfying established screening performance benchmarks for systematic reviews.Conclusions Refined GPT-4o prompts demonstrated high sensitivity and moderate specificity while reducing human workload. This approach shows potential for integrating LLMs into systematic review workflows to enhance efficiency.",
  "authors": [
    {
      "affiliations": [
        "Department of Psychiatry, University of Oxford, Oxford, UK",
        "Division of Clinical Informatics, Beth Israel Deaconess Medical Center, Boston, Massachusetts, USA"
      ],
      "name": "Ava Homiar"
    },
    {
      "affiliations": [
        "University College London, London, UK"
      ],
      "name": "James Thomas"
    },
    {
      "affiliations": [
        "Department of Psychiatry, University of Oxford, Oxford, UK"
      ],
      "name": "Edoardo G Ostinelli"
    },
    {
      "affiliations": [
        "Department of Psychiatry, University of Oxford, Oxford, UK"
      ],
      "name": "Jaycee Kennett"
    },
    {
      "affiliations": [
        "Department of Psychiatry, University of Oxford, Oxford, UK"
      ],
      "name": "Claire Friedrich"
    },
    {
      "affiliations": [
        "Vrije Universiteit Amsterdam, Amsterdam, The Netherlands"
      ],
      "name": "Pim Cuijpers"
    },
    {
      "affiliations": [
        "Psychology & Digital Mental Health Care, Technical University of Munich, Munchen, Germany"
      ],
      "name": "Mathias Harrer"
    },
    {
      "affiliations": [
        "Psychiatry and Psychotherapy, Technical University of Munich School of Medicine, Munich, Germany"
      ],
      "name": "Stefan Leucht"
    },
    {
      "affiliations": [
        "Department of Clinical, Neuro and Developmental Psychology, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands"
      ],
      "name": "Clara Miguel"
    },
    {
      "affiliations": [
        "Technical University of Munich, Munich, Germany"
      ],
      "name": "Alessandro Rodolico"
    },
    {
      "affiliations": [
        "Kyoto Min-iren Asukai Hospital, Kyoto, Japan"
      ],
      "name": "Yuki Kataoka"
    },
    {
      "affiliations": [
        "Kyoto University, Kyoto, Japan"
      ],
      "name": "Tomohiro Takayama"
    },
    {
      "affiliations": [
        "Kyoto University, Kyoto, Japan"
      ],
      "name": "Keisuke Yoshimura"
    },
    {
      "affiliations": [
        "Okayama Psychiatric Medical Center, Okayama, Japan"
      ],
      "name": "Ryuhei So"
    },
    {
      "affiliations": [
        "Kyoto University, Kyoto, Japan"
      ],
      "name": "Yasushi Tsujimoto"
    },
    {
      "affiliations": [
        "The University of Tokyo, Bunkyo, Tokyo, Japan"
      ],
      "name": "Yosuke Yamagishi"
    },
    {
      "affiliations": [
        "Independent Researcher, Kyoto, Japan"
      ],
      "name": "Shiro Takagi"
    },
    {
      "affiliations": [
        "Nagoya City University Graduate School of Medical Sciences and Medical School, Nagoya, Japan"
      ],
      "name": "Masatsugu Sakata"
    },
    {
      "affiliations": [
        "Vrije Universiteit Amsterdam, Amsterdam, The Netherlands"
      ],
      "name": "Đorđe Bašić"
    },
    {
      "affiliations": [
        "Department of Clinical, Neuro and Developmental Psychology, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands"
      ],
      "name": "Eirini Karyotaki"
    },
    {
      "affiliations": [
        "NIHR Oxford Health Biomedical Research Centre, Oxford, UK"
      ],
      "name": "Jennifer Potts"
    },
    {
      "affiliations": [
        "University of Bern, Bern, Switzerland"
      ],
      "name": "Georgia Salanti"
    },
    {
      "affiliations": [
        "Kyoto University Graduate School of Medicine Faculty of Medicine, Kyoto, Japan"
      ],
      "name": "Toshi A Furukawa"
    },
    {
      "affiliations": [
        "Department of Psychiatry, University of Oxford, Oxford, UK"
      ],
      "name": "Andrea Cipriani"
    }
  ],
  "title": "Development and evaluation of prompts for a large language model to screen titles and abstracts in a living systematic review",
  "uid": "f7380dfd-a43f-5159-a234-38527eec1611"
}