{
  "abstract": "Background This study evaluated the effectiveness of large language models (LLMs), specifically ChatGPT 4o and a custom-designed model, Meta-Analysis Librarian, in generating accurate search strings for systematic reviews (SRs) in the field of anesthesiology.Methods We selected 85 SRs from the top 10 anesthesiology journals, according to Web of Science rankings, and extracted reference lists as benchmarks. Using study titles as input, we generated four search strings per SR: three with ChatGPT 4o using general prompts and one with the Meta-Analysis Librarian model, which follows a structured, Population, Intervention, Comparator, Outcome-based approach aligned with Cochrane Handbook standards. Each search string was used to query PubMed, and the retrieved results were compared with the PubMed retrieved studies from the original search string in each SR to assess retrieval accuracy. Statistical analysis compared the performance of each model.Results Original search strings demonstrated superior performance with a 65% (IQR: 43%–81%) retrieval rate, which was statistically different from both LLM groups in PubMed retrieved studies (p=0.001). The Meta-Analysis Librarian achieved a superior median retrieval rate to ChatGPT 4o (median, (IQR); 24% (13%–38%) vs 6% (0%–14%), respectively).Conclusion The findings of this study highlight the significant advantage of using original search strings over LLM-generated search strings in PubMed retrieval studies. The Meta-Analysis Librarian demonstrated notable superiority in retrieval performance compared with ChatGPT 4o. Further research is needed to assess the broader applicability of LLM-generated search strings, especially across multiple databases.",
  "authors": [
    {
      "affiliations": [
        "Department of Medicine (DIMED), Padua University Hospital, University of Padua, Padova, Italy",
        "Anesthesia and Intensive Care Unit, Padua University Hospital, University-Hospital of Padova, Padova, Italy"
      ],
      "name": "Alessandro De Cassai"
    },
    {
      "affiliations": [
        "Department of Anesthesiology and Reanimation, Ondokuz Mayis University Faculty of Medicine, Samsun, Turkey"
      ],
      "name": "Burhan Dost"
    },
    {
      "affiliations": [
        "Department of Anesthesiology and Reanimation, Ataturk University, Erzurum, Turkey"
      ],
      "name": "Yunus Emre Karapinar"
    },
    {
      "affiliations": [
        "Department of Anesthesiology and Reanimation, Samsun University Faculty of Medicine, Canik, Turkey"
      ],
      "name": "Müzeyyen Beldagli"
    },
    {
      "affiliations": [
        "Department of Anesthesiology and Reanimation, Ataturk University, Erzurum, Turkey"
      ],
      "name": "Mirac Selcen Ozkal Yalin"
    },
    {
      "affiliations": [
        "Department of Anesthesiology and Reanimation, Ondokuz Mayis University Faculty of Medicine, Samsun, Turkey"
      ],
      "name": "Esra Turunc"
    },
    {
      "affiliations": [
        "Department of Anesthesiology, Istanbul Health Science University Kanuni Sultan Süleyman Education and Training Hospital, Istanbul, Turkey"
      ],
      "name": "Engin Ihsan Turan"
    },
    {
      "affiliations": [
        "Anesthesia and Intensive Care Unit, Padua University Hospital, University-Hospital of Padova, Padova, Italy"
      ],
      "name": "Nicolò Sella"
    }
  ],
  "title": "Evaluating the utility of large language models in generating search strings for systematic reviews in anesthesiology: a comparative analysis of top-ranked journals",
  "uid": "16c37b99-0461-5069-88cd-f2ad89202bfc"
}