{
  "abstract": "Objectives Antimicrobial resistance is a critical public health threat. Large language models (LLMs) show great capability for providing health information. This study evaluates the effectiveness of LLMs in providing information on antibiotic use and infection management.Methods Using a mixed-method approach, responses to healthcare expert-designed scenarios from ChatGPT 3.5, ChatGPT 4.0, Claude 2.0 and Gemini 1.0, in both Italian and English, were analysed. Computational text analysis assessed readability, lexical diversity and sentiment, while content quality was assessed by three experts via DISCERN tool.Results 16 scenarios were developed. A total of 101 outputs and 5454 Likert-scale (1–5) scores were obtained for the analysis. A general positive performance gradient was found from ChatGPT 3.5 and 4.0 to Claude to Gemini. Gemini, although producing only five outputs before self-inhibition, consistently outperformed the other models across almost all metrics, producing more detailed, accessible, varied content and a positive overtone. ChatGPT 4.0 demonstrated the highest lexical diversity. A difference in performance by language was observed. All models showed a median score of 1 (IQR=2) regarding the domain addressing antimicrobial resistance.Discussion The study highlights a positive performance gradient towards Gemini, which showed superior content quality, accessibility and contextual awareness, although acknowledging its smaller dataset. Generating appropriate content to address antimicrobial resistance proved challenging.Conclusions LLMs offer great promise to provide appropriate medical information. However, they should play a supporting role rather than representing a replacement option for medical professionals, confirming the need for expert oversight and improved artificial intelligence design.",
  "authors": [
    {
      "affiliations": [
        "Section of Hygiene, University Department of Life Science and Public Health, Università Cattolica del Sacro Cuore, Campus di Roma, Rome, Lazio, Italy",
        "Italian Society for Artificial Intelligence in Medicine (SIIAM), Rome, Italy"
      ],
      "name": "Marcello Di Pumpo"
    },
    {
      "affiliations": [
        "UniCamillus, Saint Camillus International University of Health and Medical Sciences, Rome, Italy"
      ],
      "name": "Maria Rosaria Gualano"
    },
    {
      "affiliations": [
        "Department of Woman and Child Health and Public Health, Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Lazio, Italy",
        "Dipartimento di Scienze della Vita e Sanità Pubblica, Università Cattolica del Sacro Cuore, Campus di Roma, Rome, Lazio, Italy"
      ],
      "name": "Danilo Buonsenso"
    },
    {
      "affiliations": [
        "Department of Laboratory and Infectivology Sciences, Fondazione Policlinico Universitario A Gemelli IRCCS, Rome, Italy"
      ],
      "name": "Francesca Raffaelli"
    },
    {
      "affiliations": [
        "Department of Women’s and Children’s Health, University of Padova, Padua, Italy"
      ],
      "name": "Daniele Donà"
    },
    {
      "affiliations": [
        "Jefferson College of Population Health, Thomas Jefferson University, Philadelphia, Pennsylvania, USA",
        "Asano-Gonnella Center for Research in Medical Education and Health Care, Thomas Jefferson University, Philadelphia, Pennsylvania, USA"
      ],
      "name": "Vittorio Maio"
    },
    {
      "affiliations": [
        "Section of Hygiene, University Department of Life Science and Public Health, Università Cattolica del Sacro Cuore, Campus di Roma, Rome, Lazio, Italy",
        "Department of Woman and Child Health and Public Health, Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Lazio, Italy"
      ],
      "name": "Patrizia Laurenti"
    },
    {
      "affiliations": [
        "Section of Hygiene, University Department of Life Science and Public Health, Università Cattolica del Sacro Cuore, Campus di Roma, Rome, Lazio, Italy"
      ],
      "name": "Walter Ricciardi"
    },
    {
      "affiliations": [
        "Section of Hygiene, University Department of Life Science and Public Health, Università Cattolica del Sacro Cuore, Campus di Roma, Rome, Lazio, Italy",
        "UniCamillus, Saint Camillus International University of Health and Medical Sciences, Rome, Italy"
      ],
      "name": "Leonardo Villani"
    }
  ],
  "title": "Large language models as information providers for appropriate antimicrobial use: computational text analysis and expert-rated comparison of ChatGPT, Claude and Gemini",
  "uid": "05ed4f68-01c9-5238-9eb8-0504d437ba4d"
}