{
  "abstract": "Introduction Few artificial intelligence (AI) clinical decision support systems (CDSSs) are ever evaluated in practice. Although some signal of clinical effectiveness may be needed to justify AI deployment and testing, such data are typically unavailable in early-stage research. This conundrum is especially relevant in the intensive care unit (ICU), where conditions like sepsis and acute respiratory distress syndrome (ARDS) require high-stakes decisions. Our group developed the AI ventilator assistant (AVA), a novel AI CDSS for patients with sepsis ARDS receiving invasive mechanical ventilation. But the promising results of predictive performance estimates are not sufficient to assess AVA’s clinical safety and appropriateness prior to future evaluation and deployment. Therefore, we propose a Clinician Turing Test as a novel validation approach to determine whether clinicians can distinguish AVA-generated treatment recommendations from those enacted by real human clinicians. If AVA’s recommendations are consistently indistinguishable from those of real clinicians, thereby ‘passing’ this Turing test, this would provide a strong preclinical signal of safety and appropriateness.Methods and analysis This multisite, randomised, electronic, vignette-based Phase 1b study will use a Clinician Turing Test design. We aim to recruit 350 critical care clinicians, including physicians and advanced practice providers from six US hospitals. Participants will review nine clinical vignettes of patients with sepsis and ARDS derived from the Molecular Epidemiology of Severe Sepsis in the ICU cohort and an associated profile of a suggested treatment plan. For each participant–vignette combination, the source of the treatment profile will be randomly assigned (AI-generated by AVA vs the actually enacted treatment from real human clinicians) in a 1:1 allocation. The primary endpoint is the participants’ accuracy in identifying whether a treatment profile was AI-generated or human-generated, assessed using equivalence testing through a mixed-effects logistic regression model with random effects for participants and vignettes. Secondarily, a fitted binary classifier will assess discrimination ability using the C-statistic. Secondary endpoints include clinicians’ perceptions of the safety and appropriateness of the treatment profiles, confidence in distinguishing AI-generated and human-generated recommendations, interest in AI CDSSs for sepsis and ventilator management and the time to complete the survey. This novel Phase 1b design provides preliminary but essential information about an AI CDSS’s clinical appropriateness without the risk or cost of actual deployment, thereby informing decisions about future clinical implementation and evaluation in real clinical environments.Ethics and dissemination This protocol was approved by the Institutional Review Board of the University of Pennsylvania (Protocol #858201). Results are expected in 2026 and will be submitted for publication in peer-reviewed journals and presented at scientific conferences.Trial registration number NCT07025096.",
  "authors": [
    {
      "affiliations": [
        "Palliative and Advanced Illness Research (PAIR) Center, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA"
      ],
      "name": "Antonia Angeli Gazola"
    },
    {
      "affiliations": [
        "Palliative and Advanced Illness Research (PAIR) Center, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA"
      ],
      "name": "Nicholas S Bishop"
    },
    {
      "affiliations": [
        "Palliative and Advanced Illness Research (PAIR) Center, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA"
      ],
      "name": "Benjamin E Schmid"
    },
    {
      "affiliations": [
        "Department of Anesthesia and Perioperative Care, University of California San Francisco, San Francisco, California, USA"
      ],
      "name": "Romain Pirracchio"
    },
    {
      "affiliations": [
        "Division of Pulmonary and Critical Care, Department of Medicine, University of Michigan, Ann Arbor, Michigan, USA",
        "Center for Bioethics and Social Sciences in Medicine, University of Michigan, Ann Arbor, Michigan, USA",
        "Institute for Healthcare Policy and Innovation, University of Michigan, Ann Arbor, Michigan, USA",
        "Department of Veterans Affairs, VA Ann Arbor Healthcare System, Ann Arbor, Michigan, USA"
      ],
      "name": "Thomas S Valley"
    },
    {
      "affiliations": [
        "Department of Medicine, Emory University, Atlanta, Georgia, USA",
        "Emory Critical Care Center, Atlanta, Georgia, USA",
        "Division of Pulmonary, Allergy, Critical Care and Sleep Medicine, Emory University School of Medicine, Atlanta, Georgia, USA"
      ],
      "name": "Sivasubramanium V Bhavani"
    },
    {
      "affiliations": [
        "Divisions of Pulmonary, Critical Care, and Sleep Medicine, University of Nebraska Medical Center, Omaha, Nebraska, USA",
        "Department of Anesthesiology, University of Nebraska Medical Center, Omaha, Nebraska, USA"
      ],
      "name": "Dustin C Krutsinger"
    },
    {
      "affiliations": [
        "Department of Medicine, University of Pennsylvania Division of Pulmonary Allergy and Critical Care, Philadelphia, Pennsylvania, USA",
        "Virtua Pulmonology and Sleep Medicine, Virtua Health Inc, Marlton, New Jersey, USA"
      ],
      "name": "Heather M Giannini"
    },
    {
      "affiliations": [
        "Palliative and Advanced Illness Research (PAIR) Center, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA"
      ],
      "name": "Yingying Lu"
    },
    {
      "affiliations": [
        "Department of Computer and Information Science (CIS), University of Pennsylvania, Philadelphia, Pennsylvania, USA"
      ],
      "name": "Lyle H Ungar"
    },
    {
      "affiliations": [
        "Department of Medicine, University of Pennsylvania Division of Pulmonary Allergy and Critical Care, Philadelphia, Pennsylvania, USA",
        "Translational Lung Biology, Lung Biology Institute, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA"
      ],
      "name": "Nuala J Meyer"
    },
    {
      "affiliations": [
        "Palliative and Advanced Illness Research (PAIR) Center, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA",
        "Department of Medicine, University of Pennsylvania Division of Pulmonary Allergy and Critical Care, Philadelphia, Pennsylvania, USA",
        "Leonard Davis Institute of Health Economics, Philadelphia, Pennsylvania, USA"
      ],
      "name": "Meeta Prasad Kerlin"
    },
    {
      "affiliations": [
        "Palliative and Advanced Illness Research (PAIR) Center, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA",
        "Department of Medicine, University of Pennsylvania Division of Pulmonary Allergy and Critical Care, Philadelphia, Pennsylvania, USA",
        "Leonard Davis Institute of Health Economics, Philadelphia, Pennsylvania, USA",
        "Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, Pennsylvania, USA"
      ],
      "name": "Gary E Weissman"
    }
  ],
  "title": "Evaluating AI-based comprehensive clinical decision support for sepsis and ARDS: protocol for a Clinician Turing Test",
  "uid": "fcb0c2d6-2bc9-5262-9933-63eb2cbd7d32"
}