{
  "abstract": "Background/Objectives Large language models (LLMs) have many touted benefits for medical education, including scoring written content. This pilot study aimed to examine the delivery and scoring of case-based interactions by an LLM, as compared to a panel of faculty educators.Methods Scoring criteria for each case was determined by consensus by four independent expert scorers. The maximum score in each case ranged from five to eight key points that had to be elicited. Student interactions with the LLM-based cases were recorded through the online platform and then independently scored by each expert. The LLM was then provided with the same scoring criteria and asked to ascertain how many key points each student elicited in the individual interactions. In the event of an offset between LLM and expert scorers, a repeat analysis applying calibration was planned to optimise fit.Results The LLM showed high correlation (Pearson R was 0.89, 0.90, 0.85, and 0.67 as compared to the four experts) and high ICC (singled fixed raters 0.82). The ANOVA and post-hoc comparisons demonstrated a significant difference between LLM scores and expert scores. Bland-Altman analysis demonstrated that the LLM scores were approximately 12 percentage points higher than the experts on average. Following calibration, there were no statistically significant differences between LLM and expert scores on ANOVA and post-hoc comparisons.Conclusions LLM scoring of LLM-based case interactions correlate well with expert scorers. Developing and improving LLM-based scoring systems alongside with faculty evaluators may help realize the potential for LLM scoring while mitigating its limitations.",
  "authors": [
    {
      "affiliations": [
        "Adelaide Medical School, University of Adelaide, Adelaide, SA, Australia"
      ],
      "name": "Christina Gao"
    },
    {
      "affiliations": [
        "Harvard Medical School, Boston, USA"
      ],
      "name": "Carolyn Qian"
    },
    {
      "affiliations": [
        "Harvard Medical School, Boston, USA"
      ],
      "name": "Sang-O Park"
    },
    {
      "affiliations": [
        "Harvard Medical School, Boston, USA"
      ],
      "name": "Haelynn Gim"
    },
    {
      "affiliations": [
        "Adelaide Medical School, University of Adelaide, Adelaide, SA, Australia"
      ],
      "name": "Kelly Hou"
    },
    {
      "affiliations": [
        "Adelaide Medical School, University of Adelaide, Adelaide, SA, Australia"
      ],
      "name": "Benjamin Cook"
    },
    {
      "affiliations": [
        "Adelaide Medical School, University of Adelaide, Adelaide, SA, Australia"
      ],
      "name": "Jasmin Le"
    },
    {
      "affiliations": [
        "Adelaide Medical School, University of Adelaide, Adelaide, SA, Australia"
      ],
      "name": "Brandon Stretton"
    },
    {
      "affiliations": [
        "Lyell McEwin Hospital, Elizabeth Vale, SA, Australia"
      ],
      "name": "John Maddison"
    },
    {
      "affiliations": [
        "Massachusetts Institute of Technology, Boston, USA"
      ],
      "name": "Liam McCoy"
    },
    {
      "affiliations": [
        "Adelaide Medical School, University of Adelaide, Adelaide, SA, Australia"
      ],
      "name": "Rudy Goh"
    },
    {
      "affiliations": [
        "Adelaide Medical School, University of Adelaide, Adelaide, SA, Australia"
      ],
      "name": "Matthew Arnold"
    },
    {
      "affiliations": [
        "Harvard Medical School, Boston, USA",
        "Department of Neurology, Mass General Brigham, Boston, USA"
      ],
      "name": "Haatem Reda"
    },
    {
      "affiliations": [
        "Harvard Medical School, Boston, USA",
        "Department of Neurology, Mass General Brigham, Boston, USA"
      ],
      "name": "Tamara Kaplan"
    },
    {
      "affiliations": [
        "Harvard Medical School, Boston, USA",
        "Department of Neurology, Mass General Brigham, Boston, USA"
      ],
      "name": "Galina Gheihman"
    },
    {
      "affiliations": [
        "Adelaide Medical School, University of Adelaide, Adelaide, SA, Australia",
        "Harvard Medical School, Boston, USA",
        "Lyell McEwin Hospital, Elizabeth Vale, SA, Australia",
        "Massachusetts Institute of Technology, Boston, USA",
        "Department of Neurology, Mass General Brigham, Boston, USA",
        "Flinders University, Bedford Park, SA, Australia"
      ],
      "name": "Stephen Bacchi"
    }
  ],
  "title": "3449 Large language model scoring of interactive cases: a pilot study",
  "uid": "e84b4eca-5c3e-5fd2-a8e6-a5dbb390fc00"
}