{
  "abstract": "Background Diagnosis of Diffuse interstitial lung diseases (DILDs) is a complex task requiring observer experience. This study analyses the accuracy of Large Language Models (LLMs) in making DILD diagnoses from transcribed radiological reports and compares their accuracy to that of thoracic radiologists.Methods LLMs analysed included ChatGPT (GPT-4, o1 and o3-mini) and Gemini (Gemini 2.0 Flash and Gemini 2.0 Flash Thinking). Radiologists of differing seniority (specialist registrars, senior research fellows, consultants [n=11]) from a tertiary referral centre were invited to participate. No patient data were used. Instead, 40 ‘Textbook’ descriptors based on HRCT patterns/signs of common DILDs were created by a consultant radiologist. To mimic varying radiologists’ reporting styles, four descriptors were generated for each DILD: narrative, telegraphic, with/without relevant negative findings. LLMs and radiologists were presented with the same HRCT descriptors in random order. Different prompting methods were used: ‘unrestricted’, ‘restricted’ and ‘engineered’ prompts. For the ‘unrestricted’ prompt, the groups were asked to provide their differential diagnoses in decreasing order of likelihood. For the ‘restricted’ prompt (LLMs only), LLMs were restricted to three differential diagnoses. Few-shot prompting techniques were also used to make an ‘engineered’ prompt, with the aim of improving diagnostic accuracy of LLMs.Results Overall accuracy of the LLMs (unrestricted and restricted prompts): GPT-4o:90%, 93.3%; 01:84.2%, 74.2%; o3-mini: 96.7%, 100%; Gemini 2.0 Flash: 73.3%, 73.3%; Gemini 2.0 Flash Thinking:90%, 90%. Prompt Engineering did not improve diagnostic accuracy of LLMs. Overall accuracy of radiologists (unrestricted prompt): specialist registrars: 81.5%; senior research fellows: 96.5%; consultants: 92.1%.Abstract P95 Figure 1Comparisons of diagnostic accuracy of LLMs and radiologists when given unrestricted prompts. The diagnostic accuracy of five LLMs (GPT-4o, o1, o3-mini, Gemini 2.0 Flash and Gemini 2.0 Flash Thinking) was compared to the diagnostic accuracy of A) Specialist registrars (SpRs), B) Senior research fellows, and C) Consultants. Accuracy of 100% = all first-choice diagnoses were correct. Tukey’s multiple comparisons test was used to compare differences between LLMs. Significant P values are labelled on the graph. Statistical significance was assumed at P<0.05. Error bars signify mean ±SEMConclusion LLMs executed excellent diagnostic capabilities comparable to senior research fellows and consultants. LLMs could be used as adjunctive tools to improve the diagnostic accuracy of less-experienced radiologists. The results of prompt engineering techniques are potentially attributable to imperfect prompting methods. Future work should be targeted at establishing the diagnostic accuracy of LLMs designed for medical use, across a wider variety of DILDs, whilst ensuring safe diagnosis of patients.Reference Sun SH, Huynh K, Cortes G, Hill R, Tran J, Yeh L, et al. Testing the ability and limitations of ChatGPT to generate differential diagnoses from transcribed radiologic findings. Radiology 2024 Oct;313(1):e232346.",
  "authors": [
    {
      "affiliations": [
        "Imperial College London, London, UK"
      ],
      "name": "J Benson"
    },
    {
      "affiliations": [
        "Royal Brompton Hospital, London, UK"
      ],
      "name": "S Desai"
    }
  ],
  "title": "P95 Evaluation of ChatGPT and Gemini in generating differential diagnoses in diffuse interstitial lung diseases (DILDs) from transcribed radiological reports",
  "uid": "998fb954-d6ec-53ef-990a-4902ebcdc03c"
}