{
  "abstract": "Background/aims To compare the performance and cost-effectiveness of DeepSeek-R1 with OpenAI o1 in diagnosing and managing ophthalmology clinical cases.Methods In this cross-sectional study, a total of 300 clinical cases spanning 10 ophthalmology subspecialties were collected from StatPearls, each with a multiple-choice question on diagnosis or management. DeepSeek-R1 was accessed through its public chat interface, while OpenAI o1 was queried via its Application Programming Interface with a standardised temperature of 0.3. Both models were prompted using plan-and-solve+. Performance was calculated as the proportion of correct answers. McNemar’s test was employed to compare the two models’ performance on paired data. Intermodel agreement for correct diagnoses was evaluated via Cohen’s kappa. Token-based cost analyses were performed to estimate the comparative expenditures of running each model at scale, including input prompts and model-generated output.Results DeepSeek-R1 and OpenAI o1 achieved an identical overall performance of 82.0% (n=246/300; 95% CI: 77.3 to 85.9). Subspecialty-specific analysis revealed numerical variation in performance, though none of these comparisons reached statistical significance (p>0.05). Agreement in performance between the models was moderate overall (κ=0.503, p<0.001), with substantial agreement in refractive management/intervention (κ=0.698, p<0.001) and moderate agreement in retina/vitreous (κ=0.561, p<0.001) and ocular pathology/oncology (κ=0.495, p<0.01) cases. Cost analysis indicated an approximately 15-fold reduction in per-query, token-related expenses when using DeepSeek-R1 vs OpenAI o1 for the same workload.Conclusions DeepSeek-R1 shows strong diagnostic and management performance comparable to OpenAI o1 across ophthalmic subspecialties, while significantly reducing costs. These results support its use as a cost-effective, open-weight alternative to proprietary models.",
  "authors": [
    {
      "affiliations": [
        "Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada"
      ],
      "name": "David Mikhail"
    },
    {
      "affiliations": [
        "Faculty of Medicine, McGill University, Ottawa, Ontario, Canada"
      ],
      "name": "Andrew Farah"
    },
    {
      "affiliations": [
        "Department of Software Engineering, University of Waterloo, Waterloo, Ontario, Canada"
      ],
      "name": "Jason Milad"
    },
    {
      "affiliations": [
        "Department of Ophthalmology and Vision Sciences, University of Toronto, Toronto, Ontario, Canada"
      ],
      "name": "Wissam Nassrallah"
    },
    {
      "affiliations": [
        "Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada"
      ],
      "name": "Andrew Mihalache"
    },
    {
      "affiliations": [
        "Department of Ophthalmology, University of Montreal, Montreal, Quebec, Canada",
        "Department of Ophthalmology, Centre Hospitalier de l’Universite de Montreal, Montreal, Quebec, Canada",
        "Department of Ophthalmology, Hopital Maisonneuve-Rosemont, Montreal, Quebec, Canada"
      ],
      "name": "Daniel Milad"
    },
    {
      "affiliations": [
        "Department of Ophthalmology, University of Montreal, Montreal, Quebec, Canada",
        "Department of Ophthalmology, Centre Hospitalier de l’Universite de Montreal, Montreal, Quebec, Canada",
        "Department of Ophthalmology, Hopital Maisonneuve-Rosemont, Montreal, Quebec, Canada",
        "The CHUM School of Artificial Intelligence in Healthcare (SAIH), Centre Hospitalier de l'Université de Montréal (CHUM), Montreal, Quebec, Canada",
        "Cole Eye Institute, Cleveland Clinic, Cleveland, OH, USA"
      ],
      "name": "Fares Antaki"
    },
    {
      "affiliations": [
        "Department of Ophthalmology and Vision Sciences, University of Toronto, Toronto, Ontario, Canada"
      ],
      "name": "Michael Balas"
    },
    {
      "affiliations": [
        "Department of Ophthalmology and Vision Sciences, University of Toronto, Toronto, Ontario, Canada",
        "Department of Ophthalmology, St. Michael’s Hospital/Unity Health Toronto, Toronto, Ontario, Canada"
      ],
      "name": "Marko Popovic"
    },
    {
      "affiliations": [
        "Department of Ophthalmology, University Vita Salute Hospital San Raffaele, Milano, Italy",
        "Retina Division, Stein and Doheny Eye Institutes, Department of Ophthalmology, University of California, Los Angeles, California, USA"
      ],
      "name": "Alessandro Feo"
    },
    {
      "affiliations": [
        "Department of Ophthalmology and Vision Sciences, University of Toronto, Toronto, Ontario, Canada",
        "Department of Ophthalmology, St. Michael’s Hospital/Unity Health Toronto, Toronto, Ontario, Canada"
      ],
      "name": "Rajeev H Muni"
    },
    {
      "affiliations": [
        "Institute of Ophthalmology, University College London, London, UK",
        "Medical Retina, Moorfields Eye Hospital NHS Foundation Trust, London, UK"
      ],
      "name": "Pearse A Keane"
    },
    {
      "affiliations": [
        "Department of Ophthalmology, University of Montreal, Montreal, Quebec, Canada",
        "Department of Ophthalmology, Hopital Maisonneuve-Rosemont, Montreal, Quebec, Canada"
      ],
      "name": "Renaud Duval"
    }
  ],
  "title": "Performance of DeepSeek-R1 in ophthalmology: an evaluation of clinical decision-making and cost-effectiveness",
  "uid": "a534c34b-2fb6-50d1-b7eb-7d4de6cacfae"
}