{
  "abstract": "Objectives To evaluate the performance of reasoning large language models (LLMs) with human-like thinking in ophthalmic question answering.Methods We evaluated two state-of-the-art open-source reasoning LLMs (DeepSeek-R1 and QwQ-32B) and one conventional non-reasoning LLM (LLaMA-3.3-70B-Instruct) models on ophthalmology questions, assessing not only answer accuracy (ACC) but also the quality of their reasoning processes. First, we curated MedQA-Eye, a dataset of 967 ophthalmology questions across 10 subspecialties, 3 scenarios, 5 medical entities and 3 languages. Second, we proposed a novel framework considering human thinking patterns essential to medical practice to evaluate the thinking performance of reasoning LLMs on MedQA-Eye.Results DeepSeek-R1 demonstrated superior overall ACC (90.59%, 95% CI 88.59% to 92.27%) to LLaMA-3.3-70B-Instruct (87.90%, 95% CI 85.69% to 89.81%, p=0.015) and QwQ-32B (84.28%, 95% CI 81.85% to 86.44%, p<0.001) with performance varying across subspecialties. Analysis of reasoning LLMs revealed incorrect logical inference as the primary point of failure, accounting for 93.41%–94.74% of incorrectly answered questions. We further quantified semantic uncertainty in reasoning LLM thinking as a predictor of answer reliability. DeepSeek-R1 exhibited lower semantic uncertainty (1.04±3.63) compared with QwQ-32B (4.31±40.70), p<0.001.Conclusion Reasoning LLMs demonstrated superior performance in ophthalmology question answering, with DeepSeek-R1 achieving the highest ACC. Our findings demonstrate that reasoning LLM can better simulate human-like thinking processes compared with conventional non-reasoning LLM, suggesting its potential for more trustworthy LLM systems in ophthalmology.",
  "authors": [
    {
      "affiliations": [
        "Ningbo Key Laboratory of Medical Research on Blinding Eye Diseases, Ningbo Eye Institute, Ningbo Eye Hospital, Wenzhou Medical University, Ningbo, China"
      ],
      "name": "Zhouqian Wang"
    },
    {
      "affiliations": [
        "Ningbo Key Laboratory of Medical Research on Blinding Eye Diseases, Ningbo Eye Institute, Ningbo Eye Hospital, Wenzhou Medical University, Ningbo, China"
      ],
      "name": "Chenjia Xu"
    },
    {
      "affiliations": [
        "National Clinical Research Center for Ocular Diseases, Eye Hospital, Wenzhou Medical University, Wenzhou, China"
      ],
      "name": "Lei Wang"
    },
    {
      "affiliations": [
        "Ningbo Key Laboratory of Medical Research on Blinding Eye Diseases, Ningbo Eye Institute, Ningbo Eye Hospital, Wenzhou Medical University, Ningbo, China"
      ],
      "name": "Wei Qiang"
    },
    {
      "affiliations": [
        "Ningbo Key Laboratory of Medical Research on Blinding Eye Diseases, Ningbo Eye Institute, Ningbo Eye Hospital, Wenzhou Medical University, Ningbo, China"
      ],
      "name": "Yanzhen Li"
    },
    {
      "affiliations": [
        "Department of Ophthalmology, Affiliated Hospital of Guizhou Medical University, Guiyang, China"
      ],
      "name": "Daoyuan Li"
    },
    {
      "affiliations": [
        "Department of Ophthalmology, Qilu Hospital, Shandong University, Jinan, China"
      ],
      "name": "Fabao Xu"
    },
    {
      "affiliations": [
        "Ningbo Key Laboratory of Medical Research on Blinding Eye Diseases, Ningbo Eye Institute, Ningbo Eye Hospital, Wenzhou Medical University, Ningbo, China"
      ],
      "name": "Yanyan Zhang"
    },
    {
      "affiliations": [
        "School of Electronic Engineering, Xi’an University of Posts and Telecommunications, Xi’an, China"
      ],
      "name": "Jiewei Jiang"
    },
    {
      "affiliations": [
        "Ningbo Key Laboratory of Medical Research on Blinding Eye Diseases, Ningbo Eye Institute, Ningbo Eye Hospital, Wenzhou Medical University, Ningbo, China"
      ],
      "name": "Zhongwen Li"
    }
  ],
  "title": "Evaluating reasoning large language models with human-like thinking in ophthalmic question answering",
  "uid": "80bf7c09-80b5-565c-bf84-9c541c8f24e6"
}