{
  "abstract": "Background/aims Large language models (LLMs) have substantial potential to enhance the efficiency of academic research. The accuracy and performance of LLMs in a systematic review, a core part of evidence building, has yet to be studied in detail.Methods We introduced two LLM-based approaches of systematic review: an LLM-enabled fully automated approach (LLM-FA) utilising three different GPT-4 plugins (Consensus GPT, Scholar GPT and GPT web browsing modes) and an LLM-facilitated semi-automated approach (LLM-SA) using GPT4’s Application Programming Interface (API). We benchmarked these approaches using three published systematic reviews that reported the prevalence of diabetic retinopathy across different populations (general population, pregnant women and children).Results The three published reviews consisted of 98 papers in total. Across these three reviews, in the LLM-FA approach, Consensus GPT correctly identified 32.7% (32 out of 98) of papers, while Scholar GPT and GPT4’s web browsing modes only identified 19.4% (19 out of 98) and 6.1% (6 out of 98), respectively. On the other hand, the LLM-SA approach not only successfully included 82.7% (81 out of 98) of these papers but also correctly excluded 92.2% of 4497 irrelevant papers.Conclusions Our findings suggest LLMs are not yet capable of autonomously identifying and selecting relevant papers in systematic reviews. However, they hold promise as an assistive tool to improve the efficiency of the paper selection process in systematic reviews.",
  "authors": [
    {
      "affiliations": [
        "Tsinghua Medicine, Tsinghua University, Beijing, China"
      ],
      "name": "Haichao Chen"
    },
    {
      "affiliations": [
        "Tsinghua Medicine, Tsinghua University, Beijing, China"
      ],
      "name": "Zehua Jiang"
    },
    {
      "affiliations": [
        "Institute of Medical Technology, Peking University Health Science Center, Beijing, China"
      ],
      "name": "Xinyu Liu"
    },
    {
      "affiliations": [
        "Singapore Eye Research Institute, Singapore National Eye Centre, Singapore"
      ],
      "name": "Can Can Xue"
    },
    {
      "affiliations": [
        "Centre for Innovation and Precision Eye Health, Yong Loo Lin School of Medicine, National University of Singapore, Singapore",
        "Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore"
      ],
      "name": "Samantha Min Er Yew"
    },
    {
      "affiliations": [
        "MOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China",
        "Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China"
      ],
      "name": "Bin Sheng"
    },
    {
      "affiliations": [
        "Singapore Eye Research Institute, Singapore National Eye Centre, Singapore",
        "State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangzhou, China"
      ],
      "name": "Ying-Feng Zheng"
    },
    {
      "affiliations": [
        "Beijing Advanced Innovation Center for Biomedical Engineering, Key Laboratory for Biomechanics and Mechanobiology of Ministry of Education, School of Biological Science and Medical Engineering, Beihang University, Beijing, China"
      ],
      "name": "Xiaofei Wang"
    },
    {
      "affiliations": [
        "Tsinghua University School of Medicine, Beijing, China",
        "Department of Health Policy and Management, Johns Hopkins University, Baltimore, Maryland, USA"
      ],
      "name": "You Wu"
    },
    {
      "affiliations": [
        "Moorfields Eye Hospital City Road Campus, London, UK"
      ],
      "name": "Sobha Sivaprasad"
    },
    {
      "affiliations": [
        "Tsinghua Medicine, Tsinghua University, Beijing, China",
        "Singapore Eye Research Institute, Singapore National Eye Centre, Singapore",
        "School of Clinical Medicine, Beijing Tsinghua Changgung Hospital, Tsinghua University, Beijing, China"
      ],
      "name": "Tien Yin Wong"
    },
    {
      "affiliations": [
        "Department of Surgery, Division of Ophthalmology, McMaster University, Hamilton, Ontario, Canada"
      ],
      "name": "Varun Chaudhary"
    },
    {
      "affiliations": [
        "Singapore Eye Research Institute, Singapore National Eye Centre, Singapore",
        "Centre for Innovation and Precision Eye Health, Yong Loo Lin School of Medicine, National University of Singapore, Singapore",
        "Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore",
        "Ophthalmology and Visual Science Academic Clinical Program, Duke-NUS Medical School, Singapore"
      ],
      "name": "Yih Chung Tham"
    }
  ],
  "title": "Can large language models fully automate or partially assist paper selection in systematic reviews?",
  "uid": "faa9ebc8-909e-5150-8c5c-868fd47db740"
}