{
  "abstract": "Objective To examine the potential errors of a general large language model (LLM) (ie, Claude 3.5 Sonnet) on data extraction from randomised controlled trials (RCTs).Design and setting An empirical study comparing Claude 3.5 Sonnet extractions against a human-performed verification dataset. The extraction tasks for Claude 3.5 Sonnet were based solely on original RCT portable document format (PDF) files. For PDFs that could not be directly extracted by Claude 3.5 Sonnet, optical character recognition was employed to convert them into text format before extraction.Participants A random sample of 664 trials was selected from a well-established trial bank and a final data pool was established based on rigorous manual cross-checking as a reference standard.Data sources PubMed, EMBASE, Scopus, Web of Science (all databases) and the Cochrane Central Register of Controlled Trials (CENTRAL) up to February 2023.Eligibility criteria for selecting studies RCTs on children involving medication and adverse events.Main outcome measures Claude 3.5 Sonnet was applied to extract the basic information (eg, trial design, population information and source of funding) and adverse outcomes (ie, name of adverse events, number of events). Claude 3.5 Sonnet outputs were compared against the final data pool and all errors were recorded. Results are presented as error rates and with 95% CI, estimated using a generalised linear mixed model.Results For the 664 trials, a total of 23 069 data cells were extracted via Claude 3.5 Sonnet, with 10 624 for basic information and 12 445 for adverse outcomes. The overall error rate for data extraction was 6.6% (95% CI 5.4% to 8.2%), with 5.7% (95% CI 5.2% to 6.1%) in basic information and 7.6% (95% CI 4.9% to 11.8%) in adverse outcomes. When stratified the 1542 total errors by error types, misallocation (assigning data to incorrect fields; 57.1%, 881/1542) and missed or omitted data (incomplete extraction of available data; 23.2%, 357/1542) accounted for the two most frequent errors, with misallocation occurring more in basic information (53.3%, 470/881), while missed or omitted data occurred more in adverse outcomes (96.1%, 343/357). Post hoc analysis examining the association between trial reporting quality (assessed using Consolidated Standards of Reporting Trials (CONSORT) 2025 and LLM data extraction error rates indicated that higher CONSORT adherence was associated with lower extraction error rates.Conclusions The data extraction error of Claude was relatively low, but it alerts LLM applications in evidence synthesis. Detailed checking for LLM outputs should be the primary consideration for evidence synthesisers.",
  "authors": [
    {
      "affiliations": [
        "Proof of Concept Center, Shanghai Eastern Hepatobiliary Surgery Hospital, Shanghai, China"
      ],
      "name": "Shiqi Fan"
    },
    {
      "affiliations": [
        "School of Medicine, Shihezi University, Shihezi, China"
      ],
      "name": "Ming Chen"
    },
    {
      "affiliations": [
        "Department of Population Medicine, Qatar University, Doha, Qatar"
      ],
      "name": "Suhail A Doi"
    },
    {
      "affiliations": [
        "The First School of Clinical Medicine, Anhui Medical University, Hefei, China"
      ],
      "name": "Zhangnan Ye"
    },
    {
      "affiliations": [
        "School of Public Health, Anhui Medical University, Hefei, China"
      ],
      "name": "Zhen Peng"
    },
    {
      "affiliations": [
        "Proof of Concept Center, Shanghai Eastern Hepatobiliary Surgery Hospital, Shanghai, China"
      ],
      "name": "Yuan Tian"
    },
    {
      "affiliations": [
        "Department of Information Management, Nanjing University of Science and Technology, Nanjing, China"
      ],
      "name": "Chengzhi Zhang"
    },
    {
      "affiliations": [
        "The University of Queensland, Saint Lucia, Queensland, Australia"
      ],
      "name": "Luis Furuya-Kanamori"
    },
    {
      "affiliations": [
        "Department of Epidemiology and Biostatistics, The University of Arizona Mel and Enid Zuckerman College of Public Health, Tucson, Arizona, USA"
      ],
      "name": "Lifeng Lin"
    },
    {
      "affiliations": [
        "Department of Epidemiology, UNC Gillings School of Global Public Health, Chapel Hill, North Carolina, USA"
      ],
      "name": "Evan Mayo-Wilson"
    },
    {
      "affiliations": [
        "Preventive Medicine, Mayo Clinic Minnesota, Rochester, Minnesota, USA"
      ],
      "name": "Mohammad Hassan Murad"
    },
    {
      "affiliations": [
        "School of Public Health, Anhui Medical University, Hefei, China"
      ],
      "name": "Xiuhong Meng"
    },
    {
      "affiliations": [
        "Proof of Concept Center, Shanghai Eastern Hepatobiliary Surgery Hospital, Shanghai, China",
        "School of Public Health, Center for Big Data and Population Health of IHM, Anhui Medical University, Hefei, Anhui, China"
      ],
      "name": "Chang Xu"
    }
  ],
  "title": "Evaluating data extraction error by a large language model from randomised controlled trials: a large-scale empirical study",
  "uid": "746c98c4-2ec6-5731-9573-35ebc9abe0c3"
}