{ "abstract": "Background Every measurement comes with statistical uncertainties, arising from the finite data sample, and systematic uncertainties, from errors in the measurement or model. Biomedical analyses typically account only for the statistical uncertainty from the number of patients. These are reported as p-values, power calculations, and other similar metrics.As datasets grow and this statistical uncertainty shrinks, it is not necessarily the dominant uncertainty anymore. A closer look at other uncertainties that affect analyses is critically needed.1 Methods We processed mIF slides from four cohorts containing a total of n=75 pre-treatment specimens from patients with NSCLC using the AstroPath platform, 2 which applies numerous image corrections and produces comprehensive cell maps.We identified CD8+FoxP3+ cells as a positive predictor of response to anti-PD1 immunotherapy in non-small-cell lung cancer (NSCLC). We further developed the Diversity of Niches Unlocking Treatment Sensitivity (DONUTS) biomarker. The DONUTS are niches that mimic the neighborhoods around CD8+FoxP3+ cells but are 150x more common than the cells themselves.3 4 We evaluated both biomarkers’ performance and calculated the statistical uncertainties resulting both from the finite number of patients and from the finite number of cells or DONUTS within each patient‘s biopsy.We processed our data through the AstroPath pipeline again, removing some of the corrections. We estimated the systematic uncertainties that would have been present had we not applied the corrections and the remaining systematic uncertainties resulting from imperfect corrections.We developed the ROC Picker package5 to propagate these uncertainties to ROC and Kaplan-Meier curves.Results With the relatively small cohort sizes (largest: n=25) used in this analysis, the statistical uncertainty from the number of patients still dominates, but other uncertainties contribute appreciably.The statistical uncertainty from the number of CD8+FoxP3+ cells is comparable to the statistical uncertainty from the number of patients (figure 1A), but the statistical uncertainty from the more common DONUTS is much smaller (figure 1B).The systematic uncertainties were also substantial before the corrections in the pipeline, but dropped by more than half after applying those corrections.Conclusions We demonstrate a proof of concept for evaluating statistical and systematic uncertainties in mIF. While standard practice is to consider only one statistical uncertainty, other uncertainties are becoming more important as datasets grow and, if not accounted for, will impact biomarkers and clinical decision making. 6 Our methodology can be applied to biomarkers from all data modalities, ensuring that they remain reliable in the era of big data.References Berry S, Giraldo N, Green B, et al. Analysis of multispectral imaging with the AstroPath platform informs efficacy of PD-1 blockade. Science. 2021;372.Green B, et al. AstroPath Pipeline v0.1.0. Zenodo. 2021.Cottrell T, Roskes J, Cohen E, et al. Early, effector CD8+FoxP3+ cells and their topology associate with outcomes in patients with non-small cell lung carcinoma (NSCLC) receiving neoadjuvant anti-PD-1-based therapy. In preparation. 2025.Cohen E, et al. CD8+FoxP3+ cells represent early, effector T-cells and predict outcomes in patients with resectable non-small cell lung carcinoma (NSCLC) receiving neoadjuvant anti-PD-1-based therapy. J Immunother Cancer. 2022;10:A63.Roskes J. ROC Picker v1.1.0. GitHub. 2024.Taube J, Sunshine J, et al. Society for immunotherapy of cancer: updates and best practices for multiplex immunohistochemistry (IHC) and immunofluorescence (IF) image analysis and data sharing. J Immunother Cancer. 2025;13.Abstract 97 Figure 1Kaplan-Meier curves for regression-free survival for patients with NSCLC, stratified by density of (A) CD8+FoxP3+ cells or (B) DONUTs. The error bands show the statistical error resulting from the finite number of cells or DONUTs", "authors": [ { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Jeffrey S Roskes" }, { "affiliations": [ "Sinclair Cancer Research Institute, Kingston, ON, Canada" ], "name": "Michael R Fotheringham" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Benjamin Green" }, { "affiliations": [ "Memorial Sloan Kettering Cancer Center, Middletown, NJ, USA" ], "name": "Emily Cohen" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Margaret Eminizer" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Sam J Tabrisky" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Sigfredo Soto-Diaz" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Logan L Engle" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Boyang Zhang" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Daphne Wang" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Daniel Jimenez-Sanchez" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Justina X Caushi" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Jiajia Zhang" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Nina D’Amiano" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Joel C Sunshine" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Julie S Deutsch" }, { "affiliations": [ "Queen’s University, Kingston, ON, Canada" ], "name": "Sonali Uttam" }, { "affiliations": [ "Queen’s University, Kingston, ON, Canada" ], "name": "Alexa Fiorante" }, { "affiliations": [ "Queen’s University, Kingston, ON, Canada" ], "name": "Nicole Espinosa" }, { "affiliations": [ "Queen’s University, Kingston, ON, Canada" ], "name": "Teodora Popa" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Aleksandra Ogurtsova" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Dmitry Medvedev" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Andrew Jorquera" }, { "affiliations": [ "Memorial Sloan Kettering Cancer Center, Middletown, NJ, USA" ], "name": "Jamie E Chaft" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Julie R Brahmer" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Michael Conroy" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Joshua Reuss" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Hongkai Ji" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Patrick Forde" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Drew M Pardoll" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Kellie N Smith" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Alexander Szalay" }, { "affiliations": [ "Johns Hopkins University, Baltimore, MD, USA" ], "name": "Janis M Taube" }, { "affiliations": [ "Queen’s University, Kingston, ON, Canada" ], "name": "Tricia R Cottrell" } ], "title": "97 Statistical and systematic uncertainties affect biomarker reliability: a case study in multiplex immunofluorescence", "uid": "580ba54c-e089-5c37-8b49-39a2c342f831" }