Download PDF
Appl Clin Inform 2024; 15(03): 489-500
DOI: 10.1055/s-0044-1787185
Research Article

Comparing Clinician Estimates versus a Statistical Tool for Predicting Risk of Death within 45 Days of Admission for Cancer Patients

Adrianna Z. Herskovits   *
1   Department of Pathology and Laboratory Medicine, Memorial Sloan Kettering Cancer Center, New York, New York, United States
,
Tiffanny Newman   *
2   Department of Strategy and Innovation, Memorial Sloan Kettering Cancer Center, New York, New York, United States
,
Kevin Nicholas   *
2   Department of Strategy and Innovation, Memorial Sloan Kettering Cancer Center, New York, New York, United States
,
Cesar F. Colorado-Jimenez
1   Department of Pathology and Laboratory Medicine, Memorial Sloan Kettering Cancer Center, New York, New York, United States
,
Claire E. Perry
2   Department of Strategy and Innovation, Memorial Sloan Kettering Cancer Center, New York, New York, United States
,
Alisa Valentino
1   Department of Pathology and Laboratory Medicine, Memorial Sloan Kettering Cancer Center, New York, New York, United States
,
Isaac Wagner
2   Department of Strategy and Innovation, Memorial Sloan Kettering Cancer Center, New York, New York, United States
,
Barbara Egan
3   Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, New York, United States
,
Dmitriy Gorenshteyn
4   Thirty Madison, New York, New York, United States
,
Andrew J. Vickers
5   Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, New York, United States
,
Melissa S. Pessin
6   Department of Pathology, University of Chicago, Chicago, Illinois, United States
› Author Affiliations Funding Supported in part by the National Institutes of Health/National Cancer Institute Cancer Center Support grant P30 CA008748.
› Further Information
Also available at
    Permissions and Reprints

Abstract

Objectives While clinical practice guidelines recommend that oncologists discuss goals of care with patients who have advanced cancer, it is estimated that less than 20% of individuals admitted to the hospital with high-risk cancers have end-of-life discussions with their providers. While there has been interest in developing models for mortality prediction to trigger such discussions, few studies have compared how such models compare with clinical judgment to determine a patient's mortality risk.

Methods This study is a prospective analysis of 1,069 solid tumor medical oncology hospital admissions (n = 911 unique patients) from February 7 to June 7, 2022, at Memorial Sloan Kettering Cancer Center. Electronic surveys were sent to hospitalists, advanced practice providers, and medical oncologists the first afternoon following a hospital admission and they were asked to estimate the probability that the patient would die within 45 days. Provider estimates of mortality were compared with those from a predictive model developed using a supervised machine learning methodology, and incorporated routine laboratory, demographic, biometric, and admission data. Area under the receiver operating characteristic curve (AUC), calibration and decision curves were compared between clinician estimates and the model predictions.

Results Within 45 days following hospital admission, 229 (25%) of 911 patients died. The model performed better than the clinician estimates (AUC 0.834 vs. 0.753, p < 0.0001). Integrating clinician predictions with the model's estimates further increased the AUC to 0.853 (p < 0.0001). Clinicians overestimated risk whereas the model was extremely well-calibrated. The model demonstrated net benefit over a wide range of threshold probabilities.

Conclusion The inpatient prognosis at admission model is a robust tool to assist clinical providers in evaluating mortality risk, and it has recently been implemented in the electronic medical record at our institution to improve end-of-life care planning for hospitalized cancer patients.

Keywords

mortality risk - model - oncology - prognosis - end-of-life - laboratory values

Protection of Human and Animal Subjects

The study was reviewed by the institutional review board of Memorial Sloan Kettering Cancer Center and is in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects.


* Shared first authorship.


Supplementary Material



Publication History

Received: 15 December 2023

Accepted: 29 April 2024

Article published online:
26 June 2024

© 2024. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Knutzen KE, Sacks OA, Brody-Bizar OC. et al. Actual and missed opportunities for end-of-life care discussions with oncology patients: a qualitative study. JAMA Netw Open 2021; 4 (06) e2113193
    CrossrefPubMedGoogle Scholar
  • 2 Dow LA, Matsuyama RK, Ramakrishnan V. et al. Paradoxes in advance care planning: the complex relationship of oncology patients, their physicians, and advance medical directives. J Clin Oncol 2010; 28 (02) 299-304
    CrossrefPubMedGoogle Scholar
  • 3 Lambden J, Zhang B, Friedlander R, Prigerson HG. Accuracy of oncologists' life-expectancy estimates recalled by their advanced cancer patients: correlates and outcomes. J Palliat Med 2016; 19 (12) 1296-1303
    CrossrefPubMedGoogle Scholar
  • 4 Deelen J, Kettunen J, Fischer K. et al. A metabolic profile of all-cause mortality risk identified in an observational study of 44,168 individuals. Nat Commun 2019; 10 (01) 3346
    CrossrefPubMedGoogle Scholar
  • 5 Christakis NA, Lamont EB. Extent and determinants of error in physicians' prognoses in terminally ill patients: prospective cohort study. West J Med 2000; 172 (05) 310-313
    CrossrefPubMedGoogle Scholar
  • 6 Hui D, Park M, Liu D. et al. Clinician prediction of survival versus the Palliative Prognostic Score: which approach is more accurate?. Eur J Cancer 2016; 64: 89-95
    CrossrefPubMedGoogle Scholar
  • 7 Cheon S, Agarwal A, Popovic M. et al. The accuracy of clinicians' predictions of survival in advanced cancer: a review. Ann Palliat Med 2016; 5 (01) 22-29
    PubMedGoogle Scholar
  • 8 Glare P, Virik K, Jones M. et al. A systematic review of physicians' survival predictions in terminally ill cancer patients. BMJ 2003; 327 (7408) 195-198
    CrossrefPubMedGoogle Scholar
  • 9 Chu C, Anderson R, White N, Stone P. Prognosticating for adult patients with advanced incurable cancer: a needed oncologist skill. Curr Treat Options Oncol 2020; 21 (01) 5
    CrossrefPubMedGoogle Scholar
  • 10 Hui D, Kilgore K, Nguyen L. et al. The accuracy of probabilistic versus temporal clinician prediction of survival for patients with advanced cancer: a preliminary report. Oncologist 2011; 16 (11) 1642-1648
    CrossrefPubMedGoogle Scholar
  • 11 Liao L, Mark DB. Clinical prediction models: are we building better mousetraps?. J Am Coll Cardiol 2003; 42 (05) 851-853
    CrossrefPubMedGoogle Scholar
  • 12 Gupta S, Tran T, Luo W. et al. Machine-learning prediction of cancer survival: a retrospective study using electronic administrative records and a cancer registry. BMJ Open 2014; 4 (03) e004007
    CrossrefPubMedGoogle Scholar
  • 13 Kawai N, Yuasa N. Laboratory prognostic score for predicting 30-day mortality in terminally ill cancer patients. Nagoya J Med Sci 2018; 80 (04) 571-582
    PubMedGoogle Scholar
  • 14 Manz CR, Chen J, Liu M. et al. Validation of a machine learning algorithm to predict 180-day mortality for outpatients with cancer. JAMA Oncol 2020; 6 (11) 1723-1730
    CrossrefPubMedGoogle Scholar
  • 15 Parikh RB, Manz C, Chivers C. et al. Machine learning approaches to predict 6-month mortality among patients with cancer. JAMA Netw Open 2019; 2 (10) e1915997
    CrossrefPubMedGoogle Scholar
  • 16 Temple MW, Lehmann CU, Fabbri D. Natural language processing for cohort discovery in a discharge prediction model for the neonatal ICU. Appl Clin Inform 2016; 7 (01) 101-115
    Article in Thieme ConnectPubMedGoogle Scholar
  • 17 Chopannejad S, Sadoughi F, Bagherzadeh R, Shekarchi S. Predicting major adverse cardiovascular events in acute coronary syndrome: a scoping review of machine learning approaches. Appl Clin Inform 2022; 13 (03) 720-740
    Article in Thieme ConnectPubMedGoogle Scholar
  • 18 Farion KJ, Wilk S, Michalowski W, O'Sullivan D, Sayyad-Shirabad J. Comparing predictions made by a prediction model, clinical score, and physicians: pediatric asthma exacerbations in the emergency department. Appl Clin Inform 2013; 4 (03) 376-391
    Article in Thieme ConnectPubMedGoogle Scholar
  • 19 DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988; 44 (03) 837-845
    CrossrefPubMedGoogle Scholar
  • 20 Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making 2006; 26 (06) 565-574
    CrossrefPubMedGoogle Scholar
  • 21 Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ 2016; 352: i6
    CrossrefPubMedGoogle Scholar
  • 22 Selby D, Chakraborty A, Lilien T, Stacey E, Zhang L, Myers J. Clinician accuracy when estimating survival duration: the role of the patient's performance status and time-based prognostic categories. J Pain Symptom Manage 2011; 42 (04) 578-588
    CrossrefPubMedGoogle Scholar
  • 23 Viganò A, Dorgan M, Bruera E, Suarez-Almazor ME. The relative accuracy of the clinical estimation of the duration of life for patients with end of life cancer. Cancer 1999; 86 (01) 170-176
    CrossrefPubMedGoogle Scholar
  • 24 Bertsimas D, Dunn J, Pawlowski C. et al. Applied informatics decision support tool for mortality predictions in patients with cancer. JCO Clin Cancer Inform 2018; 2: 1-11
    CrossrefPubMedGoogle Scholar
  • 25 Manz CR, Parikh RB, Small DS. et al. Effect of integrating machine learning mortality estimates with behavioral nudges to clinicians on serious illness conversations among patients with cancer: a stepped-wedge cluster randomized clinical trial. JAMA Oncol 2020; 6 (12) e204759
    CrossrefPubMedGoogle Scholar
  • 26 Oo TH. et al. Improved palliative care practices through machine-learning prediction of 90-day risk of mortality following hospitalization. NEJM Catal 2023; 4 (01) 1-20
    PubMedGoogle Scholar
  • 27 Wright AA, Zhang B, Ray A. et al. Associations between end-of-life discussions, patient mental health, medical care near death, and caregiver bereavement adjustment. JAMA 2008; 300 (14) 1665-1673
    CrossrefPubMedGoogle Scholar
  • 28 Zhang B, Wright AA, Huskamp HA. et al. Health care costs in the last week of life: associations with end-of-life conversations. Arch Intern Med 2009; 169 (05) 480-488
    CrossrefPubMedGoogle Scholar
  • 29 Cronin S, Li A, Bai YQ. et al. How do respondents of primary care surveys compare to typical users of primary care? A comparison of two surveys. BMC Prim Care 2023; 24 (01) 80
    CrossrefPubMedGoogle Scholar
  • 30 Wong ST, Hogg W, Burge F, Johnston S, French I, Blackman S. Using the CollaboraKTion framework to report on primary care practice recruitment and data collection: costs and successes in a cross-sectional practice-based survey in British Columbia, Ontario, and Nova Scotia, Canada. BMC Fam Pract 2018; 19 (01) 87
    CrossrefPubMedGoogle Scholar
  • 31 Hendra R, Hill A. Rethinking response rates: new evidence of little relationship between survey response rates and nonresponse bias. Eval Rev 2019; 43 (05) 307-330
    CrossrefPubMedGoogle Scholar
  • 32 Cheng L, DeJesus AY, Rodriguez MA. Using laboratory test results at hospital admission to predict short-term survival in critically ill patients with metastatic or advanced cancer. J Pain Symptom Manage 2017; 53 (04) 720-727
    CrossrefPubMedGoogle Scholar
  • 33 Owusuaa C, van der Padt-Pruijsten A, Drooger JC. et al. Development of a clinical prediction model for 1-year mortality in patients with advanced cancer. JAMA Netw Open 2022; 5 (11) e2244350
    CrossrefPubMedGoogle Scholar
  • 34 Zhu Z, Li L, Ye Z. et al. Prognostic value of routine laboratory variables in prediction of breast cancer recurrence. Sci Rep 2017; 7 (01) 8135
    CrossrefPubMedGoogle Scholar
  • 35 Curtin D, Dahly DL, van Smeden M. et al. Predicting 1-year mortality in older hospitalized patients: external validation of the HOMR model. J Am Geriatr Soc 2019; 67 (07) 1478-1483
    CrossrefPubMedGoogle Scholar
  • 36 van Walraven C, Forster AJ. The HOMR-Now! Model Accurately Predicts 1-Year Death Risk for Hospitalized Patients on Admission. Am J Med 2017; 130 (08) 991.e9-991.e16
    CrossrefPubMedGoogle Scholar
  • 37 van Walraven C, McAlister FA, Bakal JA, Hawken S, Donzé J. External validation of the Hospital-patient One-year Mortality Risk (HOMR) model for predicting death within 1 year after hospital admission. CMAJ 2015; 187 (10) 725-733
    CrossrefPubMedGoogle Scholar
  • 38 Wegier P, Kurahashi A, Saunders S. et al. mHOMR: a prospective observational study of an automated mortality prediction model to identify patients with unmet palliative needs. BMJ Support Palliat Care 2021; 14 (e1): e969-e975
    CrossrefPubMedGoogle Scholar
  • 39 Davis M, Vanenkevort E, Young A. et al. Validation of the surprise question and the development of a multivariable model. J Pain Symptom Manage 2023; 65 (05) 456-464
    CrossrefPubMedGoogle Scholar
  • 40 Moss AH, Lunney JR, Culp S. et al. Prognostic significance of the “surprise” question in cancer patients. J Palliat Med 2010; 13 (07) 837-840
    CrossrefPubMedGoogle Scholar
  • 41 Aquino YSJ, Carter SM, Houssami N. et al. Practical, epistemic and normative implications of algorithmic bias in healthcare artificial intelligence: a qualitative study of multidisciplinary expert perspectives. J Med Ethics 2023; (e-pub ahead of print) DOI: 10.1136/jme-2022-108850.
    CrossrefPubMedGoogle Scholar
  • 42 Medicare.gov. 2022 [cited 2022 12/23/2022]; Accessed May 13, 2024 at: https://www.medicare.gov/coverage/hospice-care
    PubMedGoogle Scholar