Randomized Comparison of Electronic Health Record Alert Types in Eliciting Responses about Prognosis in Gynecologic Oncology Patients

Abstract

Objectives To compare the ability of different electronic health record alert types to elicit responses from users caring for cancer patients benefiting from goals of care (GOC) conversations.

Methods A validated question asking if the user would be surprised by the patient's 6-month mortality was built as an Epic BestPractice Advisory (BPA) alert in three versions—(1) Required on Open chart (pop-up BPA), (2) Required on Close chart (navigator BPA), and (3) Optional Persistent (Storyboard BPA)—randomized using patient medical record number. Meaningful responses were defined as “Yes” or “No,” rather than deferral. Data were extracted over 6 months.

Results Alerts appeared for 685 patients during 1,786 outpatient encounters. Measuring encounters where a meaningful response was elicited, rates were highest for Required on Open (94.8% of encounters), compared with Required on Close (90.1%) and Optional Persistent (19.7%) (p < 0.001). Measuring individual alerts to which responses were given, they were most likely meaningful with Optional Persistent (98.3% of responses) and least likely with Required on Open (68.0%) (p < 0.001). Responses of “No,” suggesting poor prognosis and prompting GOC, were more likely with Optional Persistent (13.6%) and Required on Open (10.3%) than with Required on Close (7.0%) (p = 0.028).

Conclusion Required alerts had response rates almost five times higher than optional alerts. Timing of alerts affects rates of meaningful responses and possibly the response itself. The alert with the most meaningful responses was also associated with the most interruptions and deferral responses. Considering tradeoffs in these metrics is important in designing clinical decision support to maximize success.

Keywords

electronic health records - alert fatigue - advance care planning

Background and Significance

While electronic health records (EHRs) have the potential to improve the delivery of safe patient care, concerns about alert fatigue and decreased end-user responsiveness are well described,[1] [2] [3] [4] [5] [6] and multiple groups have reported large-scale success at reducing interruptive alerts.[7] [8] As EHR systems have evolved, options for clinical decision support (CDS), including alerts, have expanded. For example, the EHR developed by Epic Systems Corporation (Verona, Wisconsin, United States) now allows passive alerts on the “Storyboard” (which we call “Optional Persistent” alerts in this article) to remain present throughout the user's experience, providing persistent visibility not previously possible.[9] Also, a scrollable, passive alert within a “Navigator” tab can be configured to require an answer with a hard stop on closing the chart (“Required on Close”). Both are potential alternatives to an interruptive pop-up on opening the chart (“Required on Open”).

With more CDS options available, it is critical to assess how these tools perform in clinical practice to ensure patient care is not negatively impacted and end-users are not unnecessarily burdened. Informatics data are needed to understand the relative effectiveness of different forms of alerts in the EHR, particularly comparing interruptive alerts (with their greater risk of alert fatigue) to passive alerts (with their greater risk of being ignored). In a randomized comparison of interruptive versus passive alerts for patients with heart failure, Blecker et al found increases in response rates and use of recommended medication with the interruptive strategy.[10] During human-centered design testing of CDS for managing anemia in inflammatory bowel disease, Miller et al were surprised to find that their test users stated preferences for an interruptive pop-up alert over a noninterruptive one.[11] To our knowledge, however, traditional interruptive alerts have not been compared head-to-head with current options for Storyboard and required Navigator alerts.

In clinical care, early and ongoing goals of care (GOC) conversations are a critical component of advance care planning for patients with advanced or recurrent cancer to ensure the delivery of goal-concordant cancer care.[12] Despite this knowledge, persistent barriers exist in both the identification of appropriate patients and documentation of these conversations in the EHR. Previous data demonstrated that more than one-third of patients with recurrent, incurable ovarian cancer did not have a GOC conversation documented until their final hospital admission before death.[13] This prompted a quality improvement initiative that successfully identified appropriate patients earlier in an outpatient setting,[14] but it relied on nursing efforts to identify patients for GOC conversations. Since that model was considered unsustainable from a workforce perspective, there was a need to explore other methods of prompting GOC conversations, such as EHR alerts.

The conventional “surprise question” (“Would you be surprised if this patient died in the next 12 months?”) is designed to identify patients at high risk of death within 12 months.[15] It has been validated in multiple settings, including gynecologic oncology (Gyn Onc) patients.[16] In the context of gynecologic cancers, our group's previous work has focused on timely identification of patients at high risk of death within 6 months,[17] a cohort in whom GOC discussion is particularly appropriate. We therefore employed a modified 6-month surprise question to prospectively study three alert strategies.

Objectives

Given the goal of our Gyn Onc and palliative care (PC) clinicians to increase GOC conversations, as well as the lack of quantitative data comparing interruptive pop-up alerts to less-interruptive alternatives, our objective was to compare the performance of different versions of an EHR alert in eliciting responses to the surprise question. We hoped to inform design considerations often encountered by informatics teams managing the EHR and facing an array of technical options.

Interruptive or passive? How will response rate be affected by the various alert strategies, given differences in workflow timing and requirement for response? Will the content of the answer change if the user can respond later in their workflow? Will the user be more likely to provide an answer, rather than defer?

Required or optional? Of the less-interruptive options, how many more responses will there be if we do require a response (Required on Close) than if we do not (Optional Persistent)? Does the persistent visibility of the Storyboard affect the content of the answer?

Methods

We conducted a randomized, prospective study of three different configurations of a surprise question alert designed to identify patients with gynecologic cancers who are at risk of death within 6 months and to assist in clinical decision-making and action.

Alert Construction

Institutional review board exemption was obtained prior to study activities. Gyn Onc and PC physicians collaborated with an EHR analyst to build the alert. We utilized BestPractice Advisory (BPA) technology, a common technique for building CDS alerts within the Epic EHR. Criteria were built to identify any patient who had: at least one clinical code that may be associated with advanced or recurrent gynecologic cancer; a prior visit to Gyn Onc clinic within the past year; and either a computed tomography scan, CA125 blood test, or chemotherapy encounter associated with the current visit. The alert was then configured to appear for those patients who lacked GOC documentation (defined as an “Advance Care Planning” note) within the previous 30 days.

Three different versions of the alert were created ([Fig. 1]), sharing a common design but appearing in different places, under different scenarios, and with different response requirements. The Required on Open alert ([Fig. 1A]) was an interruptive pop-up triggered by opening the patient's outpatient encounter, requiring a response to close the modal window and proceed in the clinical workflow. The Required on Close alert ([Fig. 1B]) became visible on a scrollable “Navigator” when the user opened either the “Charting” or “Plan” tab (both commonly, but not always, used) and, once visible, required a response before closing the chart. The Optional Persistent alert ([Fig. 1C]) appeared as a small but persistent yellow banner on the “Storyboard” left sidebar; it never required a response, and users saw its full interface (including response buttons) only if they chose to click on it.

Fig. 1 Composite view of all three alert types. (Note that for real patients, only one type of alert appeared.) (A) Required on Open popped up on opening the chart and could not be dismissed without a response. Color annotations indicate which response buttons were considered meaningful (green) or deferral (red). (B) Required on Close appeared when Charting or Plan tab was accessed; requirement to respond was enforced on closing the chart (e.g., signing the encounter). (C) Optional Persistent remained visible throughout visit in Storyboard sidebar and never required a response; clicking the yellow banner caused the alert to pop up and reveal response buttons.

Clinical Setting, Timeline, and Randomization

The alerts appeared for patients seen by one of eight Gyn Onc physicians or seven advanced practice providers at two outpatient locations within the health system during a 6-month time period (August 1, 2021–January 31, 2022).

Only one of the three alerts appeared for each patient, based on the last digit of the medical record number. When an eligible patient was identified, if the last digit was 0 to 3, the Required on Close alert appeared; if 4 to 6, the Required on Open alert appeared; and if 7 to 9, the Optional Persistent alert appeared. We recognized that this strategy would result in more patients triggering Required on Close (40%) than Required on Open (30%) and Optional Persistent (30%), so metrics were focused on rates and proportions rather than total counts.

Clinician Interface and Intended Responses

The user interface of the alert asked the clinician, “Would you be surprised if the patient passed away in the next 6 months?” Clinicians responded by selecting an Acknowledge Reason button (see [Fig. 1A]), and the text captions of these buttons were used to guide the appropriate next steps. Meaningful responses: if the clinician expected a longer life expectancy, then “Yes-I would be surprised” would be selected, and no further action was recommended. If they would not be surprised (i.e., the clinician expected a life expectancy of 6 months or less), then “No-Document GOC/Consider PC consult” was selected, and the clinician was prompted to initiate and document a GOC discussion and consider a referral to PC, similar to our prior work.[14] Once a meaningful answer was provided, the alert was suppressed for the remainder of the encounter. Deferral response: the final option, “Show me this next time,” let the clinician dismiss the alert temporarily but allowed it to reappear under the same triggering conditions (e.g., re-opening the chart). The deferral option was intended for clinicians unable to provide an answer in that moment, due to lack of familiarity with the clinical details, and it was available only for required alerts.

Definitions of Effectiveness

Our primary outcome was the proportion of encounters in which a meaningful response (“Yes” or “No”) was obtained. Secondary outcomes included total alerts per encounter, the proportion of encounters with any response (including deferrals), and the proportion of responses that indicated a worse prognosis (answered with “No”). Clinical outcomes were collected as part of this study and will be presented in a separate manuscript upon completion of clinical follow-up.

Data on the triggered alerts were obtained via the “SlicerDicer” system in Epic, which allows self-service analytics. Users access the system to define a dataset (e.g., all unique patients triggering any of these three BPAs) and then subdivide (“slice”) the population by certain variables (e.g., by specific BPA and then by the “Reason” button selected) and selecting desired metrics (e.g., unique alerts, unique patients, unique encounters).

Statistical Analysis

Descriptive statistics were used for the three cohorts based on alert format. To assess differences in responses to the various alert formats, we performed chi-square testing of categorical variables. p-Values ≤0.05 were considered statistically significant.

Results

Demographics

The alert fired on 685 unique patients during 1,786 outpatient encounters over the 6-month study period. The median age of patients was 64 years (range 17–89). Most patients were non-Hispanic (90.1%) and white (65.0%). As shown in [Table 1], overall counts of patients and encounters for each alert type suggest that 4:3:3 randomization produced the expected numbers of patients in each group: Required on Close alerted for 279 (40.7%) patients, and Required on Open and Optional Persistent each alerted for 203 (29.6%) patients.

Table 1 Patient and encounter distribution across three alert types

	Alert type
	Required on Open	Required on Close	Optional Persistent
Allocation method	3 digits (4–6)	4 digits (0–3)	3 digits (7–9)
Unique patients	203 (29.6%)	279 (40.7%)	203 (29.6%)
Median age	63 years	66 years	66 years
Encounters	521	677	588
Encounters/patient	2.6	2.4	2.9

Eliciting User Responses

Recognizing that these alerts can appear multiple times in a single outpatient encounter, we focused first on metrics at the encounter level. We assessed the overall response rate (providing any answer, including deferral) by alert type. As shown in [Table 2], the two required alerts, Required on Close and Required on Open, elicited responses in 96.1 and 100% of encounters, respectively. In contrast, the Optional Persistent alert was visible during 588 encounters, but users responded in only 117 (19.9%; p < 0.001).

Table 2 Comparison of three alert types on measures of user response

	Alert type			p-Values[a]
	Required on Open	Required on Close	Optional Persistent	3-way	RO vs. RC	RO vs. OP	RC vs. OP
Total alerts	757	873	–[b]	–	–	–	–
Total encounters	521	677	588	–	–	–	–
Any response	521 (100%)	651 (96.1%)	117 (19.9%)	<0.001	<0.001	<0.001	<0.001
Meaningful response	494 (94.8%)	610 (90.1%)	116 (19.7%)	<0.001	0.003	<0.001	<0.001
Initial deferral	165 (31.7%)	74 (10.9%)	2 (0.3%)	<0.001	<0.001	<0.001	<0.001
Alerts/encounter	1.5	1.3	–[b]	–	–	–	–
Total responses	757	697	120	–	–	–	–
Meaningful	515 (68.0%)	616 (88.4%)	118 (98.3%)	<0.001	<0.001	<0.001	<0.001
“No” (worse prognosis)	53 (10.3%)	43 (7.0%)	16 (13.6%)	0.028	0.047	0.304	0.016
“Yes”	462 (89.7%)	573 (93.0%)	102 (86.4%)

Abbreviations: OP, Optional Persistent; RC, Required on Close; RO, Required on Open.

^a Pairwise tests were performed if 3-way p-value was statistically significant.

^b Count of alerts was not applicable for Optional Persistent since it remained persistently visible and noninterruptive.

The proportion of encounters with a meaningful response (i.e., with at least one “Yes” or “No” response) was highest for Required on Open (94.8% of encounters), followed closely by Required on Close (90.1%) and far ahead of Optional Persistent (19.7%; p < 0.001).

Alert-level metrics were also important, since each interruptive alert could negatively impact clinical workflow. For each required alert, the user was much more likely to give a meaningful response to Required on Close (88.4%) than to Required on Open (68.0%); in other words, Required on Open was much more likely to elicit a deferral response. For the Optional Persistent alert, where the user must make the conscious choice to respond (which thus happens much less often than with the other types), the response was almost always meaningful (98.3%, p < 0.001).

To assess whether different alert types were correlated with different predictions of prognosis, we compared the proportion of “No” versus “Yes” responses to the surprise question. A “No” response (predicting more severe illness and, for the user, recommending additional tasks for GOC conversation and PC referral) was selected more often with Optional Persistent (13.6% of meaningful responses) than with Required on Open (10.3%) or Required on Close (7.0%; p = 0.028). In pairwise comparisons ([Table 2]), there were statistically significant differences between Optional Persistent and Required on Close (p = 0.016) and between Required on Close and Required on Open (p = 0.047), but not between Optional Persistent and Required on Open (p = 0.304).

Discussion

This study provides a randomized, prospective comparison of three separate types of CDS alerts in outpatients with gynecologic malignancies.

Required versus Optional Alerts

Our user response data were consistent with the conventional notion that a required alert generates more responses[10]—in this case, a fivefold increase over the optional, passive alert. Furthermore, requiring the alert later in the workflow (Required on Close), rather than earlier (Required on Open), led to a small decrease in total responses (96.1 vs. 100% of encounters). We suspect this was because Required on Close was not triggered when the user did not open the Charting or Plan tab. Despite generating the highest rate of initial deferral responses (31.7% of encounters), Required on Open ultimately elicited the highest meaningful response rate (94.8% of encounters) of the three strategies. Based on internal data on more than 20 other BPA alerts at our institution with a similar deferral button (“show me next time”), that button is consistently popular among users; thus, we deduce that users generally have a low threshold to choose a deferral option. When they instead choose “Yes” or “No,” we assume there is a good likelihood that it is a meaningful response, representing their sense of the patient's clinical prognosis.

Our data are consistent with those reported by Scheepers-Hoeks et al,[18] who compared four alert strategies in an intensive care unit, and Blecker et al,[10] who compared interruptive and passive alerts in the hospital setting; both studies demonstrated that active alerts, such as pop-ups, were more effective at generating a user response than more passive strategies. To our knowledge, however, this is the first direct comparison of conventional pop-up alerts with modern alternatives in the Epic EHR such as Required on Close (required Navigator BPA) and Optional Persistent (Storyboard BPA).

It is important to acknowledge that the escalation in alert volume associated with these increased responses may raise the risk of alert fatigue, which our study did not directly assess. Prior work has indicated that the actual time burden spent addressing interruptive alerts is small, often less than a minute per month, suggesting that alert fatigue is more related to the interruptive and potentially noncritical nature of the alerts.[4] Therefore, in alert design, informatics teams must weigh the benefits of greater response generation from interruptive alerts against the risk of greater alert fatigue.

Differences in Responses about Prognosis

The surprise question has been validated in prior studies[15] [16]; our data indicate that the answer to this question may change based on the timing and modality with which the question is asked. The proportion of meaningful responses that chose “No” (worse prognosis) was different between the three alert types, with Optional Persistent most associated with “No.” We suspect users who chose to interact with Optional Persistent were more likely motivated by a sense of the patient's advanced illness and thus more attuned to the presence of the passive sidebar alert. We also note that Optional Persistent has significantly more on-screen proximity to the Advance Care Planning banner in our EHR, where GOC documentation is most easily accessed. The increased association of “No” responses with Required on Open than with Required on Close (p = 0.047) may relate to more willingness to flag a patient as needing a GOC discussion at the beginning of the visit (rather than on closing the chart, when the patient might have already left), to an assumption of greater level of illness before full evaluation of the patient, or to other factors.

Limitations and Recommendations for Future Study

Our study compared three EHR alert strategies used at our institution but was not an exhaustive analysis of all alert strategies; further evaluation of other strategies will be important to assess the impact on patient care and end-users to ensure EHR advances are maximally beneficial without causing unnecessary clinician work or interfering with optimal patient care. Our study also did not address users' qualitative experience with each alert; an in-line mechanism for collecting user feedback (e.g., an option within the alert to click and send feedback) may yield valuable insights, as it did for Rubins et al.[19]

Alerts like these are worthwhile only if they ultimately improve the care of patients. In a separate manuscript, we plan to compare these alert types on patient-centered metrics, including documentation of GOC notes, PC referrals, and survival.

Applying Lessons Learned to Future Alerts

Our study strove to help answer the question of whether users should be required to respond to an alert, as well as whether an optional but persistent alert on the Epic Storyboard obviates the need for interruptions. Our findings show the quantitative advantage of requiring an answer (a much higher proportion of meaningful responses), and they support the common suspicion that optional, passive alerts are often ignored. Required on Open alerts had a much higher rate of initial deferral response (31.7% of encounters) than Required on Close (10.9%), suggesting that users often were not prepared to answer the question as they open the chart; this also reinforces the value of a “Show me this next time” button, which allowed users to postpone giving a meaningful answer. In fact, it may be surprising that deferral responses to Required on Open were chosen in only 31.7% of encounters, meaning that more than two-thirds of the time users were able to give a meaningful answer upon first opening the chart. And even after initial deferral, those users ultimately provided a meaningful answer the vast majority of the time (94.8% of encounters), opting for perpetual deferral in only 5.2% of encounters. Taken together, these findings may suggest that there was not a high level of alert fatigue. In addition, we suspect that the greater the level of user “buy-in” (e.g., if the alert was created at their request), the lower the risk of alert fatigue.

Selecting the optimal timing for a required alert (e.g., opening vs. closing the chart) can also be informed by our findings. Alerts on opening the chart led to a modest increase in meaningful responses but at the cost of more alerts per encounter (all of which were interruptive) and a much higher likelihood of a deferral response. While the annoyance of open-chart alerts is easy to perceive, it does not necessarily follow that close-chart alerts are less annoying, since they might be perceived as occurring too late in the workflow (e.g., after the patient has left clinic). It is also notable that, while Required on Close failed to capture responses in every encounter like Required on Open, this may be due to the Epic EHR's specific handling of this close-chart requirement, and modest changes in EHR functionality might close that gap.

Conclusion

Using the EHR to deliver CDS to promote safer, higher quality care remains an important and common goal. Our study quantifies the benefit of required alerts: a nearly fivefold increase in meaningful user responses compared with optional, passive alerts. However, the risks of alerts—especially those that are required or interruptive—were not assessed directly and need future study. Alert fatigue is a major concern in the EHR, not only as a suspected contributor to decreased safety but also given rising awareness of burnout among clinicians.

These findings may help guide health care institutions as they design CDS alerts for their own projects. A Required on Open alert may be advantageous for projects where a major goal is maximizing the number of responses, where users are more “bought-in” (thus less prone to being annoyed or fatigued), where intimate knowledge of the patient is less critical, or where user awareness before the patient leaves is critical. A Required on Close alert may be advantageous when alerting is less time-sensitive or would even benefit from being addressed later in the visit, when users may not be familiar with patients at the start of the encounter (e.g., emergency care), or when more visual persistence of the alert is beneficial. An Optional Persistent alert may be favored when the goal is a gentle reminder that is not worth interrupting workflow (e.g., lower clinical urgency or higher risk of false positivity) or when persistent visibility as the user navigates the chart is paramount.

Clinical Relevance Statement

Basic design choices in the construction of CDS alerts—required versus optional, timing, location, etc.—can have a large effect on users' responses, as shown in this study. Not only could this affect their patient care directly, but it could also affect providers and their patients indirectly via alert fatigue and burnout.

Multiple Choice Questions

1. Which alert type is best classified as passive?

   • Required on Open

   • Optional Persistent

   • Required on Close

   • None of the above

   Correct Answer: The correct answer is (b) Optional Persistent. This alert type, known in the Epic EHR as a “Storyboard” alert, remains persistently visible during an encounter. It does not, however, pop up unrequested (like Required on Open) or enforce a hard stop (like Required on Close)—interruptive features that make them less likely to be considered passive.

2. Why is alert fatigue an important consideration when designing clinical decision support (CDS) interventions?

   • Excessive alerts can cause users to ignore them.

   • Excessive alerts can frustrate users and increase burnout.

   • Alert fatigue can decrease patient safety.

   • All of the above

   Correct Answer: The correct answer is (d) All of the above. Alert fatigue poses risks to the user experience and thus patient care. CDS alerts should be targeted to help providers provide timely and safe patient care. Therefore, if alerts are ignored due to alert fatigue, patient safety may be diminished.

3. Based on this study, which feature of an alert is most likely to increase responses from users?

   • Required

   • Optional

   • Passive

   • Active

   Correct Answer: The correct answer is (a) Required. This study showed that requiring a response to an alert can increase responses fivefold, likely by interrupting the user's workflow. Because interruptions could increase risks such as alert fatigue, special care should be taken to ensure that required alerts are well targeted and highly useful to users.

   The impact of being passive versus active is less clear in an alert that is required. The Required on Close alert has more passive features, whereas Required on Open is a more typical active alert, yet both were required and led to similar increases in user response.

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and it was reviewed by the Duke Institutional Review Board.

Author Contributions

B.A.D., L.J.H., and D.J.C. conceived the study. J.B. performed the technical build within the electronic health record. R.C.M., R.S., and S.I. supported the data extraction and analyses. R.C.M. and B.A.D. did primary composition of the document, with edits by the other authors. All authors approved the manuscript.

• References

• 1 AHRQ Patient Safety Network (PSNet). Alert fatigue. Published September 7, 2019. Accessed October 20, 2023 at: https://psnet.ahrq.gov/primer/alert-fatigue
• 2 Embi PJ, Leonard AC. Evaluating alert fatigue over time to EHR-based clinical trial alerts: findings from a randomized controlled study. J Am Med Inform Assoc 2012; 19 (e1): e145-e148
• 3 Murad DA, Tsugawa Y, Elashoff DA, Baldwin KM, Bell DS. Distinct components of alert fatigue in physicians' responses to a noninterruptive clinical decision support alert. J Am Med Inform Assoc 2022; 30 (01) 64-72
• 4 Elias P, Peterson E, Wachter B, Ward C, Poon E, Navar AM. Evaluating the impact of interruptive alerts within a health system: use, response time, and cumulative time burden. Appl Clin Inform 2019; 10 (05) 909-917
• 5 Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17 (01) 36
• 6 Samal L, Wu E, Aaron S. et al. Refining clinical phenotypes to improve clinical decision support and reduce alert fatigue: a feasibility study. Appl Clin Inform 2023; 14 (03) 528-537
• 7 McCoy AB, Russo EM, Johnson KB. et al. Clinician collaboration to improve clinical decision support: the Clickbusters initiative. J Am Med Inform Assoc 2022; 29 (06) 1050-1059
• 8 Chaparro JD, Hussain C, Lee JA, Hehmeyer J, Nguyen M, Hoffman J. Reducing interruptive alert burden using quality improvement methodology. Appl Clin Inform 2020; 11 (01) 46-58
• 9 McDonald S, Lytle K, Musser RC, Shaw RJ. Transitioning a fall risk care plan interruptive alert to in-line support. J Inform Nurs 2022; 7 (03) 27-32
• 10 Blecker S, Austrian JS, Horwitz LI. et al. Interrupting providers with clinical decision support to improve care for heart failure. Int J Med Inform 2019; 131: 103956
• 11 Miller SD, Murphy Z, Gray JH. et al. Human-centered design of a clinical decision support for anemia screening in children with inflammatory bowel disease. Appl Clin Inform 2023; 14 (02) 345-353
• 12 Wright AA, Zhang B, Ray A. et al. Associations between end-of-life discussions, patient mental health, medical care near death, and caregiver bereavement adjustment. JAMA 2008; 300 (14) 1665-1673
• 13 Lopez-Acevedo M, Havrilesky LJ, Broadwater G. et al. Timing of end-of-life care discussion with performance on end-of-life quality indicators in ovarian cancer. Gynecol Oncol 2013; 130 (01) 156-161
• 14 Davidson BA, Puechl AM, Watson CH. et al. Promoting timely goals of care conversations between gynecologic cancer patients at high-risk of death and their providers. Gynecol Oncol 2022; 164 (02) 288-294
• 15 Downar J, Goldman R, Pinto R, Englesakis M, Adhikari NKJ. The “surprise question” for predicting death in seriously ill patients: a systematic review and meta-analysis. CMAJ 2017; 189 (13) E484-E493
• 16 Rauh LA, Sullivan MW, Camacho F. et al. Validation of the surprise question in gynecologic oncology: a one-question screen to promote palliative care integration and advance care planning. Gynecol Oncol 2020; 157 (03) 754-758
• 17 Foote J, Lopez-Acevedo M, Samsa G. et al. Predicting 6- and 12-month risk of mortality in patients with platinum-resistant advanced-stage ovarian cancer: prognostic model to guide palliative care referrals. Int J Gynecol Cancer 2018; 28 (02) 302-307
• 18 Scheepers-Hoeks AMJ, Grouls RJ, Neef C, Ackerman EW, Korsten EH. Physicians' responses to clinical decision support on an intensive care unit–comparison of four different alerting methods. Artif Intell Med 2013; 59 (01) 33-38
• 19 Rubins D, McCoy AB, Dutta S. et al. Real-time user feedback to support clinical decision support system improvement. Appl Clin Inform 2022; 13 (05) 1024-1032

Publication History

Received: 06 August 2023

Accepted: 16 January 2024

Accepted Manuscript online:
17 January 2024

Article published online:
13 March 2024