Increasing Completion of Daily Patient-Reported Outcomes in Psychotherapies for Late-Life Depression through User-Centered Design

Abstract

Background Treatment of depressive symptoms in older adults is a growing public health concern. Collecting patient-reported outcomes (PROs) may facilitate efficiently scaling psychotherapy for older adults but user-specific tailoring is needed to improve completion.

Objectives This study investigates (1) the effect of updating PRO collection tools for middle-aged and older adults with depressive symptoms through a user-centered design process on user completion of PRO questions, (2) what sociodemographic factors correspond with participant completion, and (3) how completion of PRO questions change during the course of a psychotherapy intervention.

Methods Analysis was conducted on 139 middle-aged and older adults with depressive symptoms from three clinical trials at the Weill Cornell ALACRITY Center. Overall response percentages to daily PRO questionnaires were compared before and after the implementation of findings from a multiphase user-centered design process. Grouped least absolute shrinkage and selection operator (LASSO) was employed to examine which baseline factors correspond with patient completion and linear regression was conducted to explore the association. Changes in daily dichotomized completion over time were analyzed with mixed-effect logistic regression.

Results After user-centered updates, there was a significantly higher (p < 0.001) percentage of completion (mean [standard deviation (SD)] percentage, 67.0 [35.6]%) than before (mean [SD] percentage, 24.9 [28.9]%). Additional years of education, age, and total annual household income greater than $25,000 were significant with completion percentage. Mixed-effects logistic regression showed that the odds of high completion increased each day (OR = 1.019 [95% CI: 1.014, 1.023; p < 0.001]).

Conclusion This study has shown that user-centered technology tailoring may be associated with increased PRO completion among middle-aged and older adults with depressive symptoms. PRO-supported psychotherapies are promising for middle-aged and older adults with depressive symptoms. Likewise, this study has demonstrated the potential benefits of employing a rigorous user-centered design process with PRO technology.

Keywords

patient-reported outcomes - usability testing - user-centered design - older adults - psychotherapy - depression

Background and Significance

Utility of Patient-Reported Outcomes in Mental Health Interventions

Patient-reported outcomes (PROs) allow for repeated measures of patients' current experiences and in their natural environment. PROs reduce the risk of recall bias and provide insight into dynamic within-subject fluctuations over time about their well-being.[1] PROs are beneficial for mental health interventions, particularly those with (mobile health) integration, as they allow for greater self-assessment and mood recognition in patients,[2] [3] as well as additional information for health professionals to better understand patient symptoms and tailor their treatment.[4] [5] [6] [7] There are several studies that incorporate PROs in mental health interventions.[8] [9] [10] [11] [12] [13] [14] [15]

Mining of PRO data has also been utilized to model individual responses to treatment, predict events, and tailor treatment.[16] [17] [18] [19] [20] [21] PRO responses were found to accurately predict the use of behavioral or dialectical behavioral therapy skills in young adults and can be used to determine optimal personalized treatment plans.[21] Technologically supported intervention strategies, including those reinforced by PROs, are widely used and tested in younger populations but not implemented equitably in older adults.[22] [23] Despite the success of digital interventions in enhancing treatment outcomes for younger adults, there is a pressing need to extend these technologically supported strategies to meet the mental health needs of the burgeoning middle-aged and older adult population.

Depression in Older Adults and Need for Improved Digital Psychotherapies

The United States experienced a 38.6% growth rate of older adults between 2010 and 2020, the fastest ever, as the population rose from 40.3 to 55.8 million,[24] and this number is projected to increase to just over 82 million by 2050.[25] Depression in older adults goes widely underdetected and undertreated,[26] [27] [28] [29] and the rapidly increasing population of older adults only exacerbates this issue. One common treatment for depression, pharmacotherapy, has been shown to be less effective in older adults than in younger populations.[30] [31] [32] Psychological changes in the aging process, comorbidities, and polypharmacy also heighten the risk of adverse drug effects in older adults.[26] [33] In addition, medication adherence can be difficult in both older adults and depressed patients, which is only amplified by the confluence of these factors.[33] [34] [35] Psychotherapies, talk therapies for treating mental health disorders, provide an efficacious alternative to pharmacotherapy treatment for older adults but have been limited by the current quantity of providers specialized for older adults and the lack of scalability in their traditional forms, as they are typically complex and community-based therapists (i.e., social workers and care managers) are not adequately trained to deliver them.[27] [36] [37] [38] [39]

The use of PROs may facilitate efficiently scaling psychotherapy for older adults by promoting self-reflection while preventing recall bias, providing more insight into health professionals, and helping to personalize treatments.[40] However, in order to do this effectively, the mechanisms for eliciting PROs and utilizing them to inform treatment must be rigorously tested and designed with the needs of older adults in mind.[41] [42]

Objectives

This study presented a unique opportunity to assess differences in PRO completion before and after tailoring the digital components of the PRO collection approach through a multiphase user-centered design process. While previous studies have stressed the importance of technological tailoring to support the health of middle-aged and older adults,[41] [42] fewer have been able to examine the effects of user-centered tailoring on outcomes such as PRO completion. We also examined what factors correspond with participant completion and how completion of PRO questions changes during the course of psychotherapy intervention. The study sample included middle-aged and older adults with depressive symptoms from three clinical trials conducted at the Weill Cornell ALACRITY Center before and after the implementation of findings from a multiphase user-centered design approach.[43] This study focused on middle-aged adults, in addition to older adults, in these psychotherapeutic interventions because the processes and conditions related to late-life mood disorders start in midlife. For example, vascular changes and amyloid deposition start in midlife and may lead to mood disorders.[44] [45] Likewise, age may be an unreliable index of aging especially in minority populations.

Methods

Study Populations

Participants took part in three clinical trials for depression conducted at the Weill Cornell ALACRITY Center, of which the interventions all follow the “Engage” framework to target behavioral activation and social reward exposure.[46] [47] The common inclusion criteria for the three interventions were participants middle-aged and older (>50 years old) diagnosed with major depressive disorder who had the capacity to provide informed consent. The common exclusion criteria for all three intervention trials were active suicidal ideation, the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) diagnosis other than unipolar depression or generalized anxiety disorder, or severe or life-threatening medical illness.

The three studies tested the effectiveness of three interventions namely, REDS, a community care intervention for senior center clients, RELIEF, a psychotherapy for primary care patients with chronic pain and depression symptoms, and PROTECT, a problem-solving psychotherapy for elder abuse victims. Clinical outcomes have been published separately.[48] [49] These interventions included 9-weekly psychotherapeutic sessions delivered by trained, licensed clinical social workers.

Study Phases and Interventions

The three interventions were evaluated in two Phases (Phase 1 and Phase 2). The psychotherapeutic components of the interventions remained similar in the two phases. Both phases also involved PROs reported via mHealth, and passive sensing data (e.g., activity, sleep, etc.) were collected in both phases and summarized daily. In each phase, smartphone ownership was not a prerequisite, and devices and internet service were provided to those who did not have it and were interested in participating.

The digital components of the intervention were modified between phases. Initial modifications to the technology platforms were first made based on trial findings and user feedback from Phase 1 (see [Table 1] and “Data Collection” section). We then conducted heuristic evaluations (by two usability experts) and a targeted usability pilot on the updated technology platforms as part of the user-centered design process. Specifically, PRO utilization was compared before (Phase 1) and after (Phase 2) the adaption based on feedback from Phase 1 and the pilot usability study. The pilot usability study, published separately, was completed with a group of 15 middle-aged and older adults from a senior center to improve the suitability of the digital PRO collection elements for middle-aged and older adults.[43] Of note, this was a convenience sample of participants who were not necessarily experiencing depression, although 2/15 (13.3%) screened positive for depression. Given recruitment from a single senior center, participant race (11 Asian [73.3%], 2 White [13.3%], 1 Black [6.7%], 1 Other [6.7%]) and age (50–59: 1 [7.1%], 60–69: 10 [71.4%], 70–79: 2 [14.2%], 80 + : 1 [7.1%]) were also skewed. See Benda et al, 2020, for more details.[43] Participants completed initial interviews, a 1-week pilot period with the digital intervention, and final interviews to discuss their thoughts on the intervention. Our updates focused on changes to study text messaging format and technology setup instructions, as the data collection platforms involved commercial technologies with limited options for customization. Various changes were made prior to Phase 2 of interventions, including adjustments to text messages with scales to avoid comprehension errors, providing an “out-of-office” message, breaking long study initiation messages into multiple segments, improving device settings to fit their physiological and cognitive needs (e.g., font size), and a redesign of the technology instructions received by participants.[43] In Phase 2, behavioral economics-based gamification was implemented to promote adherence to designated behavioral activation tasks between two consecutive sessions of psychotherapy. Analysis of the effects of gamification on adherence to psychotherapeutic homework and depression outcomes have been submitted for publication separately.[50]

Table 1 Changes between phases of the studies

Component/activity	Phase I	Phase II
PRO management application	HealthRhythms	WayToHealth
Passive sending device	Smartphone	Smartwatch
PRO collection approach	Through HealthRhythms app	Through SMS message
Gamification	No	Yes
Prior usability testing	None	15-participant pilot which informed enhancements[43]

Abbreviation: PRO, patient-reported outcome.

Data Collection

At the start of the 9-week intervention, self-reported demographic, clinical, and financial characteristics were collected, including reported depression, measured by the Montgomery–Åsberg Depression Rating Scale (MADRS).[51] The technology platform used to facilitate communication with participants and store activity data was developed by the company HealthRhythms©[52] in Phase 1, and in Phase 2 it was developed by WayToHealth©[53] (see [Fig. 1]). The HealthRhythms platform recorded, via its mHealth app, daily responses to five PRO questions on anhedonia, sadness, stress, and pain, and their adherence to psychotherapeutic homework. In Phase 2, the WayToHealth platform recorded participants' answers to PRO questions via text message about their current level of anhedonia, sadness, stress, and pain in the morning and their adherence to psychotherapeutic homework in the evening.

Fig. 1 Format and structure of technology and psychotherapy in both phases. PRO, patient-reported outcome.

Statistical Analysis

Patient-level completion was defined by the percentage of completed responses to the five PRO questions on anhedonia, sadness, stress, and pain and adherence to psychotherapeutic homework. Overall completion of PRO questions for each patient was defined by averaging daily patient-level completion percentages over the number of days in the study. The days between the consent date and the last clinical measurement at the end of the study interventions were included in the analysis. Differences in demographic and baseline clinical characteristics, as well as overall completion, were assessed by phase using Wilcoxon rank-sum tests for continuous variables and either Fisher's exact test or chi-square test, as appropriate, for categorical variables.

To examine which factors are significant in determining patient completion, missing values were first imputed using the missForest algorithm.[54] [55] Age is known to be associated with engagement with technology,[56] [57] and was significantly correlated with completion in these data. To assess the nonlinear effects of age, we performed leave-one-out cross-validation (LOOCV) to determine the optimal degrees of freedom for a restricted natural cubic spline, using the R² metric as our measure of goodness of fit. Grouped least absolute shrinkage and selection operator (LASSO) was employed, which incorporated the natural spline of age without penalization, to identify key demographic and baseline clinical factors associated with overall patient completion. Variables on gender, race, ethnicity, marital status, religion, years of education completed, financial status, binary total annual household income greater than or less than $25,000, Medicare, Medicaid, and food stamp use, and baseline MADRS, along with the natural spline of age were included as predictors in the grouped LASSO model. Subsequently, linear regression was implemented with the LASSO-selected variables to quantify the association between these variables and completion percentage.

Patient-level daily PRO completion was dichotomized as high (>50% of PRO questions answered or three or more PROs answered) or low completion (<50% of PRO questions answered or two or fewer PROs answered) due to the discrete nature of this outcome. Changes in daily dichotomized completion over time were analyzed with mixed-effect logistic regression with the fixed effect of days from the start of the study and the LASSO-selected variables and a patient-level random intercept. Higher-order polynomials of days from the start were explored but not chosen based on the Bayesian Information Criterion. To test for the overall effect of the spline of age on patient completion, we conducted a hypothesis test that considers all degrees of freedom simultaneously. We ran an analysis of variance that compares the full model including spline terms with a reduced model excluding spline terms. Statistical analysis was performed using R Statistical Software (v4.3.0, R Foundation). For all analyses, two-sided p-values <0.05 were defined as statistically significant.

Results

Participant Characteristics

A total of 127 participants were enrolled in the Phase 1, the preusability study, and 78 in the Phase 2, the postusability study. After excluding participants who did not start therapy, who were not in the treatment arm, and who did not enroll in device use, a total of 139 participants (mean [SD] age, 70.0 [8.33] years; 37 middle-aged (26.6%), 102 (73.4%) older adult; 117 [84.2%] female), 78 participants from the Phase 1 and 61 participants from the Phase 2, were included in the analysis ([Table 2]). Participants identified with the following racial groups: 85 (61.2%) White, 47 (33.8%) Black, and 7 (5.0%) Other. There were no significant differences in demographic and baseline clinical characteristics between phases, besides the intervention subtype (Phase 1: 30.8% PROTECT, 28.2% REDS, 41.0% RELIEF; Phase 2: 57.4% PROTECT, 27.9% REDS, 14.8% RELIEF; p < 0.001) and religion (Phase 1: 38.2% Christian, 26.3% Jewish, 35.5% Other; Phase 2: 42.6% Christian, 9.3% Jewish, 48.1% Other; p < 0.0459).

Table 2 Demographic and baseline clinical characteristics of the overall sample and subsamples by phase

	Phase 1 (N = 78)	Phase 2 (N = 61)	p-Value	Overall (N = 139)
Study length
Mean (SD)	95.7 (34.9)	85.3 (27.4)	0.115[a]	91.1 (32.2)
Median [min, max]	91.0 [12.0, 230]	90.0 [13.0, 145]	–	91.0 [12.0, 230]
Study
PROTECT	24 (30.8%)	35 (57.4%)	0.001[b]	59 (42.4%)
REDS	22 (28.2%)	17 (27.9%)	–	39 (28.1%)
RELIEF	32 (41.0%)	9 (14.8%)	–	41 (29.5%)
Completion (%)
Mean (SD)	24.9 (28.9)	67.0 (35.6)	<0.001[a]	43.4 (38.2)
Median [min, max]	12.7 [0, 98.9]	88.0 [0, 99.5]	–	38.7 [0, 99.5]
Age
Mean (SD)	70.7 (8.52)	69.1 (8.05)	0.376[a]	70.0 (8.33)
Median [min, max]	69.1 [52.7, 93.9]	70.0 [51.0, 89.0]	–	70.0 [51.0, 93.9]
Missing	0 (0%)	3 (4.9%)	–	3 (2.2%)
Gender
Female	65 (83.3%)	52 (85.2%)	0.942[b]	117 (84.2%)
Male	13 (16.7%)	9 (14.8%)	–	22 (15.8%)
Marital status
Divorced/separated	22 (28.2%)	20 (32.8%)	0.16[b]	42 (30.2%)
Married	10 (12.8%)	14 (23.0%)	–	24 (17.3%)
Single	22 (28.2%)	17 (27.9%)	–	39 (28.1%)
Widowed	24 (30.8%)	10 (16.4%)	–	34 (24.5%)
Ethnicity
Hispanic	12 (15.4%)	4 (6.6%)	0.177[b]	16 (11.5%)
Non-Hispanic	66 (84.6%)	57 (93.4%)	–	123 (88.5%)
Race
Black	20 (25.6%)	27 (44.3%)	0.0708[c]	47 (33.8%)
White	53 (67.9%)	32 (52.5%)	–	85 (61.2%)
Other	5 (6.4%)	2 (3.3%)	–	7 (5.0%)
Religion
Christianity	29 (37.2%)	23 (37.7%)	0.0459[b]	52 (37.4%)
Jewish	20 (25.6%)	5 (8.2%)	–	25 (18.0%)
Other	27 (34.6%)	26 (42.6%)	–	53 (38.1%)
Missing	2 (2.6%)	7 (11.5%)	–	9 (6.5%)
Years of education
Mean (SD)	14.9 (2.96)	15.1 (3.05)	0.913[a]	15.0 (2.99)
Median [min, max]	16.0 [5.00, 20.0]	16.0 [5.00, 22.0]	–	16.0 [5.00, 22.0]
Missing	0 (0%)	1 (1.6%)	–	1 (0.7%)
Financial status
Cannot make ends meet	15 (19.2%)	14 (23.0%)	0.456[b]	29 (20.9%)
Have just enough	39 (50.0%)	35 (57.4%)	–	74 (53.2%)
Are comfortable	22 (28.2%)	12 (19.7%)	–	34 (24.5%)
Missing	2 (2.6%)	0 (0%)	–	2 (1.4%)
Total household income
< $25,000	41 (52.6%)	29 (47.5%)	0.677[b]	70 (50.4%)
≥ $25,000	37 (47.4%)	32 (52.5%)	–	69 (49.6%)
On Medicaid
Yes	17 (21.8%)	19 (31.1%)	0.313[b]	36 (25.9%)
No	60 (76.9%)	42 (68.9%)	–	102 (73.4%)
Missing	1 (1.3%)	0 (0%)	–	1 (0.7%)
On Medicare
Yes	62 (79.5%)	43 (70.5%)	0.242[b]	105 (75.5%)
No	15 (19.2%)	18 (29.5%)	–	33 (23.7%)
Missing	1 (1.3%)	0 (0%)	–	1 (0.7%)
Receiving food stamps or benefits from SNAP
Yes	17 (21.8%)	15 (24.6%)	0.919[b]	32 (23.0%)
No	59 (75.6%)	46 (75.4%)	–	105 (75.5%)
Missing	2 (2.6%)	0 (0%)	–	2 (1.4%)
Baseline MADRS
Mean (SD)	21.0 (6.64)	21.9 (7.38)	0.648[a]	21.4 (6.96)
Median [min, max]	22.0 [9.00, 35.0]	22.0 [8.00, 45.0]	–	22.0 [8.00, 45.0]
Missing	0 (0%)	2 (3.3%)	–	2 (1.4%)

Abbreviations: MADRS, Montgomery–Åsberg Rating Scale; SD, standard deviation; SNAP, Supplemental Nutrition Assistance Program.

^a Wilcoxon rank-sum test.

^b Chi-square test.

^c Fisher's exact test.

Changes in Patient-Reported Outcome Completion Following User-Centered Technological Tailoring

After updates to the technological elements had been made based on our user-centered design steps (Phase 2), there was a significantly higher (p < 0.001) overall percentage of completion (mean [SD] percentage, 67.0 [35.6]%) than in Phase 1 (mean [SD] percentage, 24.9 [28.9]%).

Effect of Patient Characteristics on Patient-Reported Outcome Completion

LOOCV determined a natural spline with 3 degrees of freedom to have the lowest R² and was implemented in the grouped LASSO. The model selected years of education, total annual household income greater than or less than $25,000, and baseline MADRS with nonlinear effects of age not used in the selection criteria for LASSO. The subsequent model with the three selected variables and age indicated that years of education (p = 0.003), total annual household income greater than or less than $25,000 (p = 0.005), and the nonlinear effect of age (p = 0.04) were significantly associated with completion but not baseline MADRS (p = 0.28; [Table 3], R² = 0.226). Overall, the completion percentage decreased as the age increased to 65 years, plateauing between 65 and 75 years, and decreasing further with age greater than 75 years ([Fig. 2]). For each additional year of education, the expected completion percentage increased by 3.06% (95% CI: 1.05, 5.08). Participants with a total annual household income of $25,000 or more were expected to have an average of 17.40% (95% CI: 5.43, 29.36) higher completion than those with less than $25,000 of annual household income.

Table 3 Association between patient demographic and baseline clinical variables and patient-reported outcome completion for variables selected using grouped least absolute shrinkage and selection operator

Variable	Effect on completion percentage (β, 95% CI)	p-Value
Additional year of education	3.06 (1.05, 5.08)	0.003
Total annual household income (≥$25,000 to <$25,000)	17.40 (5.43, 29.36)	0.005
Point increase in baseline MADRS	−0.47 (−1.32, 0.38)	0.28
Natural spline of age		0.003

Abbreviation: MADRS, Montgomery–Åsberg Depression Rating Scale.

Fig. 2 Nonlinear association of age (natural spline with 3 degrees of freedom) and adjusted PRO completion rate (adjusted for education, income, and baseline depression severity). PRO, patient-reported outcome.

Changes in Patient-Level Completion Over Time

Mixed-effects logistic regression using a subject-specific random intercept, with a fixed effect for days from start showed that the odds of high completion increased each day (OR = 1.019 [95% CI: 1.014, 1.023; p < 0.001]; [Fig. 3] and [Table 4]). The mixed model also included fixed effects for years of education, total annual household income, baseline MADRS, and nonlinear age (see “Effect of Patient Characteristics on Patient-Reported Outcome Completion” section). For each additional year of education, the odds of high completion on any given day increased by 48% (OR = 1.481 [95% CI: 1.187, 1.849; p < 0.001]). Less than $25,000 of annual household income decreases the odds of high completion on any given day by 85% (OR = 0.1508 [95% CI: 0.042, 0.545; p = 0.004]). For each additional point of MADRS at baseline, the odds of high completion decrease on any given day by 11% (OR = 0.892 [95% CI: 0.810, 0.983; p = 0.02]). There was also a significant nonlinear effect of age (p < 0.001).

Fig. 3 Mixed-effect logistic model estimated odds ratio of high completion as measured by the days from start, with a 95% CI.

Table 4 Multivariable mixed-effect logistic model result for high completion

Variable	Odds ratio (95% CI)	p-Value
Additional day in study	1.019 (1.014, 1.023)	<0.001
Additional year of education	1.481 (1.187, 1.849)	<0.001
Total annual household income (<$25,000/≥ $25,000)	0.1508 (0.042, 0.545)	0.004
Point increase in baseline MADRS	0.892 (0.810, 0.983)	0.02
Natural spline of age		<0.001

Abbreviation: MADRS, Montgomery–Åsberg Depression Rating Scale.

Discussion

This study presented a unique opportunity to assess differences in PRO utilization by middle-aged and older adults before and after a multiphase user-centered design process informed enhancements to the digital elements of the PRO collection. Our analysis found that middle-aged and older adults with depression and undergoing psychotherapy had significantly higher responses (42.1%) to PRO questions on mood, anhedonia, stress, pain, and homework completion during the course of the intervention in Phase 2 of the interventions after user-centered design findings had been implemented compared to Phase 1. While this result is not necessarily surprising, it provides evidence to support previous literature that suggests that usability testing and subsequent tailoring PRO technology to middle-aged and older adults would be beneficial in improving completion.[41] [42]

Older age, fewer years of education, and lower annual income were all associated with decreased completion in this study, which is consistent with previous literature on technology use in middle-aged and older adults.[56] [57] [58] Additional usability testing is needed in these subpopulations of middle-aged and older adults to improve their engagement with technology. Contrary to prior findings of PRO completion in participants with depressive symptoms decreases over time,[10] [12] the likelihood of high levels of completion was found to increase over the course of the study. This contrary finding could be due to the fact that as patients improve on their depressive symptoms through behavioral activation during the course of the intervention,[59] [60] increase activity, including possibly engaging more with the technological aspects of the intervention. However, this needs further investigation.

This study was not without limitations. First, given that user-centered tailoring involved multiple elements from different approaches, we cannot independently assess the effects of the various updates. Second, those who received the user-centered design-enhanced version versus those who did not were not randomized and therefore the effects cannot be interpreted as causal effects. Third, the usability pilot study was conducted in a single community center, and therefore not representational of the middle-aged and older adult population in New York City. Further usability testing with a more racially and ethnically representative sample would be beneficial to broaden the scope for whom the technology should be tailored. In addition, these clinical trials focused on specific subsets of the general population of middle-aged and older adults with depression (senior center clients, primary care patients with chronic pain, and victims of elder abuse), and therefore are not representative of the middle-aged and older adult population with depressive symptoms altogether. Finally, there was no measure of digital literacy in this study, which could be a factor in completion, and therefore should be investigated further.

This study has shown that enhancements to digital PRO collection approaches informed by a user-centered design process may be associated with an increased response rate to PRO questionnaires among middle-aged and older adults with depressive symptoms. For mental health interventions, these improvements allow for greater self-assessment and mood recognition in patients.[2] [3] In turn, PRO-supported psychotherapies with community-based therapists are promising and feasible avenues for wider scalability and increased access to mental health care for middle-aged and older adults with depressive symptoms. Agnostic of the clinical domain, PROs provide more information for health professionals to gain a better understanding of the patient's symptoms in their natural environment and in real-time and tailor their treatment. Based on PRO response and patterns of daily symptoms, additional therapeutic interventions can be pushed to patients in a digital format (e.g., nudges for intervention adherence, therapeutic task completion) that might improve the effectiveness of the intervention. This study also provides the feasibility and importance of collecting PRO data as the daily summaries and trajectories can be integrated into the intervention for patient monitoring and adjusting weekly therapeutic elements. Increasing PRO response is a particular feat among those with depressive symptoms given coincident apathy and lack of motivation. We anticipate these results may translate to other clinical settings but this would require further evaluation. Likewise, this study has demonstrated the potential benefits of rigorous testing in designing interventions for the specific population that will use them. Future user-centered assessment is needed in the subpopulations that had low completion, including the oldest of older adults, middle-aged and older adults with fewer years of education, and middle-aged and older adults with lower annual income.

Conclusion

This study has shown that among middle-aged and older adults with depressive symptoms, user-centered design tailoring may be associated with an increased response rate to PRO questionnaires in psychotherapy. These results suggest that employing a user-centered design process to tailor PRO technology to middle-aged and older adults would be beneficial in improving completion and the possibility of wider scalability and increased access to mental health care for middle-aged and older adults with depressive symptoms. The factors of older age, fewer years of education, and lower annual income are all associated with decreased completion in this study, so further testing and tailoring could benefit these subpopulations.

Clinical Relevance Statement

Employing a user-centered design processing improves completion rates with PRO elicitation tools, specifically those used to support psychotherapy. These PRO responses provide daily symptom trajectories that be utilized to deliver interventions in a digital format (e.g., nudges for intervention adherence, therapeutic task completion) that have the potential to improve the effectiveness of psychotherapies. This study also provides the feasibility and importance of collecting PRO data as the daily summaries and trajectories can be integrated into the intervention for patient monitoring and adjusting weekly therapeutic elements.

Multiple-Choice Questions

1. In considering Ecological Momentary Assessment, which of the following benefits may patients experience?

   • Measures in the patient's natural environment

   • Demonstrates fluctuations over time

   • Reduces the risk of recall bias

   • All of the above.

   Correct Answer: The correct answer is option d. All of these are benefits of Ecological Momentary Assessments.

2. Which of the following may be a benefit of design optimization through usability testing?

   • Ensuring user-centered design

   • Improving participant completion

   • Increasing data quality and accuracy

   • All of the above.

   Correct Answer: The correct answer is option d. All of these are potential benefits of design optimization through usability testing.

3. What important pitfalls may occur in usability testing?

   • More information can be obtained from a population of interest.

   • Certain groups may benefit more if they are well-represented in the usability study sample.

   • User-centered design optimization can occur as a result.

   • There are no pitfalls to usability testing.

   Correct Answer: The correct answer is option b. A potential pitfall of usability testing is that certain groups may benefit more if they are well-represented in the usability study sample. Likewise, certain groups may be disadvantaged if they are not well-represented in the usability study sample.

Conflict of Interest

None declared.

Protection of Human Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was reviewed by the Weill Cornell Medicine Institutional Review Board.

• References

• 1 Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol 2008; 4: 1-32
• 2 Folkersma W, Veerman V, Ornée DA, Oldehinkel AJ, Alma MA, Bastiaansen JA. Patients' experience of an ecological momentary intervention involving self-monitoring and personalized feedback for depression. Internet Interv 2021; 26: 100436
• 3 Saunders KE, Bilderbeck AC, Panchal P, Atkinson LZ, Geddes JR, Goodwin GM. Experiences of remote mood and activity monitoring in bipolar disorder: a qualitative study. Eur Psychiatry 2017; 41: 115-121
• 4 Epskamp S, van Borkulo CD, van der Veen DC. et al. Personalized network modeling in psychopathology: the importance of contemporaneous and temporal connections. Clin Psychol Sci 2018; 6 (03) 416-427
• 5 Mumma GH, Marshall AJ, Mauer C. Person-specific validation and testing of functional relations in cognitive-behavioural case formulation: guidelines and options. Clin Psychol Psychother 2018; 25 (05) 672-691
• 6 Piot M, Mestdagh M, Riese H. et al. Practitioner and researcher perspectives on the utility of ecological momentary assessment in mental health care: a survey study. Internet Interv 2022; 30: 100575
• 7 Wenze SJ, Miller IW. Use of ecological momentary assessment in mood disorders research. Clin Psychol Rev 2010; 30 (06) 794-804
• 8 Kauer SD, Reid SC, Crooke AH. et al. Self-monitoring using mobile phones in the early stages of adolescent depression: randomized controlled trial. J Med Internet Res 2012; 14 (03) e67
• 9 Bastiaansen JA, Ornée DA, Meurs M, Oldehinkel AJ. An evaluation of the efficacy of two add-on ecological momentary intervention modules for depression in a pragmatic randomized controlled trial (ZELF-i). Psychol Med 2020; 52 (13) 1-10
• 10 Burns MN, Begale M, Duffecy J. et al. Harnessing context sensing to develop a mobile intervention for depression. J Med Internet Res 2011; 13 (03) e55
• 11 Hollands L, Lambert J, Price L, Powell D, Greaves C. Ecological momentary assessment of mood and physical activity in people with depression. J Affect Disord 2020; 271: 293-299
• 12 Hung S, Li M-S, Chen Y-L, Chiang JH, Chen YY, Hung GC. Smartphone-based ecological momentary assessment for Chinese patients with depression: an exploratory study in Taiwan. Asian J Psychiatr 2016; 23: 131-136
• 13 Schoevers RA, van Borkulo CD, Lamers F. et al. Affect fluctuations examined with ecological momentary assessment in patients with current or remitted depression and anxiety disorders. Psychol Med 2021; 51 (11) 1906-1915
• 14 Targum SD, Sauder C, Evans M, Saber JN, Harvey PD. Ecological momentary assessment as a measurement tool in depression trials. J Psychiatr Res 2021; 136: 256-264
• 15 Yim SJ, Lui LMW, Lee Y. et al. The utility of smartphone-based, ecological momentary assessment for depressive symptoms. J Affect Disord 2020; 274: 602-609
• 16 David SJ, Marshall AJ, Evanovich EK, Mumma GH. Intraindividual Dynamic Network Analysis - implications for clinical assessment. J Psychopathol Behav Assess 2018; 40 (02) 235-248
• 17 Fisher AJ, Bosley HG, Fernandez KC. et al. Open trial of a personalized modular treatment for mood and anxiety. Behav Res Ther 2019; 116: 69-79
• 18 Fisher AJ, Boswell JF. Enhancing the personalization of psychotherapy with dynamic assessment and modeling. Assessment 2016; 23 (04) 496-506
• 19 Piccirillo ML, Beck ED, Rodebaugh TL. A clinician's primer for idiographic research: considerations and recommendations. Behav Ther 2019; 50 (05) 938-951
• 20 Rubel JA, Fisher AJ, Husen K, Lutz W. Translating person-specific network models into personalized treatments: development and demonstration of the Dynamic Assessment Treatment Algorithm for Individual Networks (DATA-IN). Psychother Psychosom 2018; 87 (04) 249-251
• 21 Webb CA, Forgeard M, Israel ES, Lovell-Smith N, Beard C, Björgvinsson T. Personalized prescriptions of therapeutic skills from patient characteristics: an ecological momentary assessment approach. J Consult Clin Psychol 2022; 90 (01) 51-60
• 22 Peng R, Guo Y, Zhang C. et al. Internet-delivered psychological interventions for older adults with depression: a scoping review. Geriatr Nurs 2024; 55: 97-104
• 23 Seifert A, Reinwand DA, Schlomann A. Designing and using digital mental health interventions for older adults: being aware of digital inequality. Front Psychiatry 2019; 10: 568
• 24 Caplan Z. US Older Population Grew From 2010 to 2020 at Fastest Rate Since 1880 to 1890. United States Census Bureau; 2023. . Accessed June 14, 2024 at: https://www.census.gov/library/stories/2023/05/2020-census-united-states-older-population-grew.html
• 25 United States Census Bureau. 2023 National Population Projections Tables: Main Series. 2023
• 26 Kok RM, Reynolds III CF. Management of depression in older adults: a review. JAMA 2017; 317 (20) 2114-2122
• 27 Lavingia R, Jones K, Asghar-Ali AA. A systematic review of barriers faced by older adults in seeking and accessing mental health care. J Psychiatr Pract 2020; 26 (05) 367-382
• 28 Mitchell AJ, Rao S, Vaze A. Do primary care physicians have particular difficulty identifying late-life depression? A meta-analysis stratified by age. Psychother Psychosom 2010; 79 (05) 285-294
• 29 Unützer J. Clinical practice. Late-life depression. N Engl J Med 2007; 357 (22) 2269-2276
• 30 Alexopoulos GS. “The depression-executive dysfunction syndrome of late life”: a specific target for D3 agonists?. Am J Geriatr Psychiatry 2001; 9 (01) 22-29
• 31 Alexopoulos GS, Meyers BS, Young RC. et al. Executive dysfunction and long-term outcomes of geriatric depression. Arch Gen Psychiatry 2000; 57 (03) 285-290
• 32 Kalayam B, Alexopoulos GS. Prefrontal dysfunction and treatment response in geriatric depression. Arch Gen Psychiatry 1999; 56 (08) 713-718
• 33 Cahoon CG. Depression in older adults. Am J Nurs 2012; 112 (11) 22-30 , quiz 31
• 34 Campbell NL, Boustani MA, Skopelja EN, Gao S, Unverzagt FW, Murray MD. Medication adherence in older adults with cognitive impairment: a systematic evidence-based review. Am J Geriatr Pharmacother 2012; 10 (03) 165-177
• 35 Insel K, Morrow D, Brewer B, Figueredo A. Executive function, working memory, and medication adherence among older adults. J Gerontol B Psychol Sci Soc Sci 2006; 61 (02) 102-107
• 36 Alexopoulos GS, Raue PJ, Kiosses DN. et al. Problem-solving therapy and supportive therapy in older adults with major depression and executive dysfunction: effect on disability. Arch Gen Psychiatry 2011; 68 (01) 33-41
• 37 Bartels SJ, Naslund JA. The underside of the silver tsunami–older adults and mental health care. N Engl J Med 2013; 368 (06) 493-496
• 38 Eden J, Maslow K, Le M. et al. The mental health and substance use workforce for older adults: in whose hands. Committee in the Mental Health Workforce for Geriatric Populations; Board on Health Care Services; Institute of Medicine; 2012
• 39 Institute of Medicine Committee on the Future Health Care Workforce for Older Americans. Retooling for an Aging America: Building the Health Care Workforce. Washington (DC): National Academies Press (US); 2008.
• 40 Moore RC, Depp CA, Wetherell JL, Lenze EJ. Ecological momentary assessment versus standard assessment instruments for measuring mindfulness, depressed mood, and anxiety among older adults. J Psychiatr Res 2016; 75: 116-123
• 41 Price M, Yuen EK, Goetter EM. et al. mHealth: a mechanism to deliver more accessible, more effective mental health care. Clin Psychol Psychother 2014; 21 (05) 427-436
• 42 Fortuna KL, Torous J, Depp CA. et al. A future research agenda for digital geriatric mental healthcare. Am J Geriatr Psychiatry 2019; 27 (11) 1277-1285
• 43 Benda NC, Alexopoulos GS, Marino P, Sirey JA, Kiosses D, Ancker JS. The age limit does not exist: a pilot usability assessment of a SMS-messaging and smartwatch-based intervention for older adults with depression. AMIA Annu Symp Proc 2021; 2020: 213-222
• 44 Alexopoulos GS, Meyers BS, Young RC, Campbell S, Silbersweig D, Charlson M. ‘Vascular depression’ hypothesis. Arch Gen Psychiatry 1997; 54 (10) 915-922
• 45 Taylor WD, Aizenstein HJ, Alexopoulos GS. The vascular depression hypothesis: mechanisms linking vascular disease with depression. Mol Psychiatry 2013; 18 (09) 963-974
• 46 Alexopoulos GS, Arean P. A model for streamlining psychotherapy in the RDoC era: the example of ‘Engage’. Mol Psychiatry 2014; 19 (01) 14-19
• 47 Alexopoulos GS, Raue PJ, Banerjee S. et al. Comparing the streamlined psychotherapy “Engage” with problem-solving therapy in late-life major depression. A randomized clinical trial. Mol Psychiatry 2021; 26 (09) 5180-5189
• 48 Sirey JA, Berman J, Salamone A. et al. Feasibility of integrating mental health screening and services into routine elder abuse practice to improve client outcomes. J Elder Abuse Negl 2015; 27 (03) 254-269
• 49 Sirey JA, Solomonov N, Guillod A. et al. PROTECT: a novel psychotherapy for late-life depression in elder abuse victims. Int Psychogeriatr 2021; 33 (05) 521-525
• 50 Qiu Y, Carter E, Benda NC. et al. Gamification via mHealth to Improve Adherence to Psychotherapy and Clinical Outcomes in Depressed Older Adults. Submitted 2024.
• 51 Montgomery SA, Åsberg M. A new depression scale designed to be sensitive to change. Br J Psychiatry 1979; 134: 382-389
• 52 Gilbert PM. HealthRhythms.
• 53 Volpp K, Asch D. Way to Health. Center for Health Care Transformation and Innovation and Center for Health Incentives and Behavioral Economics; 2009
• 54 Stekhoven DJ. missForest: Nonparametric Missing Value Imputation using Random Forest. 1.5 ed. 2022
• 55 Stekhoven DJ, Bühlmann P. MissForest–non-parametric missing value imputation for mixed-type data. Bioinformatics 2012; 28 (01) 112-118
• 56 Gell NM, Rosenberg DE, Demiris G, LaCroix AZ, Patel KV. Patterns of technology use among older adults with and without disabilities. Gerontologist 2015; 55 (03) 412-421
• 57 Drazich BF, Li Q, Perrin NA. et al. The relationship between older adults' technology use, in-person engagement, and pandemic-related mental health. Aging Ment Health 2023; 27 (01) 156-165
• 58 Choi NG, Dinitto DM. Internet use among older adults: association with health needs, psychological capital, and social capital. J Med Internet Res 2013; 15 (05) e97
• 59 Alexopoulos GS, O'Neil R, Banerjee S. et al. “Engage” therapy: prediction of change of late-life major depression. J Affect Disord 2017; 221: 192-197
• 60 Raue PJ, Sirey JA, Dawson A, Berman J, Bruce ML. Lay-delivered behavioral activation for depressed senior center clients: pilot RCT. Int J Geriatr Psychiatry 2019; 34 (11) 1715-1723

Publication History

Received: 15 March 2024

Accepted: 20 August 2024

Article published online:
20 November 2024