Visual Evidence: Increasing Usability of Systematic Reviews in Health Systems Guidelines Development

Abstract

Background Integration of evidence from systematic reviews is an essential step in the development of clinical guidelines. The current practice for reporting uses a static structure that does not allow for dynamic investigation. A need exists for an alternate reporting modality to facilitate dynamic visualization of results to match different end-users' queries.

Objectives We developed a dynamic visualization of data from a systematic review using the commercial product Tableau and assessed its potential to permit customized inquiries.

Methods Data were selected and extracted from a previously completed systematic review. The resulting dataset was then used to develop an interactive, web-based report designed for use by a guidelines development committee.

Results A novel example of combining existing reporting standards for systematic review data and modern reporting tools was developed to investigate potential benefits of a dynamic report. Demonstrations of the report to clinicians sitting on previous and future guideline committees received positive feedback for its potential benefit in guidelines development. The report received a runner-up award during the design challenge at the 2018 Workshop on Visual Analytics in Health Care.

Conclusion The use of interactive, accessible data may increase the use of systematic reviews and aid decision makers in developing evidence-based practice changes.

Keywords

data visualization - clinical informatics - clinical practice guidelines - systematic reviews - data access - integration and analysis - evidence-based medicine and nursing - guidelines and protocols - evaluation

Background and Significance

Leveraging research evidence to inform guidelines and clinical practice is essential to improving quality and efficiency of patient care. Systematic reviews are rigorous, peer-reviewed sources of evidence intended to support the translation from bench to bedside, but the data often are complex and presented in dense, static formats. The fixed structure of inquiry and extensive text-based reports may hinder uptake of these data by persons tasked with developing evidence-based guidelines.[1] [2] [3] [4] [5] [6] Even in well-written reviews, the structure is bound to the original analysis which may make comparisons across groups more difficult. Alternate reporting methods exist that allow users to interact with the data outside the original structure of the report and informed by the local contexts of their health systems, but these have rarely been incorporated into existing reviews. Harnessing these alternate tools to complement the large volume of data in systematic reviews may increase accessibility and usability of research evidence and facilitate adoption of evidence-based health care in guidelines.

The Agency for Healthcare Research and Quality (AHRQ) asked established Evidence-Based Practice Centers (EPCs) to conduct pilot projects to investigate potential solutions. During this process, the Pacific Northwest EPC performed a needs assessment for alternative reporting and assessed numerous existing software solutions. Tableau[7] was chosen for this aspect of the project due to its ease of use, familiar Microsoft Excel-based data structure (Microsoft Office 2016) robust customer service and community support, and widespread adoption by many health systems.[8]

Objectives

The main objective of this project was to develop and evaluate a prototype report using Tableau. This prototype allowed for a demonstration of the software's visualization capabilities, the feasibility of creating similar reports in the future, and the overall value to stakeholders involved in the development of clinical guidelines.

Methods

Data were extracted from the previously completed report on noninvasive nonpharmacological treatment for chronic pain.[9] Information for all studies was extracted from the forest plots and the summaries for each study in the report. As the original data were not available in an accessible format, all data used in the project were manually extracted and stored in a spreadsheet document employing a relational structure. A subset of the available data was used to allow for rapid development and simple, representative interaction. Data were extracted only for measures which were summarized using a standardized mean difference (SMD) with a single-intervention modality. The data extracted included four types of pain (chronic low back pain, chronic neck pain, osteoarthritis of the hip, and osteoarthritis of the knee), eight intervention categories (acupuncture, exercise, massage, spinal manipulation, mind–body practices, mindfulness-based stress reduction, physical modalities, and psychological therapies), two outcome categories (effect on pain and effect on function), three terms of follow-up (short, intermediate, and long). These data were extracted for the individual studies, as well as the pooled estimates, for each group.

The prototype was developed using Tableau Desktop by an informatics researcher with minimal previous experience with Tableau. The platform's support videos were the sole source of training on database development. To facilitate interpretation of the prototype by stakeholders accustomed to systematic review data visualizations, charts were modeled after traditional forest plots ([Fig. 1]) but were slightly modified to take advantage of Tableau's dynamic reporting methods ([Fig. 2]).

Fig. 1 Traditional forest plot in a systematic review (reprinted from Skelly et al,[9] 2018). AC, attention control; CI, confidence interval; MI, minimal intervention; N, number; NE, no exercise; ODI, Oswestry Disability Index; PDI, Pain Disability Index; RDQ, Roland-Morris Disability Questionnaire; SD, standard deviation; SMD, standardized mean difference; UC, usual care; WL, waitlist.

Fig. 2 Presentation of results in Tableau, simulating a traditional forest plot.

Six key stakeholders, who had previous experience with guideline development committees, were given individual, guided demonstrations of the prototype. During the guided demonstration, stakeholders were asked to qualitatively assess the accessibility and usability of the prototype in guideline development, as well as provide feedback, on improving the prototype for future development. Feedback was recorded and then analyzed by our team to establish a consensus of key themes.

Results

The data were extracted from the PDF of the original report.[9] Tableau uses a relational format for its data input, where each row of data corresponds to a single observation. For this project, we chose to use an Excel (2016) workbook, though numerous other options would have worked as well (MySQL, Microsoft Access, delimited text files, cloud servers, etc.) The data were extracted into an Excel (2016) workbook, organized in a relational format with three sheets, such as Conditions, Studies, and Outcomes ([Fig. 3]). The sheets of Conditions and Studies contained information about the conditions and studies. The final sheet, Outcomes, had individual rows for each outcome and the foreign keys for its associated condition and study. Both study-level and summary-level outcome data were stored in the Outcomes table to allow for interactivity in a shared visualization. Summary-level outcome records were denoted by an invalid Study ID in the Outcomes table, severing the link to the Studies table. Data extraction was completed over the course of a month, accounting for most of the hours spent on this project.

Fig. 3 Data from the original PDF were extracted and stored in Excel (2016) using a relational structure.

The visualization was prototyped using Tableau Desktop. Individual visualizations for each study (Study level) and for summary results across similar studies (Summary level) were developed. The studies level visualization ([Fig. 4]) included data for six possible outcomes, including three follow-up terms (short, intermediate, and long) for two outcome measures (pain or function). Outcomes were grouped across Condition, Intervention Category, Comparator, Outcome, and Term. The Summary level visualization ([Fig. 5]) included the summarized data by the factors used to group them in the original report.

Fig. 4 Study-level visualization of data extracted from the original report.

Fig. 5 Summary-level visualization of aggregated data.

The Study and Summary visualizations were combined into an interactive dashboard ([Fig. 6]) that allowed users to select factors they wish to view or hide which subsequently filtered the data depicted in the Summary and Studies sections. Additionally, selections made in the Summary section filtered the data shown in the Study section, allowing users to view the individual studies contained in a summary record. In Tableau, tooltips are custom pop-up windows that are used to display additional information about the data when selected or hovered over. For the Summary and Study data, these were displayed were displayed when hovering over data. Tooltips within the Summary level ([Fig. 7]) included aggregate measures and summary statistics for the group of studies included, such as the standardized mean difference (SMD), number of studies, and the strength of evidence. Tooltips within the Studies level ([Fig. 8]) included the participant numbers for both intervention and control, study quality according to published criteria, and citation information for the study, including a link to the PubMed abstract. Conditional formatting of tooltips was accomplished through creation of intermediate Calculated Fields. Dynamic filtering of study level data and generation of PubMed URLs was completed using Custom Actions. Lastly, a final tab was created to provide an example of the visualization's functionality by way of a guided analysis ([Fig. 9]). This functionality, known as a Tableau story, demonstrated the functionality of the tool to users using saved states of the report designed by the research team. The final product was posted publicly on Tableau Public.[10]

Fig. 6 Tableau dashboard, including the combined Summary and Studies level visualizations; global filters for data are available at the top of the page. Selecting specific output in the Summary level data will filter the visible Studies level data and show only data that were used to calculate the selected Summary output.

Fig. 7 Example of a Summary level tooltip. Conditionally formatted fields were created using “Calculated Fields” in Tableau.

Fig. 8 Example of a Study level tooltip. Conditionally formatted fields were created using “Calculated Fields” in Tableau. A hyperlink to the PubMed entry was created using the “Actions” functionality, and appears after the user clicks the data bar in the visualization.

Fig. 9 A guided “tour” of the dashboard was created using the Tableau Story functionality. This allows the user to be guided through the potential use of the visualization.

The dashboard was created with an emphasis on flexible and customizable analysis of the review data. The original report evaluated treatments by individual outcomes and conditions to answer prespecified key questions, preventing any data analysis outside the rigid report structure. When developing guidelines or investigating potential treatments, end-users may want to “slice and dice” the data outside the scope of the key questions, for example, comparing a specific treatment across all the types of pain, or looking at multiple treatments in several conditions. The dashboard allowed for novel analysis of tailored questions by using custom filtering of fields, in comparison to the traditional ‘flat’ reporting format. For example, a user could easily assess the long-term effectiveness of Pilates, a type of exercise, across chronic back pain and chronic neck pain.

Overall, response from interviewed stakeholders was positive for the prototype both aesthetically and for its potential functionality. The freedom of inquiry afforded by the prototype was highlighted as a strong positive, though the need for a more in-depth narrative as afforded in the original report may be necessary for some users with less familiarity with the data.[8]

Discussion

Systematic reviews contain a large volume of data essential for decision-making, but accessibility and usability may be hampered by “flat” reporting tools that mirror prespecified questions. We developed a dynamic visualization of data from a completed systematic review using the commercial product Tableau and assessed its potential to permit customized inquiries beyond the original report structure. While the use of Tableau for visualizing existing data are not novel, the adaptation of systematic review data into a dynamic visualization shows potential for improving dissemination of systematic review data. Stakeholders stated that the dynamic visual representation of the data would be immensely useful for investigating novel questions rooted in the local guideline committee needs; however, this method is only sufficient for looking at the results in their original context. The original meta-analyses in systematic reviews are calculated with many assumptions in mind and cannot be recalculated ad hoc without careful analysis, typically by a biostatistician.[11] Given the density and nuance of scientific and medical data required to develop guidelines, the visualization lacks the depth of detail and context described textually in the written report. While the prototype is promising in its capacity to make complex data accessible to informatics-naïve audience, the dynamic visualization should be used only to supplement the traditional reporting methods, not replace them.

Limitations

The primary limitation of the project was the method in which the systematic review data were abstracted. The template for data abstraction was developed to meet the needs of a specific format, rather than a relational structure; data points were often collapsed into a joint field, limiting the ability to programmatically extract the data. This method is standard procedure in some groups, given regulatory requirements for accessibility and presentation. If dynamic visualizations were included in the analysis and reporting plan at the beginning of the review, the abstraction template could be revised to facilitate a more amenable storage structure. Based on our experience and assessment of staff proficiency, we recommend engaging an informaticist throughout the project. This additional expertise will mitigate time- and resource-intensive training of research staff, as well as ensure development, of an appropriate and sustainable template for data extraction and storage protocols that can be shared with other centers. If a project is started with creating a visualization in mind, then steps can be taken to ensure that the data are collected in an appropriate way, reducing or eliminating the need for reprocessing of data after the fact.

The second major limitation of this project was the limited scope. The scope of the project was to develop potential visualizations of systematic review data that could be used in guideline-development committees. While this provided potential value for future work, the project did not include a rigorous analysis of the alternative reporting method. Future studies would benefit from inclusion of quantitative and qualitative comparisons between the traditional and alternative methods for reporting of systematic review data, such as comparing usability ratings of the method or assessing retention of the information by the users.

Other limitations were related to the structure of the desired visualization which prohibited the inclusion of all data from the original report. For example, trials that combined multiple interventions reported summary data, rather than outcomes by individual intervention. We did not extract these data to simplify our initial use case. Additionally, data that were not summarized using a standardized mean difference (SMD) were excluded, as data on separate scales cannot be compared without standardizing. For example, while most outcomes in the original report were reported using SMD, fibromyalgia outcomes were measured using a common scale, so SMD was not required. To address this limitation, we could recalculate these values for the visualization but chose not to do so in the interest of time.

Conclusion

As more health systems invest in the development of evidence-based practices and incorporate large volumes of complex data, the need for data dissemination will continue to grow. The current reporting paradigm for systematic reviews is not conducive to dynamic, efficient consumption of evidence. Innovative reporting tools are required to improve accessibility and usability of data. We created a novel visualization prototype using published data and an existing reporting tool and demonstrated genuine value in exploring alternative reporting modalities. To facilitate further exploration and adoption of these innovative modalities, we advocate for inclusion of informaticists on research teams to inform alternative data extraction and storage practices.

Clinical Relevance Statement

The exploration of alternative reporting methods for systematic reviews is essential to the translation of research to clinical guidelines. Current reporting via in-depth, narrative “flat” documents is necessary but not sufficient, as their predefined and rigid structure does not facilitate agile analysis of complex inquiry by subject matter experts. Dynamic visualization tools show promise in improving accessibility and usability of data, leading to more robust investigation during clinical guidelines development.

Multiple Choice Questions

1. When considering the use of Tableau for visualization, data should be stored in or be able to be represented in what format?

   • JSON.

   • Relational/worksheet.

   • Nonrelational.

   • SAS Grid.

   Correct Answer: The correct answer is option b. Tableau uses a relational format to join multiple data sources, resulting in a worksheet/data-frame format. This format is organized in that each row corresponds to an individual or observation of many variables.

2. When considering a visualization of Systematic review data, the visualization should be used to _____ the final report?

   • Replace.

   • Serve as a guide for navigating.

   • Supplement, but not replace.

   • Contradict.

   Correct Answer: The correct answer is option c. While the visualization is very powerful for analyzing the data, it is not sufficient to replace the report entirely. The rigid structure of the typical final report allows for a more in-depth analysis of the data based on the key questions, along with a narrative portion that is harder to include in a dynamic report.

Conflict of Interest

None declared.

Acknowledgments

The authors acknowledge Kenneth Dunham, MS, a master's student from the Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, for his work on the research for the project and assistance in data extraction; Elaine Graham, MLS, from the Pacific Northwest Evidence-based Practice Center, Oregon Health and Science University, for integral support throughout the project; Elizabeth Crabtree, PhD, Doan Ha, DrPH, Marcy Hager, MA, Stephanie Halvorson, MD, Ilse Larson, MD, David Lieberman, MD, and the BioData Club, all at Oregon Health and Science University, for their feedback on the dissemination products; and Andrea Skelly, PhD, and Erika Brodt, BS, from the Pacific Northwest EPC, for their work on the chronic pain review, from which the pilot test data were abstracted.

Protection of Human and Animal Subjects

Human and/or animal subjects were not included in the project.

• References

• 1 Chan KS, Morton SC, Shekelle PG. Systematic reviews for evidence-based management: how to find them and what to do with them. Am J Manag Care 2004; 10 (11 Pt 1): 806-812
• 2 Langendam MW, Akl EA, Dahm P, Glasziou P, Guyatt G, Schünemann HJ. Assessing and presenting summaries of evidence in Cochrane Reviews. Syst Rev 2013; 2: 81
• 3 McCormack L, Sheridan S, Lewis M. , et al. Communication and dissemination strategies to facilitate the use of health-related evidence. Evid Rep Technol Assess (Full Rep) 2013; (213) 1-520
• 4 Perrier L, Kealey MR, Straus SE. A usability study of two formats of a shortened systematic review for clinicians. BMJ Open 2014; 4 (12) e005919
• 5 Perrier L, Kealey MR, Straus SE. An iterative evaluation of two shortened systematic review formats for clinicians: a focus group study. J Am Med Inform Assoc 2014; 21 (e2): e341-e346
• 6 Marquez C, Johnson AM, Jassemi S. , et al. Enhancing the uptake of systematic reviews of effects: what is the best format for health care managers and policy-makers? A mixed-methods study. Implement Sci 2018; 13 (01) 84
• 7 Tableau Software. Available at: https://www.tableau.com/ . Accessed April 17, 2019
• 8 Totten AM, Smith C, Dunham K, Jungbauer RM, Graham E. Improving Access to and Usability of Systematic Review Data for Health Systems Guidelines Development. Available at: https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/methods-health-systems-guidelines-development.pdf . Accessed June 7, 2019
• 9 Skelly AC, Chou R, Dettori JR. , et al. Noninvasive Nonpharmacological Treatment for Chronic Pain: A Systematic Review. Rockville, MD: Agency for Healthcare Research and Quality (U.S.); 2018
• 10 Visualizing Systematic Review Data. Pacific Northwest Evidence-Based Practice Center. Available at: https://public.tableau.com/profile/pnw.epc#!/vizhome/VisualizingSystematicReviewData/Dashboard . Accessed June 7, 2019
• 11 Nelson HD. Systematic Reviews to Answer Health Care Questions. Philadelphia, PA: Wolters Kluwer Health; 2014