RSS-Feed abonnieren
DOI: 10.3414/ME0613
Design Issues for Socially Intelligent User Interfaces
A Discourse Analysis of a Data-to-text System for Summarizing Clinical DataPublikationsverlauf
received:
14. November 2008
accepted:
26. Oktober 2009
Publikationsdatum:
17. Januar 2018 (online)
![](https://www.thieme-connect.de/media/10.1055-s-00035037/201004/lookinside/thumbnails/10-3414-me0613-1.jpg)
Summary
Objectives: This study aims to demonstrate the usability of discourse analyses as a means of evaluating medical informatics systems by examining one particular computer-based data-to-text system for delivering neonatal health care information.
Methods: Six textual summaries of clinical information, three produced by human clinicians and three by the data-to-text system, were subjected to fine-grain discourse analysis. Analysis was performed ‘blind’ on all six textual summaries. Analysis focused on the identification of lexical items and on the potential effects of these items on users of these clinical information summaries.
Results: Results showed that there were clear differences between human- and system-generated clinical summaries, with human clinicians providing better narrative flow and textual detail. The data-to-text system successfully produced textual summaries although it fell short of human abilities.
Conclusions: These results indicate potential future improvements to the system. Discourse analysis as used here may offer significant advantages in evaluating and developing similar medical informatics systems.
-
References
- 1 Kaplan B, Shaw NT. Future directions in evaluation research: people, organizational, and social issues. Methods Inf Med 2004; 43 (03) 215-231.
- 2 Moehr JR, Anglin C, Schaafsma J, Pantazi S, Grimm N. Lest formalisms impede insight and success: evaluation in health informatics – a case study. Methods Inf Med 2006; 45 (01) 67-72.
- 3 Aarts J, Berg M. Same systems, different outcomes – comparing the implementation of computerized physician order entry in two Dutch hospitals. Methods Inf Med 2006; 45 (01) 53-61.
- 4 Dahl Y. “You have a message here”: enhancing interpersonal communication in a hospital ward with location-based virtual notes. Methods Inf Med 2006; 45 (06) 602-609.
- 5 Reiter E. An architecture for Data-to-Text systems. In: Buseman S. editor. 11th European Workshop on Natural Language Generation. Saarbrueken: DFKI GmbH; 2007. pp 97-104.
- 6 Goldberg E, Driedger N, Kittredge RI. Using natural-language processing to produce weather forecasts. IEEE Expert 1994; 9 (02) 45-53.
- 7 Reiter E, Sripada S, Hunter J, Yu J, Davy I. Choosing words in computer-generated weather forecasts. Artificial Intelligence 2005; 167 1–2 137-169.
- 8 Yu J, Reiter E, Hunter J, Mellish C. Choosing the content of textual summaries of large time-series datasets. Natural Language Engineering 2007; 13 (01) 25-49.
- 9 Turner R, Sripada S, Reiter E, Davy I. Selecting the content of textual descriptions of geographically located events in spatio-temporal weather data. In: Ellis R, Allen T, Petridis M. editors. Applications and innovations in intelligent systems XV: Proceedings of AI-2007. London: Springer; 2007: 75-88.
- 10 Portet F, Reiter E, Gatt A, Hunter J, Sripada S, Freer Y, Sykes C. Automatic Generation of Textual Summaries from Neonatal Intensive Care Data. Artificial Intelligence 2009; 173 7–8 789-816.
- 11 Law AS, Freer Y, Hunter J, Logie R, McIntosh N, Quinn J. A comparison of graphical and textual presentations of time series data to support medical decision making in the neonatal intensive care unit. J Clin Monit Comput 2005; 19 (03) 183-194.
- 12 Ammenwerth E, Spötl H-P.. The time needed for clinical documentation versus direct patient care. Methods Inf Med 2009; 48 (01) 84-91.
- 13 Hüske-Kraus D. Text Generation in Clinical Medicine: a Review. Methods Inf Med 2003; 42 (01) 51-60.
- 14 Hüske-Kraus D. Suregen 2: A shell system for the generation of clinical documents. In: European Chapter Meeting of the ACL: Proceedings of the tenth conference on European chapter of the Association for Computational Linguistics – Volume 2; 2003 April 12-17; Budapest, Hungary. Morristown, NJ:: Association for Computational Linguistics;: 2003. pp 215-218.
- 15 Cawsey A, Jones R, Pearson J. The evaluation of a personalised health information system for patients with cancer. User Modelling and User-Adapted Interaction 2000; 10: 47-72.
- 16 Hallett C. Multi-modal presentation of medical histories. In: Bradshaw J, Lieberman H, Staab S. editors. Proceedings of the 13th international conference on Intelligent user interfaces; 2008 January 13-16; Gran Canaria, Spain. New York: ACM; 2008. pp 80-89.
- 17 Webber B, Carberry S, Clarke JR, Gertner A, Harvey T, Rymon R, Washington R. Exploiting multiple goals and intentions in decision support for the management of multiple trauma: a review of the TraumAID project. Artificial Intelligence 1998; 105 1–2 263-293.
- 18 McKeown K, Pan S, Shaw J, Jordon D, Allen B. Language generation for multimedia healthcare briefings. Proceedings of the Fifth Conference on Applied Natural-Language Processing (ANLP-1997); 1997; Washington, DC. Somerset, NJ:: ACL: 1997 pp 277-282.
- 19 McKeown K, Jordon D, Feiner S, Shaw J, Chen E, Ahmad S, Kushniruk A, Patel V. A study of communication in the cardiac surgery intensive care unit and its implications for automated briefing. JAMIA 2000; 7 Supp 570-574.
- 20 Belz A, Reiter E. Comparing automatic and human evaluation of NLG systems. Proceedings of the 11th conference of the European chapter of the association for computational linguistics (EACL’06). 2006 pp 313-320.
- 21 Papineni K, Roukos S, Ward T, Zhu W-J. BLEU: a Method for Automatic Evaluation of Machine Translation. RC22176 (W0109–022). 2001. Yorktown Heights, NY, IBM Research Division, Thomas J, Watson Research Center.:
- 22 Sripada S, Reiter E, Hawizy L. Evaluation of an NLG system using post-edit data: Lessons learned. In: Willcock G, Jokinen K, Mellish C, Reiter E. editors. Proceedings of the 10th European Workshop on Natural Language Generation (ENLG 2005). 2005. pp 133-139.
- 23 van der Meulen M, Logie RH, Freer Y, Sykes C, McIntosh N, Hunter J. When a graph is poorer than 100 words: A comparison of computerized natural language generation, human generated descriptions and graphical displays in neonatal intensive care. Applied Cognitive Psychology 2010; 24: 77-89.
- 24 Dorr DA, Phillips WF, Phansalkar S, Sims SA, Hurdle JF. Assessing the difficulty and time cost of de-identification in clinical narratives. Methods Inf Med 2006; 45 (03) 246-252.
- 25 McKinlay A, McVittie C. Social Psychology and Discourse. Oxford: Wiley-Blackwell; 2008
- 26 Bamberg M. Narrative discourse and identities. In: Meister JC, Kindt T, Schernus W. editors. Narratology beyond literary criticism: Mediality Disciplinarity. Berlin: Walter de Gruyter; 2005. pp 213-237.
- 27 Hobbs P. The use of evidentiality in physicians’ progress notes. Discourse Studies 2003; 5 (04) 451-478.
- 28 Hobbs P. The role of progress notes in the professional socialization of medical residents. Journal of Pragmatics 2004; 36 (09) 1579-1607.
- 29 Ball MJ, Silva JS, Bierstock S, Douglas JV, Norcio AF, Chakraborty J, Srini J. Failure to provide clinicians useful IT systems: opportunities to leapfrog current technologies. Methods Inf Med 2008; 47 (01) 4-7.
- 30 Østerlund CS. Documents in place: Demarcating places for collaboration in healthcare settings. CSCW 2008; 17: 195-225.
- 31 Brender J. Evaluation of health information applications – challenges ahead of us. Methods Inf Med 2006; 45 (01) 62-66.
- 32 Pirnejad H, Niazkhani Z, Berg M, Bal R. Intra-organizational communication in healthcare – considerations for standardization and ICT application. Methods Inf Med 2008; 47 (04) 336-345.
- 33 Perez y Perez R, Sharples M. Three computer-based models of story telling: Brutus, Minstrel and Mexica. Knowledge-Based Systems 2004; 17 (01) 15-29.
- 34 Callaway C, Lester J. Narrative prose generation. Artificial Intelligence 2002; 139 (02) 213-252.
- 35 van Bemmel JH. Medical informatics is interdisciplinary avant la lettre. Methods Inf Med 2008; 47 (04) 318-321.
- 36 Knaup P, Dickhaus H. Editorial: perspectives of medical informatics: advancing health care requires interdisciplinarity and interoperability. Methods Inf Med 2009; 48 (01) 1-3.