Download PDF
Methods Inf Med 2010; 49(04): 379-387
DOI: 10.3414/ME0613
Original Articles
Schattauer GmbH

Design Issues for Socially Intelligent User Interfaces

A Discourse Analysis of a Data-to-text System for Summarizing Clinical Data
A. McKinlay
1   Department of Psychology, University of Edinburgh, Edinburgh, UK
,
C. McVittie
2   Department of Psychology, Queen Margaret University, Edinburgh, UK
,
E. Reiter
3   Department of Computing Science, University of Aberdeen, Aberdeen, UK
,
Y. Freer
4   School of Health and Social Science, University of Edinburgh, Edinburgh, UK
5   Simpson Centre for Reproductive Health, Royal Infirmary of Edinburgh, Edinburgh, UK
,
C. Sykes
5   Simpson Centre for Reproductive Health, Royal Infirmary of Edinburgh, Edinburgh, UK
,
R. Logie
1   Department of Psychology, University of Edinburgh, Edinburgh, UK
› Author Affiliations
Further Information

Publication History

received: 14 November 2008

accepted: 26 October 2009

Publication Date:
17 January 2018 (online)

    Permissions and Reprints

Summary

Objectives: This study aims to demonstrate the usability of discourse analyses as a means of evaluating medical informatics systems by examining one particular computer-based data-to-text system for delivering neonatal health care information.

Methods: Six textual summaries of clinical information, three produced by human clinicians and three by the data-to-text system, were subjected to fine-grain discourse analysis. Analysis was performed ‘blind’ on all six textual summaries. Analysis focused on the identification of lexical items and on the potential effects of these items on users of these clinical information summaries.

Results: Results showed that there were clear differences between human- and system-generated clinical summaries, with human clinicians providing better narrative flow and textual detail. The data-to-text system successfully produced textual summaries although it fell short of human abilities.

Conclusions: These results indicate potential future improvements to the system. Discourse analysis as used here may offer significant advantages in evaluating and developing similar medical informatics systems.

Keywords

Medical informatics applications - user-computer interface - neonate - doctor-patient relations - discourse analysis
 

  • References

  • 1 Kaplan B, Shaw NT. Future directions in evaluation research: people, organizational, and social issues. Methods Inf Med 2004; 43 (03) 215-231.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 2 Moehr JR, Anglin C, Schaafsma J, Pantazi S, Grimm N. Lest formalisms impede insight and success: evaluation in health informatics – a case study. Methods Inf Med 2006; 45 (01) 67-72.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 3 Aarts J, Berg M. Same systems, different outcomes – comparing the implementation of computerized physician order entry in two Dutch hospitals. Methods Inf Med 2006; 45 (01) 53-61.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 4 Dahl Y. “You have a message here”: enhancing interpersonal communication in a hospital ward with location-based virtual notes. Methods Inf Med 2006; 45 (06) 602-609.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 5 Reiter E. An architecture for Data-to-Text systems. In: Buseman S. editor. 11th European Workshop on Natural Language Generation. Saarbrueken: DFKI GmbH; 2007. pp 97-104.
    Google Scholar
  • 6 Goldberg E, Driedger N, Kittredge RI. Using natural-language processing to produce weather forecasts. IEEE Expert 1994; 9 (02) 45-53.
    PubMedGoogle Scholar
  • 7 Reiter E, Sripada S, Hunter J, Yu J, Davy I. Choosing words in computer-generated weather forecasts. Artificial Intelligence 2005; 167 1–2 137-169.
    CrossrefPubMedGoogle Scholar
  • 8 Yu J, Reiter E, Hunter J, Mellish C. Choosing the content of textual summaries of large time-series datasets. Natural Language Engineering 2007; 13 (01) 25-49.
    CrossrefPubMedGoogle Scholar
  • 9 Turner R, Sripada S, Reiter E, Davy I. Selecting the content of textual descriptions of geographically located events in spatio-temporal weather data. In: Ellis R, Allen T, Petridis M. editors. Applications and innovations in intelligent systems XV: Proceedings of AI-2007. London: Springer; 2007: 75-88.
    Google Scholar
  • 10 Portet F, Reiter E, Gatt A, Hunter J, Sripada S, Freer Y, Sykes C. Automatic Generation of Textual Summaries from Neonatal Intensive Care Data. Artificial Intelligence 2009; 173 7–8 789-816.
    CrossrefPubMedGoogle Scholar
  • 11 Law AS, Freer Y, Hunter J, Logie R, McIntosh N, Quinn J. A comparison of graphical and textual presentations of time series data to support medical decision making in the neonatal intensive care unit. J Clin Monit Comput 2005; 19 (03) 183-194.
    CrossrefPubMedGoogle Scholar
  • 12 Ammenwerth E, Spötl H-P.. The time needed for clinical documentation versus direct patient care. Methods Inf Med 2009; 48 (01) 84-91.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 13 Hüske-Kraus D. Text Generation in Clinical Medicine: a Review. Methods Inf Med 2003; 42 (01) 51-60.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 14 Hüske-Kraus D. Suregen 2: A shell system for the generation of clinical documents. In: European Chapter Meeting of the ACL: Proceedings of the tenth conference on European chapter of the Association for Computational Linguistics – Volume 2; 2003 April 12-17; Budapest, Hungary. Morristown, NJ:: Association for Computational Linguistics;: 2003. pp 215-218.
    Google Scholar
  • 15 Cawsey A, Jones R, Pearson J. The evaluation of a personalised health information system for patients with cancer. User Modelling and User-Adapted Interaction 2000; 10: 47-72.
    PubMedGoogle Scholar
  • 16 Hallett C. Multi-modal presentation of medical histories. In: Bradshaw J, Lieberman H, Staab S. editors. Proceedings of the 13th international conference on Intelligent user interfaces; 2008 January 13-16; Gran Canaria, Spain. New York: ACM; 2008. pp 80-89.
    Google Scholar
  • 17 Webber B, Carberry S, Clarke JR, Gertner A, Harvey T, Rymon R, Washington R. Exploiting multiple goals and intentions in decision support for the management of multiple trauma: a review of the TraumAID project. Artificial Intelligence 1998; 105 1–2 263-293.
    CrossrefPubMedGoogle Scholar
  • 18 McKeown K, Pan S, Shaw J, Jordon D, Allen B. Language generation for multimedia healthcare briefings. Proceedings of the Fifth Conference on Applied Natural-Language Processing (ANLP-1997); 1997; Washington, DC. Somerset, NJ:: ACL: 1997 pp 277-282.
    PubMedGoogle Scholar
  • 19 McKeown K, Jordon D, Feiner S, Shaw J, Chen E, Ahmad S, Kushniruk A, Patel V. A study of communication in the cardiac surgery intensive care unit and its implications for automated briefing. JAMIA 2000; 7 Supp 570-574.
    PubMedGoogle Scholar
  • 20 Belz A, Reiter E. Comparing automatic and human evaluation of NLG systems. Proceedings of the 11th conference of the European chapter of the association for computational linguistics (EACL’06). 2006 pp 313-320.
    PubMedGoogle Scholar
  • 21 Papineni K, Roukos S, Ward T, Zhu W-J. BLEU: a Method for Automatic Evaluation of Machine Translation. RC22176 (W0109–022). 2001. Yorktown Heights, NY, IBM Research Division, Thomas J, Watson Research Center.:
    Google Scholar
  • 22 Sripada S, Reiter E, Hawizy L. Evaluation of an NLG system using post-edit data: Lessons learned. In: Willcock G, Jokinen K, Mellish C, Reiter E. editors. Proceedings of the 10th European Workshop on Natural Language Generation (ENLG 2005). 2005. pp 133-139.
    Google Scholar
  • 23 van der Meulen M, Logie RH, Freer Y, Sykes C, McIntosh N, Hunter J. When a graph is poorer than 100 words: A comparison of computerized natural language generation, human generated descriptions and graphical displays in neonatal intensive care. Applied Cognitive Psychology 2010; 24: 77-89.
    CrossrefPubMedGoogle Scholar
  • 24 Dorr DA, Phillips WF, Phansalkar S, Sims SA, Hurdle JF. Assessing the difficulty and time cost of de-identification in clinical narratives. Methods Inf Med 2006; 45 (03) 246-252.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 25 McKinlay A, McVittie C. Social Psychology and Discourse. Oxford: Wiley-Blackwell; 2008
    Google Scholar
  • 26 Bamberg M. Narrative discourse and identities. In: Meister JC, Kindt T, Schernus W. editors. Narratology beyond literary criticism: Mediality Disciplinarity. Berlin: Walter de Gruyter; 2005. pp 213-237.
    Google Scholar
  • 27 Hobbs P. The use of evidentiality in physicians’ progress notes. Discourse Studies 2003; 5 (04) 451-478.
    CrossrefPubMedGoogle Scholar
  • 28 Hobbs P. The role of progress notes in the professional socialization of medical residents. Journal of Pragmatics 2004; 36 (09) 1579-1607.
    CrossrefPubMedGoogle Scholar
  • 29 Ball MJ, Silva JS, Bierstock S, Douglas JV, Norcio AF, Chakraborty J, Srini J. Failure to provide clinicians useful IT systems: opportunities to leapfrog current technologies. Methods Inf Med 2008; 47 (01) 4-7.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 30 Østerlund CS. Documents in place: Demarcating places for collaboration in healthcare settings. CSCW 2008; 17: 195-225.
    PubMedGoogle Scholar
  • 31 Brender J. Evaluation of health information applications – challenges ahead of us. Methods Inf Med 2006; 45 (01) 62-66.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 32 Pirnejad H, Niazkhani Z, Berg M, Bal R. Intra-organizational communication in healthcare – considerations for standardization and ICT application. Methods Inf Med 2008; 47 (04) 336-345.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 33 Perez y Perez R, Sharples M. Three computer-based models of story telling: Brutus, Minstrel and Mexica. Knowledge-Based Systems 2004; 17 (01) 15-29.
    PubMedGoogle Scholar
  • 34 Callaway C, Lester J. Narrative prose generation. Artificial Intelligence 2002; 139 (02) 213-252.
    CrossrefPubMedGoogle Scholar
  • 35 van Bemmel JH. Medical informatics is interdisciplinary avant la lettre. Methods Inf Med 2008; 47 (04) 318-321.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 36 Knaup P, Dickhaus H. Editorial: perspectives of medical informatics: advancing health care requires interdisciplinarity and interoperability. Methods Inf Med 2009; 48 (01) 1-3.
    Article in Thieme ConnectPubMedGoogle Scholar