Towards Usable E-Health

Summary

Background: The use of e-health can lead to several positive outcomes. However, the potential for e-health to improve healthcare is partially dependent on its ease of use. In order to determine the usability for any technology, rigorously developed and appropriate measures must be chosen.

Objectives: To identify psychometrically tested questionnaires that measure usability of e-health tools, and to appraise their generalizability, attributes coverage, and quality.

Methods: We conducted a systematic review of studies that measured usability of e-health tools using four databases (Scopus, PubMed, CINAHL, and HAPI). Non-primary research, studies that did not report measures, studies with children or people with cognitive limitations, and studies about assistive devices or medical equipment were systematically excluded. Two authors independently extracted information including: questionnaire name, number of questions, scoring method, item generation, and psychometrics using a data extraction tool with pre-established categories and a quality appraisal scoring table.

Results: Using a broad search strategy, 5,558 potentially relevant papers were identified. After removing duplicates and applying exclusion criteria, 35 articles remained that used 15 unique questionnaires. From the 15 questionnaires, only 5 were general enough to be used across studies. Usability attributes covered by the questionnaires were: learnability (15), efficiency (12), and satisfaction (11). Memorability (1) was the least covered attribute. Quality appraisal showed that face/content (14) and construct (7) validity were the most frequent types of validity assessed. All questionnaires reported reliability measurement. Some questionnaires scored low in the quality appraisal for the following reasons: limited validity testing (7), small sample size (3), no reporting of user centeredness (9) or feasibility estimates of time, effort, and expense (7).

Conclusions: Existing questionnaires provide a foundation for research on e-health usability. However, future research is needed to broaden the coverage of the usability attributes and psychometric properties of the available questionnaires.

Citation: Sousa VEC, Lopez KD. Towards usable e-health: A systematic review of usability questionnaires. Appl Clin Inform 2017; 8: 470–490 https://doi.org/10.4338/ACI-2016-10-R-0170

Keywords

Usability - e-health - questionnaires - systematic review

Human Subjects Protections

Human subjects were not included in the project.

• References

• 1 US Department of Health and Human Services. Expanding the reach and impact of consumer e-health tools. Washington, DC: : US Department of Health and Human Services; 2006. Available from: https://health.gov/communication/ehealth/ehealthtools/default.htm
• 2 Steinhubl SR, Muse ED, Topol EJ. The emerging field of mobile health. Sci Transl Med 2015; 7 (283) 283rv3.
• 3 Chen J, Bauman A, Allman-Farinelli M. A Study to Determine the Most Popular Lifestyle Smartphone Applications and Willingness of the Public to Share Their Personal Data for Health Research. Telemed J E Health 2016; 22 (08) 655-665.
• 4 West JH, Hall PC, Hanson CL, Barnes MD, Giraud-Carrier C, Barrett J. There‘s an app for that: content analysis of paid health and fitness apps. J Med Internet Res 2012; 14 (03) e72.
• 5 Silva BM, Rodrigues JJ, de la Torre Diez I, Lopez-Coronado M, Saleem K. Mobile-health: A review of current state in 2015. J Biomed Inform 2015; 56: 265-272.
• 6 Benhamou PY, Melki V, Boizel R, Perreal F, Quesada JL, Bessieres-Lacombe S, Bosson JL, Halimi S, Hanaire H. One-year efficacy and safety of Web-based follow-up using cellular phone in type 1 diabetic patients under insulin pump therapy: the PumpNet study. Diabetes Metab 2007; 33 (03) 220-226.
• 7 Ostojic V, Cvoriscec B, Ostojic SB, Reznikoff D, Stipic-Markovic A, Tudjman Z. Improving asthma control through telemedicine: a study of short-message service. Telemed J E Health 2005; 11 (01) 28-35.
• 8 DeVito Dabbs A, Dew MA, Myers B, Begey A, Hawkins R, Ren D, Dunbar-Jacob J, Oconnell E, McCurry KR. Evaluation of a hand-held, computer-based intervention to promote early self-care behaviors after lung transplant. Clin Transplant 2009; 23 (04) 537-545.
• 9 Vidrine DJ, Marks RM, Arduino RC, Gritz ER. Efficacy of cell phone-delivered smoking cessation counseling for persons living with HIV/AIDS: 3-month outcomes. Nicotine Tob Res 2012; 14 (01) 106-110.
• 10 Lim MS, Hocking JS, Aitken CK, Fairley CK, Jordan L, Lewis JA, Hellard ME. Impact of text and email messaging on the sexual health of young people: a randomised controlled trial. J Epidemiol Community Health 2012; 66 (01) 69-74.
• 11 Kunawararak P, Pongpanich S, Chantawong S, Pokaew P, Traisathit P, Srithanaviboonchai K, Plipat T. Tuberculosis treatment with mobile-phone medication reminders in northern Thailand. Southeast Asian J Trop Med Public Health 2011; 42 (06) 1444-1451.
• 12 Lester RT, Ritvo P, Mills EJ, Kariri A, Karanja S, Chung MH, Jack W, Habyarimana J, Sadatsafavi M, Najafzadeh M, Marra CA, Estambale B, Ngugi E, Ball TB, Thabane L, Gelmon LJ, Kimani J, Ackers M, Plummer FA. Effects of a mobile phone short message service on antiretroviral treatment adherence in Kenya (WelTel Kenya1): a randomised trial. Lancet 2010; 376 9755 1838-1845.
• 13 Free C, Phillips G, Galli L, Watson L, Felix L, Edwards P, Patel V, Haines A. The effectiveness of mobile-health technology-based health behaviour change or disease management interventions for health care consumers: a systematic review. PLoS Med 2013; 10 (01) e1001362.
• 14 Seidling HM, Stutzle M, Hoppe-Tichy T, Allenet B, Bedouch P, Bonnabry P, Coleman JJ, Fernandez-Llimos F, Lovis C, Rei MJ, Storzinger D, Taylor LA, Pontefract SK, van den Bemt PM, van der Sijs H, Haefeli WE. Best practice strategies to safeguard drug prescribing and drug administration: an anthology of expert views and opinions. Int J Clin Pharm 2016; 38 (02) 362-373.
• 15 Kampmeijer R, Pavlova M, Tambor M, Golinowska S, Groot W. The use of e-health and m-health tools in health promotion and primary prevention among older adults: a systematic literature review. BMC Health Services Research 2016; 16 (05) 290.
• 16 Abran A, Khelifi A, Suryn W, Seffah A. Usability meanings and interpretations in ISO standards. Software Qual J 2003; 11 (04) 325-338.
• 17 International Organization for Standardization/International Electrotechnical Commission. ISO/IEC 9241–14 Ergonomic requirements for office work with visual display terminals (VDT)s –Part 14 Menu dialogues: ISO/IEC 9241–14; 1998
• 18 Nielsen J. Usability Engineering. Boston: Academic Press; 1993
• 19 Nielsen J. Lifetime Practice Award. Paris, France: SIGCHI; 2013
• 20 Gorrell M. The 21st Century Searcher: How the Growth of Search Engines Affected the Redesign of EBSCOhost. Against the Grain 2008; 20 (03) 22-26.
• 21 Sharma SK, Chen R, Zhang J. Examining Usability of E-learning Systems-An Exploratory Study (Re-search-in-Progress). International Proceedings of Economics Development and Research 2014; 81: 120.
• 22 Yan J, El Ahmad A.S. Usability of CAPTCHAs or usability issues in CAPTCHA design. 4th symposium on Usable privacy and security. 2008. Pittsburgh, PA: ACM;
• 23 Jonsson A. Usability in three generations business support systems: Assessing perceived usability in the banking industry. Norrköping: Linköping University; 2013
• 24 Aljohani M, Blustein J. Heuristic evaluation of university institutional repositories based on DSpace. International Conference of Design, User Experience, and Usability Springer International Publishing; 2015: 119-130.
• 25 Tapani J. Game Usability in North American Video Game Industry. Oulu: University of Oulu; 2016
• 26 Lilholt PH, Jensen MH, Hejlesen OK. Heuristic evaluation of a telehealth system from the Danish Tele-Care North Trial. Int J Med Inform 2015; 84 (05) 319-326.
• 27 Lu JM, Hsu YL, Lu CH, Hsu PE, Chen YW, Wang JA. Development of a telepresence robot to rebuild the physical face-to-face interaction in remote interpersonal communication. International Design Alliance (IDA) Congress Education Congress. October, 24–26th; Taipei, Taiwan; 2011
• 28 Sparkes J, Valaitis R, McKibbon A. A usability study of patients setting up a cardiac event loop recorder and BlackBerry gateway for remote monitoring at home. Telemedicine and E-health 2012; 18 (06) 484-490.
• 29 Ozok AA, Wu H, Garrido M, Pronovost PJ, Gurses AP. Usability and perceived usefulness of Personal Health Records for preventive health care: a case study focusing on patients’ and primary care providers’ perspectives. Appl Ergon 2014; 45 (03) 613-628.
• 30 Zaidi ST, Marriott JL. Barriers and Facilitators to Adoption of a Web-based Antibiotic Decision Support System. South Med Rev 2012; 5 (02) 42-50.
• 31 Carayon P, Cartmill R, Blosky MA, Brown R, Hackenberg M, Hoonakker P, Hundt AS, Norfolk E, Wetter-neck TB, Walker JM. ICU nurses’ acceptance of electronic health records. J Am Med Inform Assoc 2011; 18 (06) 812-819.
• 32 Holden RJ, Karsh BT. The technology acceptance model: its past and its future in health care. J Biomed Inform 2010; 43 (01) 159-172.
• 33 Sun HM, Li SP, Zhu YQ, Hsiao B. The effect of user‘s perceived presence and promotion focus on usability for interacting in virtual environments. Appl Ergon 2015; 50: 126-132.
• 34 Koppel R. Is healthcare information technology based on evidence?. Yearb Med Inform 2013; 8 (01) 7-12.
• 35 Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, Strom BL. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005; 293 (10) 1197-1203.
• 36 Stead WW, Lin HS. Computational Technology for Effective Health Care: Immediate Steps and Strategic Directions. Washington (DC): 2009
• 37 Koch SH, Weir C, Haar M, Staggers N, Agutter J, Gorges M, Westenskow D. Intensive care unit nurses’ information needs and recommendations for integrated displays to improve nurses’ situation awareness. J Am Med Inform Assoc 2012; 19 (04) 583-590.
• 38 Lurio J, Morrison FP, Pichardo M, Berg R, Buck MD, Wu W, Kitson K, Mostashari F, Calman N. Using electronic health record alerts to provide public health situational awareness to clinicians. J Am Med Inform Assoc 2010; 17 (02) 217-219.
• 39 Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform 2009; 78 (05) 340-353.
• 40 Lopez KD, Febretti A, Stifter J, Johnson A, Wilkie DJ, Keenan G. Toward a More Robust and Efficient Usability Testing Method of Clinical Decision Support for Nurses Derived From Nursing Electronic Health Record Data. Int J Nurs Knowl. 2016 Jun 24.
• 41 Zhang J, Walji MF. TURF: toward a unified framework of EHR usability. J Biomed Inform 2011; 44 (06) 1056-1067.
• 42 Sousa VE, Lopez KD, Febretti A, Stifter J, Yao Y, Johnson A, Wilkie DJ, Keenan GM. Use of Simulation to Study Nurses’ Acceptance and Nonacceptance of Clinical Decision Support Suggestions. Comput Inform Nurs 2015; 33 (10) 465-472.
• 43 Christensen BL, Kockrow EO. Adult health nursing. St. Louis: Elsevier Health Sciences; 2014
• 44 Albertazzi D, Okimoto ML, Ferreira MG. Developing an usability test to evaluate the use of augmented reality to improve the first interaction with a product. Work 2011; 41: 1160-1163.
• 45 Freire LL, Arezes PM, Campos JC. A literature review about usability evaluation methods for e-learning platforms. Work-Journal of Prevention Assessment and Rehabilitation 2012; 41: 1038.
• 46 Yen PY, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012; 19 (03) 413-422.
• 47 Santos NA, Sousa VEC, Pascoal LM, Lopes MVO, Pagliuca LMF. editors. Concept analysis of usability of healthcare technologies. 17th National Nursing Research Seminar. 2013. Natal, RN, Brazil: Brazilian Nursing Association;
• 48 Lei J, Xu L, Meng Q, Zhang J, Gong Y. The current status of usability studies of information technologies in China: a systematic study. Biomed Res Int 2014; 2014: 568303.
• 49 Lyles CR, Sarkar U, Osborn CY. Getting a technology-based diabetes intervention ready for prime time: a review of usability testing studies. Curr Diab Rep 2014; 14 (10) 534.
• 50 Wootten AC, Abbott JAM, Chisholm K, Austin DW, Klein B, McCabe M, Murphy DG, Costello AJ. Development, feasibility and usability of an online psychological intervention for men with prostate cancer: My Road Ahead. Internet Interventions 2014; 1 (04) 188-195.
• 51 Friedman K, Noyes J, Parkin CG. 2-Year follow-up to STeP trial shows sustainability of structured self-monitoring of blood glucose utilization: results from the STeP practice logistics and usability survey (STeP PLUS). Diabetes Technol Ther 2013; 15 (04) 344-347.
• 52 Stafford E, Hides L, Kavanagh DJ. The acceptability, usability and short-term outcomes of Get Real: A web-based program for psychotic-like experiences (PLEs). Internet Interventions 2015; 2: 266-271.
• 53 Shah N, Jonassaint J, De Castro L. Patients welcome the Sickle Cell Disease Mobile Application to Record Symptoms via Technology (SMART). Hemoglobin 2014; 38 (02) 99-103.
• 54 Shyr C, Kushniruk A, Wasserman WW. Usability study of clinical exome analysis software: top lessons learned and recommendations. J Biomed Inform 2014; 51: 129-136.
• 55 Anders S, Albert R, Miller A, Weinger MB, Doig AK, Behrens M, Agutter J. Evaluation of an integrated graphical display to promote acute change detection in ICU patients. Int J Med Inform 2012; 81 (12) 842-851.
• 56 Yen PY, Gorman P. Usability testing of digital pen and paper system in nursing documentation. AMIA Annu Symp Proc 2005: 844-848.
• 57 McDermott RJ, Sarvela PD. Health education evaluation and measurement: A practitioner‘s perspective. 2 ed. Dubuque, Iowa: William C Brown Pub; 1998
• 58 Taveira-Gomes T, Saffarzadeh A, Severo M, Guimaraes MJ, Ferreira MA. A novel collaborative e-learning platform for medical students –ALERT STUDENT. BMC Med Educ 2014; 14: 143.
• 59 Yui BH, Jim WT, Chen M, Hsu JM, Liu CY, Lee TT. Evaluation of computerized physician order entry system-a satisfaction survey in Taiwan. J Med Syst 2012; 36 (06) 3817-3824.
• 60 Amrein K, Kachel N, Fries H, Hovorka R, Pieber TR, Plank J, Wenger U, Lienhardt B, Maggiorini M. Glucose control in intensive care: usability, efficacy and safety of Space GlucoseControl in two medical European intensive care units. BMC Endocr Disord 2014; 14: 62.
• 61 Ratwani RM, Hettinger AZ, Fairbanks RJ. Barriers to comparing the usability of electronic health records. J Am Med Inform Assoc. 2016 DOI:10.1093/jamia/ocw117
• 62 21st Century Cures Act, H.R.6. To accelerate the discovery, development, and delivery of 21st century cures, and for other purposes; 2015.
• 63 Hesselink G, Kuis E, Pijnenburg M, Wollersheim H. Measuring a caring culture in hospitals: a systematic review of instruments. BMJ Open 2013; 3 (09) e003416.
• 64 Cox CE, Wysham NG, Walton B, Jones D, Cass B, Tobin M, Jonsson M, Kahn JM, White DB, Hough CL, Lewis CL, Carson SS. Development and usability testing of a Web-based decision aid for families of patients receiving prolonged mechanical ventilation. Ann Intensive Care 2015; 5: 6.
• 65 Im EO, Chee W. editors. Evaluation of the decision support computer program for cancer pain management. Oncology nursing forum 2006. Pittsburgh, PA: Oncology Nursing Society;
• 66 Kim MS, Shapiro JS, Genes N, Aguilar MV, Mohrer D, Baumlin K, Belden JL. A pilot study on usability analysis of emergency department information system by nurses. Appl Clin Inform 2012; 3 (01) 135-153.
• 67 Kobak KA, Stone WL, Wallace E, Warren Z, Swanson A, Robson K. A web-based tutorial for parents of young children with autism: results from a pilot study. Telemed J E Health 2011; 17 (10) 804-808.
• 68 Levin ME, Pistorello J, Seeley JR, Hayes SC. Feasibility of a prototype web-based acceptance and commitment therapy prevention program for college students. J Am Coll Health 2014; 62 (01) 20-30.
• 69 Lin CA, Neafsey PJ, Strickler Z. Usability testing by older adults of a computer-mediated health communication program. J Health Commun 2009; 14 (02) 102-118.
• 70 Saleem JJ, Haggstrom DA, Militello LG, Flanagan M, Kiess CL, Arbuckle N, Doebbeling BN. Redesign of a computerized clinical reminder for colorectal cancer screening: a human-computer interaction evaluation. BMC Med Inform Decis Mak 2011; 11: 74.
• 71 Schnall R, Cimino JJ, Bakken S. Development of a prototype continuity of care record with context-specific links to meet the information needs of case managers for persons living with HIV. Int J Med Inform 2012; 81 (08) 549-555.
• 72 Schutte J, Gales S, Filippone A, Saptono A, Parmanto B, McCue M. Evaluation of a telerehabilitation system for community-based rehabilitation. Int J Telerehabil 2012; 4 (01) 15-24.
• 73 Sharma P, Dunn RL, Wei JT, Montie JE, Gilbert SM. Evaluation of point-of-care PRO assessment in clinic settings: integration, parallel-forms reliability, and patient acceptability of electronic QOL measures during clinic visits. Qual Life Res 2016; 25 (03) 575-583.
• 74 Shoup JA, Wagner NM, Kraus CR, Narwaney KJ, Goddard KS, Glanz JM. Development of an Interactive Social Media Tool for Parents With Concerns About Vaccines. Health Educ Behav 2015; 42 (03) 302-312.
• 75 Staggers N, Kobus D, Brown C. Nurses’ evaluations of a novel design for an electronic medication administration record. Comput Inform Nurs 2007; 25 (02) 67-75.
• 76 Stellefson M, Chaney B, Chaney D, Paige S, Payne-Purvis C, Tennant B, Walsh-Childers K, Sriram P, Alber J. Engaging community stakeholders to evaluate the design, usability, and acceptability of a chronic obstructive pulmonary disease social media resource center. JMIR Res Protoc 2015; 4 (01) e17.
• 77 Bozkurt S, Zayim N, Gulkesen KH, Samur MK, Karaagaoglu N, Saka O. Usability of a web-based personal nutrition management tool. Inform Health Soc Care 2011; 36 (04) 190-205.
• 78 Fritz F, Balhorn S, Riek M, Breil B, Dugas M. Qualitative and quantitative evaluation of EHR-integrated mobile patient questionnaires regarding usability and cost-efficiency. Int J Med Inform 2012; 81 (05) 303-313.
• 79 Meldrum D, Glennon A, Herdman S, Murray D, McConn-Walsh R. Virtual reality rehabilitation of balance: assessment of the usability of the Nintendo Wii((R)) Fit Plus. Disabil Rehabil Assist Technol 2012; 7 (03) 205-210.
• 80 Memedi M, Westin J, Nyholm D, Dougherty M, Groth T. A web application for follow-up of results from a mobile device test battery for Parkinson‘s disease patients. Comput Meth Prog Bio 2011; 104 (02) 219-226.
• 81 Ozel D, Bilge U, Zayim N, Cengiz M. A web-based intensive care clinical decision support system: from design to evaluation. Inform Health Soc Care 2013; 38 (02) 79-92.
• 82 Oztekin A, Kong ZJ, Uysal O. UseLearn: A novel checklist and usability evaluation method for eLearning systems by criticality metric analysis. International Journal of Industrial Ergonomics 2010; 40: 455-469.
• 83 Stjernsward S, Ostman M. Illuminating user experience of a website for the relatives of persons with depression. Int J Soc Psychiatry 2011; 57 (04) 375-386.
• 84 Van der Weegen S, Verwey R, Tange HJ, Spreeuwenberg MD, de Witte LP. Usability testing of a monitoring and feedback tool to stimulate physical activity. Patient Prefer Adherence 2014; 8: 311-322.
• 85 Hao AT, Wu LP, Kumar A, Jian WS, Huang LF, Kao CC, Hsu CY. Nursing process decision support system for urology ward. Int J Med Inform 2013; 82 (07) 604-612.
• 86 Huang H, Lee TT. Evaluation of ICU nurses’ use of the clinical information system in Taiwan. Comput Inform Nurs 2011; 29 (04) 221-229.
• 87 Lacerda TC, von Wangenheim CG, von Wangenheim A, Giuliano I. Does the use of structured reporting improve usability? A comparative evaluation of the usability of two approaches for findings reporting in a large-scale telecardiology context. Journal of Biomedical Informatics 2014; 52: 222-230.
• 88 Lee TT, Mills ME, Bausell B, Lu MH. Two-stage evaluation of the impact of a nursing information system in Taiwan. Int J Med Inform 2008; 77 (10) 698-707.
• 89 Moattari M, Moosavinasab E, Dabbaghmanesh MH, ZarifSanaiey N. Validating a Web-based Diabetes Education Program in continuing nursing education: knowledge and competency change and user perceptions on usability and quality. J Diabetes Metab Disord 2014; 13: 70.
• 90 Peikari HR, Shah MH, Zakaria MS, Yasin NM, Elhissi A. The impacts of second generation e-prescribing usability on community pharmacists outcomes. Res Social Adm Pharm 2015; 11 (03) 339-351.
• 91 Sawka AM, Straus S, Gafni A, Meiyappan S, O’Brien MA, Brierley JD, Tsang RW, Rotstein L, Thabane L, Rodin G, George SR, Goldstein DP. A usability study of a computerized decision aid to help patients with, early stage papillary thyroid carcinoma in, decision-making on adjuvant radioactive iodine treatment. Patient Educ Couns 2011; 84 (02) e24-e27.
• 92 Heikkinen K, Suomi R, Jaaskelainen M, Kaljonen A, Leino-Kilpi H, Salantera S. The creation and evaluation of an ambulatory orthopedic surgical patient education web site to support empowerment. Comput Inform Nurs 2010; 28 (05) 282-290.
• 93 Jeon E, Park HA. Development of a smartphone application for clinical-guideline-based obesity management. Healthc Inform Res 2015; 21 (01) 10-20.
• 94 Li LC, Adam PM, Townsend AF, Lacaille D, Yousefi C, Stacey D, Gromala D, Shaw CD, Tugwell P, Back-man CL. Usability testing of ANSWER: a web-based methotrexate decision aid for patients with rheumatoid arthritis. BMC Med Inform Decis Mak 2013; 13: 131.
• 95 Joshi A, Amadi C, Trout K, Obaro S. Evaluation of an interactive surveillance system for monitoring acute bacterial infections in Nigeria. Perspect Health Inf Manag 2014; 11: 1f.
• 96 Albu M, Atack L, Srivastava I. Simulation and gaming to promote health education: Results of a usability test. Health Education Journal. 2015 74(2).
• 97 Daniels JP, King AD, Cochrane DD, Carr R, Shaw NT, Lim J, Ansermino JM. A human factors and survey methodology-based design of a web-based adverse event reporting system for families. Int J Med Inform 2010; 79 (05) 339-348.
• 98 Lewis JR. IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. Int J Hum-Comput Int 1995; 7 (01) 57-78.
• 99 Lewis JR. Psychometric evaluation of the post-study system usability questionnaire: The PSSUQ. Human Factors Society 36th Annual Meeting; Atlanta: Human Factors and Ergonomics Society; 1992; 1259-1263.
• 100 Chin JP, Diehl VA, Norman KL. editors. Development of an instrument measuring user satisfaction of the human-computer interface. SIGCHI conference on Human factors in computing systems. 1988
• 101 Wilkinson A, Forbes A, Bloomfield J, Fincham Gee C. An exploration of four web-based open and flexible learning modules in post-registration nurse education. Int J Nurs Stud 2004; 41 (04) 411-424.
• 102 Lewis JR. Psychometric evaluation of an after-scenario questionnaire for computer usability studies: The ASQ. ACM Sigchi Bulletin 1991; 23 (Suppl. 01) 78-81.
• 103 Brooke J. SUS: A quick and dirty usability scale. Usability evaluation in industry 1996; 189 (Suppl. 194) 4-7.
• 104 Lewis JR, Henry SC, Mack RL. Integrated office software benchmarks: A case study. In: Dapler D. editor. Interact 90 –3rd IFIP International Conference on Human-Computer Interaction. Cambridge; UK: North-Holland: 1990: 337-243.
• 105 Lewis JR. Psychometric evaluation of the computer system usability questionnaire: The CSUQ (Tech. Report 54.723). Boca Raton, FL: International Business Machines Corporation; 1992
• 106 Frytak J. Measurement. In: Kane R. editor. Understanding health care outcomes research. Gaithersburg, MD: Aspen Publications; 1997: 213-237.
• 107 Lewis JR, Sauro J. The Factor Structure of the System Usability Scale. In: Kurosu M. editor. Human Centered Design. Heidelberg: Springer; 2009: 94-103.
• 108 Streiner DL. Starting at the beginning: An introduction to coefficient alpha and internal consistency. J Pers Assess 2003; 80 (Suppl. 01) 99-103.
• 109 Bangor A, Kortum PT, Miller JT. An Empirical Evaluation of the System Usability Scale. Journal of Human–Computer Interaction 2008; 24 (Suppl. 06) 574-594.
• 110 Bland JM, Altman DG. Statistics notes: Cronbach‘s alpha. BMJ 1997; 314: 572.
• 111 Streiner DL, Norman GR. Health measurement scales 2ed. New York: Oxford University Press; 1995
• 112 Kass RA, Tinsley HEA. Factor analysis. J Leisure Res 1979; 11: 120-138.
• 113 Brooke J. SUS: a retrospective. Journal of usability studies 2013; 8 (Suppl. 02) 29-40.
• 114 Yen PY, Wantland D, Bakken S. Development of a Customizable Health IT Usability Evaluation Scale. AMIA Annu Symp Proc 2010; 2010: 917-921.
• 115 Yen PY, Sousa KH, Bakken S. Examining construct and predictive validity of the Health-IT Usability Evaluation Scale: confirmatory factor analysis and structural equation modeling results. J Am Med Inform Assoc 2014; 21 (e2) e241-8.
• 116 Sousa VE, Matson J, Lopez KD. Questionnaire Adapting Little Changes Mean a Lot. Western J Nurs Res. 2016 0193945916678212.
• 117 Lin MI, Hong RH, Chang JH, Ke XM. Usage Position and Virtual Keyboard Design Affect Upper-Body Kinematics, Discomfort, and Usability during Prolonged Tablet Typing. PLoS One. 2015 10(12): e0143585.
• 118 Koppel R, Leonard CE, Localio AR, Cohen A, Auten R, Strom BL. Identifying and quantifying medication errors: evaluation of rapidly discontinued medication orders submitted to a computerized physician order entry system. J Am Med Inform Assoc 2008; 15 (Suppl. 04) 461-465.
• 119 Ahmed A, Chandra S, Herasevich V, Gajic O, Pickering BW. The effect of two different electronic health record user interfaces on intensive care provider task load, errors of cognition, and performance. Crit Care Med 2011; 39 (Suppl. 07) 1626-1634.
• 120 Duong M, Bertin K, Henry R, Singh D, Timmins N, Brooks D, Mathur S, Ellerton C. Developing a physiotherapy-specific preliminary clinical decision-making tool for oxygen titration: a modified delphi study. Physiother Can 2014; 66 (Suppl. 03) 286-295.
• 121 Vuong AM, Huber Jr. JC, Bolin JN, Ory MG, Moudouni DM, Helduser J, Begaye D, Bonner TJ, Forjuoh SN. Factors affecting acceptability and usability of technological approaches to diabetes self-management: a case study. Diabetes Technol Ther 2012; 14 (Suppl. 12) 1178-1182.
• 122 Jokela T, Iivari N, Matero J, Karukka M. editors. The standard of user-centered design and the standard definition of usability: analyzing ISO 13407 against ISO 9241–11. The Latin American conference on Human-computer interaction; 2003. New York: ACM;
• 123 Anastasi A. Psychological testing. New York: Macmillan; 1988
• 124 Parke CS. Missing data. In: Parke CS. editor. Essential first steps to data analysis: Scenario-based examples using SPSS. Thousand Oaks: Sage Publications; 2012: 179.
• 125 Gaber J. Face validity. In: Salkind NJ. editor. Encyclopedia of Research Design. Thousand Oaks: Sage Publications; 2010: 472-475.
• 126 Johnson EK, Nelson CP. Utility and Pitfalls in the Use of Administrative Databases for Outcomes Assessment. The Journal of Urology 2013; 190 (Suppl. 01) 17-18.
• 127 Fratello J, Kapur TD, Chasan A. Measuring success: A guide to becoming an evidence-based practice. New York: Vera Institute of Justice; 2013
• 128 Greenhalgh J, Long AF, Brettle AJ, Grant MJ. Reviewing and selecting outcome measures for use in routine practice. J Eval Clin Pract 1998; 4 (04) 339-350.
• 129 Bowen DJ, Kreuter M, Spring B, Cofta-Woerpel L, Linnan L, Weiner D, Bakken S, Kaplan CP, Squiers L, Fabrizio C, Fernandez M. How we design feasibility studies. American Journal of Preventive Medicine 2009; 36 (Suppl. 05) 452-457.
• 130 Madrigal D, McClain B. Strengths and weaknesses of quantitative and qualitative research. UXmatters; 2012 Available from: http://www.uxmatters.com/mt/archives/2012/09/strengths-and-weaknesses-of-quantitative-and-qualitative-research.php