Download PDF
Appl Clin Inform 2012; 03(02): 164-174
DOI: 10.4338/ACI-2011-11-RA-0070
Research Article
Schattauer GmbH

Assessing Electronic Note Quality Using the Physician Documentation Quality Instrument (PDQI-9)

P.D. Stetson
1   Department of Biomedical Informatics, Columbia University
2   Department of Medicine, Columbia University
,
S. Bakken
1   Department of Biomedical Informatics, Columbia University
3   School of Nursing, Columbia University
,
J.O. Wrenn
4   Department of Medical Informatics, Vanderbilt University
,
E.L. Siegler
4   Division of Geriatrics and Gerontology, Weill Cornell Medical College
› Author Affiliations
Further Information

Correspondence to:

Peter D. Stetson, MD, MA
Columbia University Medical Center
650 West 168th Street
BB2–239
New York, NY 10032
Phone: 212.342.0029   

Publication History

received: 21 November 2011

accepted: 29 March 2012

Publication Date:
16 December 2017 (online)

Also available at
 
 PDF Download Permissions and Reprints

Summary

Objective: To refine the Physician Documentation Quality Instrument (PDQI) and test the validity and reliability of the 9-item version (PDQI-9).

Methods: Three sets each of admission notes, progress notes and discharge summaries were evaluated by two groups of physicians using the PDQI-9 and an overall general assessment: one gold standard group consisting of program or assistant program directors (n = 7), and the other of attending physicians or chief residents (n = 24). The main measures were criterion-related validity (correlation coefficients between Total PDQI-9 scores and 1-item General Impression scores for each note), discriminant validity (comparison of PDQI-9 scores on notes rated as best and worst using 1-item General Impression score), internal consistency reliability (Cronbach’s alpha), and inter-rater reliability (intraclass correlation coefficient (ICC)).

Results: The results were criterion-related validity (r = –0.678 to 0.856), discriminant validity (best versus worst note, t = 9.3, p = 0.003), internal consistency reliability (Cronbach’s alphas = 0.87–0.94), and inter-rater reliability (ICC = 0.83, CI = 0.72–0.91).

Conclusion: The results support the criterion-related and discriminant validity, internal consistency reliability, and inter-rater reliability of the PDQI-9 for rating the quality of electronic physician notes. Tools for assessing note redundancy are required to complement use of PDQI-9. Trials of the PDQI-9 at other institutions, of different size, using different EHRs, and incorporating additional physician specialties and notes of other healthcare providers are needed to confirm its generaliz-ability.


#

Keywords

Electronic health record - documentation - note - quality - instrument

 


#

Conflicts of interest

None

  • References

  • 1 DesRoches CM, Campbell EG, Rao SR, Donelan K, Ferris TG, Jha A. et al. Electronic health records in ambulatory care –a national survey of physicians. N Engl J Med 2008; 359 (01) 50-60.
    CrossrefPubMedSearch in Google Scholar
  • 2 Medicare and Medicaid programs; electronic health record incentive program.. Final rule. Fed Regist 2010; 75 (144) 44313-44588.
    PubMedSearch in Google Scholar
  • 3 Amarasingham R, Plantinga L, Diener-West M, Gaskin DJ, Powe NR. Clinical information technologies and inpatient outcomes: a multiple hospital study. Arch Intern Med 2009; 169 (02) 108-114.
    CrossrefPubMedSearch in Google Scholar
  • 4 Poon EG, Wright A, Simon SR, Jenter CA, Kaushal R, Volk LA. et al. Relationship between use of electronic health record features and health care quality: results of a statewide survey. Med Care 2010; 48 (03) 203-209.
    CrossrefPubMedSearch in Google Scholar
  • 5 Johnson SB, Bakken S, Dine D, Hyun S, Mendonca E, Morrison F. et al. An electronic health record based on structured narrative. J Am Med Inform Assoc 2008; 15 (01) 54-64.
    CrossrefPubMedSearch in Google Scholar
  • 6 Kirkland LR, Bryan CS. Osler’s service: a view of the charts. J Med Biogr 2007; 15 (Suppl. 01) 50-54.
    CrossrefPubMedSearch in Google Scholar
  • 7 Weed LL. The problem oriented record as a basic tool in medical education, patient care and clinical research. Ann Clin Res 1971; 3 (03) 131-134.
    PubMedSearch in Google Scholar
  • 8 Fries JF. Alternatives in medical record formats. Med Care 1974; 12 (10) 871-881.
    CrossrefPubMedSearch in Google Scholar
  • 9 Reiser SJ. The clinical record in medicine. Part 2: Reforming content and purpose. Ann Intern Med 1991; 114 (11) 980-985.
    CrossrefPubMedSearch in Google Scholar
  • 10 Burnum JF. The misinformation era: the fall of the medical record. Ann Intern Med 1989; 110 (06) 482-484.
    CrossrefPubMedSearch in Google Scholar
  • 11 Embi PJ, Yackel TR, Logan JR, Bowen JL, Cooney TG, Gorman PN. Impacts of computerized physician documentation in a teaching hospital: perceptions of faculty and resident physicians. J Am Med Inform Assoc 2004; 11 (04) 300-309.
    CrossrefPubMedSearch in Google Scholar
  • 12 Hartzband P, Groopman J. Off the record –avoiding the pitfalls of going electronic. N Engl J Med 2008; 358 (16) 1656-1658.
    CrossrefPubMedSearch in Google Scholar
  • 13 Hirschtick RE. A piece of my mind. Copy-and-paste. Jama 2006; 295 (20) 2335-2336.
    CrossrefPubMedSearch in Google Scholar
  • 14 Siegler EL, Adelman R. Copy and paste: a remediable hazard of electronic health records. Am J Med 2009; 122 (06) 495-496.
    CrossrefPubMedSearch in Google Scholar
  • 15 Thielke S, Hammond K, Helbig S. Copying and pasting of examinations within the electronic medical record. Int J Med Inform 2007; 76 (Suppl. 01) S122-S128.
    CrossrefPubMedSearch in Google Scholar
  • 16 Weir CR, Hurdle JF, Felgar MA, Hoffman JM, Roth B, Nebeker JR. Direct text entry in electronic progress notes. An evaluation of input errors. Methods Inf Med 2003; 42 (01) 61-67.
    Thieme ConnectPubMedSearch in Google Scholar
  • 17 O’Donnell HC, Kaushal R, Barron Y, Callahan MA, Adelman RD, Siegler EL. Physicians’ attitudes towards copy and pasting in electronic note writing. Journal of general internal medicine: official journal of the Society for Research and Education in Primary Care Internal Medicine 2009; 24 (01) 63-68.
    PubMedSearch in Google Scholar
  • 18 Gelzer R, Hall T, Liette E, Reeves MG, Sundby J, Tegen A. et al. Auditing copy and paste. J AHIMA 2009; Jan 80 (01) 26-9 quiz 31-2.
    PubMedSearch in Google Scholar
  • 19 Payne T, Patel R, Beahan S, Zehner J. The Physical Attractiveness of Electronic Physician Notes. AMIA Annu Symp Proc 2010: 622-626.
    PubMed
  • 20 Institute of Medicine.. Committee on Improving the Patient R Dick RS, Steen EB, Detmer DE. The computer-based patient record : an essential technology for health care. Washington, D. C.: National Academy Press; 1997
    Search in Google Scholar
  • 21 Romm FJ, Putnam SM. The validity of the medical record. Med Care 1981; 19 (03) 310-315.
    CrossrefPubMedSearch in Google Scholar
  • 22 Tufo HM, Speidel JJ. Problems with Medical Records. Medical care 1971; 9 (06) 509-517.
    CrossrefPubMedSearch in Google Scholar
  • 23 Rosenbloom ST, Crow AN, Blackford JU, Johnson KB. Cognitive factors influencing perceptions of clinical documentation tools. J Biomed Inform 2007; 40 (02) 106-113.
    CrossrefPubMedSearch in Google Scholar
  • 24 Stetson PD, Morrison FP, Bakken S, Johnson SB. Preliminary development of the physician documentation quality instrument. J Am Med Inform Assoc 2008; 15 (04) 534-541.
    CrossrefPubMedSearch in Google Scholar
  • 25 Wrenn JO, Stein DM, Bakken S, Stetson PD. Quantifying clinical narrative redundancy in an electronic health record. J Am Med Inform Assoc 2010; 17 (01) 49-53.
    CrossrefPubMedSearch in Google Scholar
  • 26 Aronsky D, Haug PJ. Assessing the quality of clinical data in a computer-based record for calculating the pneumonia severity index. J Am Med Inform Assoc 2000; 7 (01) 55-65.
    CrossrefPubMedSearch in Google Scholar
  • 27 Coakley FV, Heinze SB, Shadbolt CL, Schwartz LH, Ginsberg MS, Lefkowitz RA. et al. Routine editing of trainee-generated radiology reports: effect on style quality. Acad Radiol 2003; 10 (03) 289-294.
    CrossrefPubMedSearch in Google Scholar
  • 28 Efthimiadis EN, Hammond KW, Laundry R, Thielke SM. Developing an EMR simulator to assess users’ perception of document quality. Proc 43rd Hawaii Int Conf on System Sciences –2010 2010: 1-9.
    PubMed
  • 29 Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. J Am Med Inform Assoc 1997; 4 (05) 342-355.
    CrossrefPubMedSearch in Google Scholar
  • 30 Logan JR, Gorman PN, Middleton B. Measuring the quality of medical records: a method for comparing completeness and correctness of clinical encounter data. Proc AMIA Symp 2001: 408-412.
    PubMed
  • 31 Myers KA, Keely EJ, Dojeiji S, Norman GR. Development of a rating scale to evaluate written communication skills of residents. Acad Med 1999; 74 (Suppl. 10) S111-S113.
    CrossrefPubMedSearch in Google Scholar
  • 32 Hammond KW, Efthimiadis EN, Weir CR, Embi PJ, Thielke SM, Laundry RM. et al. Initial steps toward validating and measuring the quality of computerized provider documentation. AMIA Annu Symp Proc 2010 2010: 271-275.
    PubMed
  • 33 Bates DW. Getting in step: electronic health records and their role in care coordination. J Gen Intern Med 2010; 25 (03) 174-176.
    CrossrefPubMedSearch in Google Scholar
  • 34 Bates DW, Bitton A. The future of health information technology in the patient-centered medical home. Health Aff (Millwood) 2010; 29 (04) 614-621.
    CrossrefPubMedSearch in Google Scholar
  • 35 O’Malley AS, Grossman JM, Cohen GR, Kemper NM, Pham HH. Are electronic medical records helpful for care coordination? Experiences of physician practices. J Gen Intern Med 2010; 25 (03) 177-185.
    CrossrefPubMedSearch in Google Scholar
  • 36 Naik AD, Singh H. Electronic health records to coordinate decision making for complex patients: what can we learn from wiki?. Med Decis Making 2010; 30 (06) 722-731.
    CrossrefPubMedSearch in Google Scholar
  • 37 Meaningful Use Workgroup Request for Comments Regarding Meaningful Use Stage 2.. Health Information Technology Policy Committee; 2011; Available from: http://healthit.hhs.gov/media/faca/ MU_RFC%20_2011–01–12_final.pdf.
    PubMed
  • 38 Hammond KW, Helbig ST, Benson CC, Brathwaite-Sketoe BM. Are electronic medical records trustworthy? Observations on copying, pasting and duplication. AMIA Annu Symp Proc 2003: 269-273.
    PubMed

Correspondence to:

Peter D. Stetson, MD, MA
Columbia University Medical Center
650 West 168th Street
BB2–239
New York, NY 10032
Phone: 212.342.0029   

  • References

  • 1 DesRoches CM, Campbell EG, Rao SR, Donelan K, Ferris TG, Jha A. et al. Electronic health records in ambulatory care –a national survey of physicians. N Engl J Med 2008; 359 (01) 50-60.
    CrossrefPubMedSearch in Google Scholar
  • 2 Medicare and Medicaid programs; electronic health record incentive program.. Final rule. Fed Regist 2010; 75 (144) 44313-44588.
    PubMedSearch in Google Scholar
  • 3 Amarasingham R, Plantinga L, Diener-West M, Gaskin DJ, Powe NR. Clinical information technologies and inpatient outcomes: a multiple hospital study. Arch Intern Med 2009; 169 (02) 108-114.
    CrossrefPubMedSearch in Google Scholar
  • 4 Poon EG, Wright A, Simon SR, Jenter CA, Kaushal R, Volk LA. et al. Relationship between use of electronic health record features and health care quality: results of a statewide survey. Med Care 2010; 48 (03) 203-209.
    CrossrefPubMedSearch in Google Scholar
  • 5 Johnson SB, Bakken S, Dine D, Hyun S, Mendonca E, Morrison F. et al. An electronic health record based on structured narrative. J Am Med Inform Assoc 2008; 15 (01) 54-64.
    CrossrefPubMedSearch in Google Scholar
  • 6 Kirkland LR, Bryan CS. Osler’s service: a view of the charts. J Med Biogr 2007; 15 (Suppl. 01) 50-54.
    CrossrefPubMedSearch in Google Scholar
  • 7 Weed LL. The problem oriented record as a basic tool in medical education, patient care and clinical research. Ann Clin Res 1971; 3 (03) 131-134.
    PubMedSearch in Google Scholar
  • 8 Fries JF. Alternatives in medical record formats. Med Care 1974; 12 (10) 871-881.
    CrossrefPubMedSearch in Google Scholar
  • 9 Reiser SJ. The clinical record in medicine. Part 2: Reforming content and purpose. Ann Intern Med 1991; 114 (11) 980-985.
    CrossrefPubMedSearch in Google Scholar
  • 10 Burnum JF. The misinformation era: the fall of the medical record. Ann Intern Med 1989; 110 (06) 482-484.
    CrossrefPubMedSearch in Google Scholar
  • 11 Embi PJ, Yackel TR, Logan JR, Bowen JL, Cooney TG, Gorman PN. Impacts of computerized physician documentation in a teaching hospital: perceptions of faculty and resident physicians. J Am Med Inform Assoc 2004; 11 (04) 300-309.
    CrossrefPubMedSearch in Google Scholar
  • 12 Hartzband P, Groopman J. Off the record –avoiding the pitfalls of going electronic. N Engl J Med 2008; 358 (16) 1656-1658.
    CrossrefPubMedSearch in Google Scholar
  • 13 Hirschtick RE. A piece of my mind. Copy-and-paste. Jama 2006; 295 (20) 2335-2336.
    CrossrefPubMedSearch in Google Scholar
  • 14 Siegler EL, Adelman R. Copy and paste: a remediable hazard of electronic health records. Am J Med 2009; 122 (06) 495-496.
    CrossrefPubMedSearch in Google Scholar
  • 15 Thielke S, Hammond K, Helbig S. Copying and pasting of examinations within the electronic medical record. Int J Med Inform 2007; 76 (Suppl. 01) S122-S128.
    CrossrefPubMedSearch in Google Scholar
  • 16 Weir CR, Hurdle JF, Felgar MA, Hoffman JM, Roth B, Nebeker JR. Direct text entry in electronic progress notes. An evaluation of input errors. Methods Inf Med 2003; 42 (01) 61-67.
    Thieme ConnectPubMedSearch in Google Scholar
  • 17 O’Donnell HC, Kaushal R, Barron Y, Callahan MA, Adelman RD, Siegler EL. Physicians’ attitudes towards copy and pasting in electronic note writing. Journal of general internal medicine: official journal of the Society for Research and Education in Primary Care Internal Medicine 2009; 24 (01) 63-68.
    PubMedSearch in Google Scholar
  • 18 Gelzer R, Hall T, Liette E, Reeves MG, Sundby J, Tegen A. et al. Auditing copy and paste. J AHIMA 2009; Jan 80 (01) 26-9 quiz 31-2.
    PubMedSearch in Google Scholar
  • 19 Payne T, Patel R, Beahan S, Zehner J. The Physical Attractiveness of Electronic Physician Notes. AMIA Annu Symp Proc 2010: 622-626.
    PubMed
  • 20 Institute of Medicine.. Committee on Improving the Patient R Dick RS, Steen EB, Detmer DE. The computer-based patient record : an essential technology for health care. Washington, D. C.: National Academy Press; 1997
    Search in Google Scholar
  • 21 Romm FJ, Putnam SM. The validity of the medical record. Med Care 1981; 19 (03) 310-315.
    CrossrefPubMedSearch in Google Scholar
  • 22 Tufo HM, Speidel JJ. Problems with Medical Records. Medical care 1971; 9 (06) 509-517.
    CrossrefPubMedSearch in Google Scholar
  • 23 Rosenbloom ST, Crow AN, Blackford JU, Johnson KB. Cognitive factors influencing perceptions of clinical documentation tools. J Biomed Inform 2007; 40 (02) 106-113.
    CrossrefPubMedSearch in Google Scholar
  • 24 Stetson PD, Morrison FP, Bakken S, Johnson SB. Preliminary development of the physician documentation quality instrument. J Am Med Inform Assoc 2008; 15 (04) 534-541.
    CrossrefPubMedSearch in Google Scholar
  • 25 Wrenn JO, Stein DM, Bakken S, Stetson PD. Quantifying clinical narrative redundancy in an electronic health record. J Am Med Inform Assoc 2010; 17 (01) 49-53.
    CrossrefPubMedSearch in Google Scholar
  • 26 Aronsky D, Haug PJ. Assessing the quality of clinical data in a computer-based record for calculating the pneumonia severity index. J Am Med Inform Assoc 2000; 7 (01) 55-65.
    CrossrefPubMedSearch in Google Scholar
  • 27 Coakley FV, Heinze SB, Shadbolt CL, Schwartz LH, Ginsberg MS, Lefkowitz RA. et al. Routine editing of trainee-generated radiology reports: effect on style quality. Acad Radiol 2003; 10 (03) 289-294.
    CrossrefPubMedSearch in Google Scholar
  • 28 Efthimiadis EN, Hammond KW, Laundry R, Thielke SM. Developing an EMR simulator to assess users’ perception of document quality. Proc 43rd Hawaii Int Conf on System Sciences –2010 2010: 1-9.
    PubMed
  • 29 Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. J Am Med Inform Assoc 1997; 4 (05) 342-355.
    CrossrefPubMedSearch in Google Scholar
  • 30 Logan JR, Gorman PN, Middleton B. Measuring the quality of medical records: a method for comparing completeness and correctness of clinical encounter data. Proc AMIA Symp 2001: 408-412.
    PubMed
  • 31 Myers KA, Keely EJ, Dojeiji S, Norman GR. Development of a rating scale to evaluate written communication skills of residents. Acad Med 1999; 74 (Suppl. 10) S111-S113.
    CrossrefPubMedSearch in Google Scholar
  • 32 Hammond KW, Efthimiadis EN, Weir CR, Embi PJ, Thielke SM, Laundry RM. et al. Initial steps toward validating and measuring the quality of computerized provider documentation. AMIA Annu Symp Proc 2010 2010: 271-275.
    PubMed
  • 33 Bates DW. Getting in step: electronic health records and their role in care coordination. J Gen Intern Med 2010; 25 (03) 174-176.
    CrossrefPubMedSearch in Google Scholar
  • 34 Bates DW, Bitton A. The future of health information technology in the patient-centered medical home. Health Aff (Millwood) 2010; 29 (04) 614-621.
    CrossrefPubMedSearch in Google Scholar
  • 35 O’Malley AS, Grossman JM, Cohen GR, Kemper NM, Pham HH. Are electronic medical records helpful for care coordination? Experiences of physician practices. J Gen Intern Med 2010; 25 (03) 177-185.
    CrossrefPubMedSearch in Google Scholar
  • 36 Naik AD, Singh H. Electronic health records to coordinate decision making for complex patients: what can we learn from wiki?. Med Decis Making 2010; 30 (06) 722-731.
    CrossrefPubMedSearch in Google Scholar
  • 37 Meaningful Use Workgroup Request for Comments Regarding Meaningful Use Stage 2.. Health Information Technology Policy Committee; 2011; Available from: http://healthit.hhs.gov/media/faca/ MU_RFC%20_2011–01–12_final.pdf.
    PubMed
  • 38 Hammond KW, Helbig ST, Benson CC, Brathwaite-Sketoe BM. Are electronic medical records trustworthy? Observations on copying, pasting and duplication. AMIA Annu Symp Proc 2003: 269-273.
    PubMed

   Permissions and Reprints