From Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting, October 5-9, 1998
HUMAN ERROR IN PATIENT-CONTROLLED ANALGESIA: INCIDENT REPORTS AND EXPERIMENTAL EVALUATION
Laura Lin
ABSTRACT
Human error is often cited as the most common cause of medical device mishaps. The role that a poorly designed user interface plays in precipitating these errors is infrequently acknowledged. This study focused on the Abbott PCA Infuser, a commonly used medical device. A review of FDA incident reports showed that human error is responsible for 68% of fatalities and serious injuries associated with the Abbott PCA. We sought to demonstrate that the incidence of human error can be significantly reduced through a human factors approach to interface design. A redesigned interface was developed, then empirically evaluated with a group of recovery room nurses, experienced PCA users. The results of the evaluations showed a 55% reduction in errors, an 18% improvement in performance time, a 14% reduction in mental workload, and a strong preference by nurses for the redesigned interface. These findings demonstrate that quantifiable improvements in equipment safety and efficiency can be achieved by adopting a human factors approach to interface design.
INTRODUCTION
As many as 100,000 deaths or serious injuries occur each year in the U.S. as a result of medical accidents (Van Cott, 1993). It is believed that a significant number are related to the misuse of medical devices (Burlington, 1995; Carstenson, 1995). Human error is often cited as the most common cause of medical device mishaps. However, what is less frequently acknowledged is the notion that a poorly designed human-machine interface predisposes medical device operation to human error (Bogner, 1994a; Bogner 1994b; Hyman, 1994; Cooper et al., 1978). The hypothesis explored in this study is that human factors engineering can significantly improve the safety, efficiency, and ease of use of medical devices. A commonly used medical device, a Patient Controlled Analgesia (PCA) machine, specifically the Abbott Lifecare 4100 PCA pump, was selected as the testbed.
INCIDENT REPORTS
According to some estimates, 67% of all problems associated with PCA pumps are attributable to user error (Callan, 1990; Heath ~ Thomas, 1993), where "user errors" include errors made by nurses in setting up the PCA pump, e.g., "misprogramming" the drug concentration. A review of Medical Device Reports (MDRs) was conducted to gain more insight into the specific errors occurring with the Abbott PCA pump investigated in this study.
A total of 2,017 MDRs describing problems with the Abbott PCA pump were reviewed for a randomly chosen year, 1993. An analysis of these reports showed that out of the cases where the device was known to be connected to the patient (308 cases), a majority of incidents, 86%, were caused by device malfunction (i.e., mechanical or electrical problems), while only 7% were caused by human error (the remaining 7% had unknown or unrelated causes). However, the reliability of these figures to accurately portray the incidence of human error is likely undermined by the under reporting of human error. Therefore, a second analysis was conducted of only the incidents in which the patient suffered serious injury or death. The rationale behind this analysis was that incidents that are more serious in nature are more likely to be reported and not overlooked. The results showed that for the 19 incidents with serious adverse outcomes, human error was found to be twice as likely as a malfunction to be the cause: P(human error I injury or death) = 0.68 compared to P(malfunction l injury or death) = 0.32. This difference was found to be statistically significant (X2(1)=2.778, p<0~05 one-tailed).
Upon closer examination of the reports where human error was involved, 'programming errors' (the setup procedure that nurses follow to enter operating values) were found to be the most common type of human error in PCA use. Furthermore, the majority of programming errors involved setting an incorrect drug concentration, all of which led to an overdelivery of medication.
PREVIOUS EXPERIMENTAL EVALUATION
In order to elucidate the design deficiencies that may have led to user errors such as the programming errors reported in the MDRs, a study was conducted (tin et al., in press) in which the existing Abbott PCA interface was analyzed using cognitive task analysis and human factors design principles. This involved bench tests with the device, field studies in the post-surgical recovery room at a local hospital (The Toronto Hospital), and an information requirements analysis of the nurse's task of programming the Abbott PCA pump. The analysis identified design deficiencies such as complexity in the system dialogue structure, insufficient system state feedback, inconsistent control functionality, poor organization of controls, and confusing message displays (Figure 1(a) illustrates the current interface).
Based on the results of the analysis, the interface was redesigned (Figure l(b)). Major design changes included a significant simplification of the system dialogue structure, and redesigned message displays which incorporated a system menu showing users their progress through the programming sequence. Messages were also reworded and modified with menu displays to present the user with decision options in parallel. Finally, functions of controls were redefined to reflect changes in the system dialogue structure, and controls were spatially organized based on functionality. Lin et al (in press) provides a detailed coverage of the interface analysis and the features of the redesigned interface.
An experiment was conducted, with nursing students as the subjects, to compare the existing interface to the redesigned interface (Lin et al., in press). The results showed that the redesigned interface led to increased safety, improved efficiency, and reduced workload. However, we did not know if these results would generalize to professional nurses who already had extensive experience programming the Abbott PCA.
METHOD
To see if our previous results would generalize to a more experienced population, a second experimental evaluation was conducted. Our hypothesis was that the redesigned interface facilitates the task of PCA programming, allowing professional nurses to complete programming tasks faster with fewer programming errors and with less mental effort.
Subjects
Twelve recovery room nurses from The Toronto Hospital participated in the experiment. The nurses were experienced PCA users with an average of 5.15 years (standard deviation = 1. 19) of clinical experience with the current Abbott PCA model.
Materials
Computer simulations of the current and redesigned interfaces were developed for use in the experiment. The simulations ran on an IBM compatible 486DX 33 MHZ PC in a Windows 3.1 environment. Subject interaction with the interfaces was electronically recorded by the simulations for later data analysis.
Experimental Design
A 2x3x2x2 mixed design was used in which the within-subject factors included interface (old and new), programming task (PCA, Continuous, and PCA+Continuous tasks), and repetition (1st and 2nd repetition of the programming task). Order of training (old first and new first) was the between-subjects factor. Each subject completed a total of 6 programming tasks with each interface. The order of training and order in which subjects received the programming tasks were each counterbalanced.
Procedure
In subjects' first session, they were trained on how to use one of the PCA interfaces and were also given 3 practice programming tasks. In the testing that followed, subjects completed 6 more programming tasks similar to their practice tasks. In these tasks, subjects programmed the PCA interface given to them, entering the prescribed operating values specified on a Toronto Hospital PCA Order Form. Subjects were brought in for a second day, and were trained and tested on the alternate interface following the same procedure as the first session.
Performance Measures
During the test, three measures of performance were recorded: task completion time, workload ratings, and number of programming errors. Interaction with the simulations was electronically logged in order to capture task completion time and programming errors. Workload ratings were obtained from subjects after each programming task using the NASATLX. At the conclusion of the second session, subjects were interviewed for their comments and subjective preference.
Data Analysis
Task completion time and workload were analyzed using a four-way analysis of variance (ANOVA) in order to test for effects from the main factors: interface, task, repetition, and order of training. In addition, chi-squared (x2) tests were conducted on number of programming errors.
RESULTS
Nurses made significantly fewer programming errors with the new interface compared to the old. There was a total of 29 programming errors made with the old interface, and 13 errors made with the new interface (Figure 2). This 55% reduction in number of errors was found to be statistically significant (X2(1)=6.1, p<0.01 one-tailed). Furthermore, there were no errors in entering the drug concentration with the new interface, whereas with the old interface, 8 such errors were made, 3 of which were not detected and were left uncorrected.
The most frequent programming error made with the old interface involved selecting the wrong mode (11 errors, 9 of which were eventually corrected). With the new interface, only 3 such mode selection errors occurred, all of which were eventually corrected. The most frequent error with the new interface involved the administration of a bolus dose (8 errors, 2 of which were eventually corrected). Of the bolus dose errors that remained uncorrected, nurses had 'forgotten' to administer the bolus dose in 5 of the 6 cases.
In terms of task completion time, programming tasks were completed more quickly with the new interface than with the old by 11 out of the 12 nurses. The average time to complete programming tasks with the new interface was 18% faster (Figure 3), a difference that was found to be statistically significant (F(1,10)=12.17, p=0.0058).
Analysis of workload ratings showed that there was a 14% decrease in workload with the new interface over the old. This reduction in mental workload, however, was not found to be statistically significant (F(1,10)=1.06, p=0.33). Also, there was an approximately equal proportion of subjects rating a lower workload with the new interface as with the old (X2(1)=3.33, P>0.05 one-tailed). The only significant effects on workload came from an interface x task x repetition interaction (F(2,20)=8.62, p=0.002). The resulting effect of this interaction was a significant decrease in the workload reported for the new interface over 2 repetitions for 2 of the 3 programming tasks (Continuous and PCA+Continuous tasks), whereas the decrease was less substantial for the old interface (see Figure 4) .
In the interviews following the experiment, a majority of subjects who expressed a preference, preferred the new interface over the old (X2(l)=6~4, p<0.012 one-tailed). Nine nurses stated that they favored the new interface, 1 preferred the old, and 2 were neutral.
DISCUSSION
The results of this empirical evaluation replicate most of the results we had previously obtained with student nurses, confirming the hypothesis of this study. Measurements of nurses' performance with programming tasks indicate an overall improvement in performance with the redesigned interface compared to the existing Abbott interface.
The redesigned interface led to a 55% reduction in the total number of programming errors. Furthermore, there were no errors in setting the drug concentration with the redesigned interface, demonstrating a degree of resistance to the most culpable error found in the Medical Device Reports. In comparison, more than one quarter of the programming errors with the existing Abbott interface were errors in setting the drug concentration.
While a programming error in setting the drug concentration carries the most serious threat to patient safety, the most frequent programming error with the existing interface was in setting the mode, accounting for 11 of the 29 errors. Nurses were observed to be quite accustomed to setting the mode to 'PCA only', the mode typically prescribed for patients at The Toronto Hospital where the study was carried out. When faced with different programming tasks, such as prescriptions for 'Continuous' and 'PCA+ Continuous' modes, nurses often made an incorrect mode selection, but eventually recovered from 9 of the 11 errors. The most obvious explanation for this type of programming error lies in the characteristics of the subject population: the nurses' prior training and experience with the Abbott PCA interface created the propensity to commit these mode selection errors. However, the new interface appeared to be effective in overcoming subjects' habits, recording only 3 such mode errors, all of which were eventually corrected. That the new interface is less prone to mode errors is likely a result of it's menu display, providing the user with a global view of options for mode selection on a single screen.
The most common error with the new interface involved the bolus dose. Typically, nurses neglected to administer a bolus dose during the programming task, an error that is not unexpected since it is uncommon for nurses in the recovery room at The Toronto Hospital to administer a bolus dose during programming as was required of them during the experiment. This aspect of the experimental conditions may have played a role in the frequent occurrence of bolus dose errors.
Accompanying the reduction in programming errors with the redesigned interface was a statistically significant improvement in task completion time. Nurses were able to complete programming tasks with the redesigned interface 18% faster than with the existing interface, despite having no prior experience with the former and several years of experience with the latter. This improvement can be attributed to not only the increased clarity of messages and functional organization of the control panel, but also to the fact that there are fewer steps in the new dialogue structures for the redesigned interface. Furthermore, with significantly fewer programming errors being made, less time was wasted recovering from errors.
The workload associated with the redesigned interface was found to be 14% lower than that associated with the current interface. Although subjective mental workload was the only measure that did not yield statistical significance, this result should be interpreted within the context of experience. Despite having less familiarity and experience with the new interface, nurses reported workload levels similar to that associated with the existing interface.
In addition to the fact that a low level of mental workload was achieved with minimal training on the new interface, workload for unfamiliar tasks decreased significantly with practice on the new interface. An interaction between interface, task, and repetition led to a more substantial reduction in workload over two repetitions for the new interface with the Continuous and PCA+ Continuous tasks compared to that for the old interface. This suggests that with the redesigned interface, nurses found it progressively easier to program these unfamiliar modes. Meanwhile, the workload reduction was less pronounced with the old interface, and even increased for the Continuous mode, despite the advantage of familiarity and experience with the old interface.
Finally, post-experiment interviews with the nurses showed that an overwhelming majority (90%) of nurses who expressed a preference, preferred the redesigned over the current interface.
In summary, all three performance measures provided converging evidence that the redesigned interface facilitates PCA programming: (i) there were significantly fewer errors made with the redesigned interface compared to the existing one; (ii) programming times were faster with the redesigned interface; and (iii) subjects reported approximately equal levels of workload despite several years of experience with the existing interface. Finally, a majority of subjects expressed a preference for the redesigned interface over the current PCA interface.
CONCLUSION
Findings from our analysis of incident reports revealed that human error is the leading cause of serious injuries and deaths associated with PCA use, underscoring the need for improving device safety. Motivated by this finding and the prevalence of human error in medical accidents in general, this study has demonstrated that adopting a human factors approach to the design of a PCA pump can lead to improved safety, efficiency, and ease of use. As far as we know, this is the only published study where a commercially available medical device has undergone a complete human factors analysis, redesign, and evaluation with clinical users. From a broad perspective, the results of this study lend support to the prospect of applying these methods to other devices to improve medical device safety.
ACKNOWLEDGMENTS
Many thanks are due to my graduate advisors, Kim Vicente and John Doyle, who have been exceptional sources of knowledge and guidance throughout the course of this research. I would also like to extend my gratitude to the recovery room nurses at The Toronto Hospital for all their time and invaluable input. This research was sponsored by research and equipment grants from the Natural Sciences and Engineering Research Council of Canada.
REFERENCES
Bogner, M.S. (1994a). Human Factors and Medical Devices: Lack of Feedback. FDA User Facility Reporting Bulletin, At, 1 -8.
Bogner, M.S. (1994b). Technology, Human Error, and Standards. In Proceedings of the Meeting for Health Care Technology Policy II (pp. 374-384). Arlington, VA: The International Society for Optical Engineering.
Burlington, B. (1995). Human Factors and the FDA's Goals: Improved Medical Device Design. In Proceedings of the AAMI/FDA Conference: Human Factors in Medical Devices: Design, Regulation, and Safety [Online]. Arlington, VA: Association for the Advancement of Medical Instrumentation. Available URL: http://www.fda.gov/cdrh/humfac/hufacimp.html
Callan, C.M. (1990). Analysis of Complaints and Complications with Patient-Controlled Analgesia. In F.M. Ferrante, G.W. Ostheimer, & B.G. Covino (Eds.), Patient-Controlled Analgesia (pp. 139-150). Boston: Blackwell Scientific Publications.
Carstenson, P.B. (1995). Overview of FDA's New Human Factors Plan: Implications for the Medical Device Industry. In Proceedings of the AAMI/FDA Conference: Human Factors in Medical Devices: Design. Regulation. and Patient Safety [Online]. Arlington, VA: Association for the Advancement of Medical Instrumentation. Available URL: http://www.fda.gov/cdrh/humfac/ hufacimp.html
Cooper, J.B., Newbower, R.S., Long, C.D. &: McPeek, B. (1978). Preventable Anesthesia Mishaps: A Study of Human Factors. Anesthesiology, 49, 399-406.
Heath, M.L., Thomas, VJ. (1993). Patient-Controlled Analgesia: Confidence in Postoperative Pain Control. Oxford: Oxford University Press.
Hyman, W.A. (1994). Errors in the Use of Medical Equipment. In Bogner (Ed.), Human Error in Medicine (pp. 327-347). New Jersey: Erlbaum Associates.
Lin, L., Isla, R., Doniz, D., Harkness, H., Vicente, K.J., Doyle, D.J. (in press). Applying Human Factors to the Design of Medical Equipment: Patient-Controlled Analgesia. Journal of Clinical Monitoring and Computing.
Van Cott, H.P. (1993). Human Error in Health Care Delivery: Cases, Causes, and Correction. In Proceedings of the Human Factors and Ergonomics Society (pp. 430-434). Santa Monica, CA: Human Factors and Ergonomics Society.
Copyright © 1998 by Human Factors and Ergonomics Society. All rights reserved. For a complete copy (with tables and figures), contact Lois Smith at hfes@compuserve.com.