Assessment of Surgical Competency

Otolaryngol Clin N Am 40 (2007) 1237–1259

Assessment of Surgical Competency
Terance T. Tsue, MD, FACSa,*, James W. Dugan, PhDb, Brian Burkey, MDc
Department of Otolaryngology–Head and Neck Surgery, University of Kansas School of Medicine, 3901 Rainbow Boulevard, Mail Stop #3010, Kansas City, KS 66160, USA b Counseling and Educational Support Services, University of Kansas School of Medicine, 3901 Rainbow Boulevard, Kansas City, KS 66160, USA c Department of Otolaryngology–Head and Neck Surgery, Vanderbilt University Medical Center, 7209 Medical Center East-South Tower, 1215 21st Avenue South, Nashville, TN 37232-8605, USA
a

Assessment of surgical competency in training and practice is an important issue confronting modern medicine. Even beyond the desire to educate competent otolaryngology–head and neck surgery (OTOHNS) residents, a focus on outcomes assessment has spread to other aspects of the health care system, influencing how and where medicine is practiced and how physicians are compensated. The Joint Commission of Accreditation of Health Care Organizations has incorporated requirements relating to competency in the general competencies [1]. Assessment of medical staff and resident staff competency is now an important focus of the hospital accreditation process. Such measures are becoming increasingly visible in licensing and credentialing procedures as well. Practitioner ‘‘performance’’ measures are being developed through efforts by the Centers for Medicare and Medicaid Services Pay-for-Performance initiative. Providers will receive payment differential incentives to assess patient care quality and use that information to improve overall patient care. Commercial insurance companies are investigating similar approaches. This increasing focus on competency has also been embraced in the lay community. Patients are more medically educated owing to the use of the Internet. Public confidence in technical performance in surgery has come under increased scrutiny, exacerbated by various highly publicized cases that suggested poor outcomes were the result of inadequate technical proficiency [2]. The increasing importance on a complication-free learning environment

* Corresponding author. E-mail address: [email protected] (T.T. Tsue). 0030-6665/07/$ - see front matter Ó 2007 Elsevier Inc. All rights reserved. doi:10.1016/j.otc.2007.07.005

oto.theclinics.com

1238

TSUE

et al

was emphasized by the British Royal Colleges of Surgery in the reply to the General Medical Council’s determination on the Bristol case, wherein they state: ‘‘there should be no learning curve as far as patient safety is concerned’’ [3]. The malpractice crisis has also spread to include some suits alleging residency program ‘‘educational malpractice’’ and responsibility of program directors for purported resident graduate negligence [4]. Reprisal litigation from residents dissatisfied with or terminated from their training programs also beckons the need for validated objective assessments during training. This focus on outcomes has also spread into the way residents are taught, evaluated, and certified. In 2001, the Accreditation Council for Graduate Medical Education (ACGME) initiated its Outcomes Project [5]. This long-term initiative focuses on the educational outcomes of residency training programs rather than the previous emphasis on the ‘‘potential’’ for a program to educate its residents through an organized curriculum and compliance with specified program requirements. The ACGME accreditation process has shifted from verifying program components to verifying the program’s educational product. At a minimum, programs are mandated to use assessments of their educational outcomes to continuously improve their educational product: a resident graduate competent in all six of the ACGME general competencies. This programmatic feedback process involves many levels of assessment beyond measuring just resident knowledge, skills, and attitudes; it also may require evaluating graduate, faculty, patient, departmental, and institutional outcomes. Residency programs are expected not only to consider aggregate learner performance data (eg, percentile ranking on in-training exams, first-attempt certification exam pass rate), but also external program performance measures. These ‘‘external indicators’’ are not yet defined for OTOHNS programs, but can include metrics like clinical quality measures, patient survey results, and complication rates. Although it is expected that such changes to residency program evaluation will be a dynamic, evolving process, documentation of the feedback loop will be necessary for both program and institutional accreditation. Finally, similar information will likely be required in the future as a component of the maintenance of certification process developed by the American Board of Otolaryngology (ABOto). The thrust toward board maintenance of certification requirements is congruent with the sentiments for continued measurement of physician competency. Although the ACGME has placed the focus on educational outcomes and not clinical outcomes, there is obvious significant overlap. All of these interrelated forces, both public and within the medical profession itself, have highlighted the need for valid assessments of trainees’ competency as surgical specialists. Although the thorough evaluation of competency in all areas of a physician’s practice by a feasible, reliable, and valid assessment process is important, at the core of an OTOHNS practice is surgical competency. Surgical competency obviously involves more than just doing

ASSESSMENT OF SURGICAL COMPETENCY

1239

the operation. Darzi and Mackay [6] describe the four essential components or framework of surgical care in which a surgeon must be competent: diagnostic ability, treatment plan formulation, technical skill performance, and postoperative care. All of these components involve competency in cognitive and personality skills such as decision making/judgment, knowledge, communication, teamwork, and leadership [7]. Thus, surgical competency requires competency in all of the six ACGME general competencies and not just patient care. Technical skill performance, of all areas of surgical care, has been the most challenging in terms of objective assessment. Within OTOHNS itselfdlet alone any other surgical fielddthese skills remain variable in terms of their nature and complexity. The current and potential future solutions to the challenge of evaluating this component of surgical competency remain the focus of this article. High-stakes assessments in other fields The field of surgery depends on a foundation of technical skill that is enhanced by technology, knowledge, and judgment. Other fields also depend on this foundation as a basis and have potentially high-stakes outcomes that can affect life, limb, or property. Such fields are well ahead of medicine with regard to expecting certification of core skill competency. High-risk industries include aviation, nuclear power, chemical manufacturing, and offshore oil drilling [8]. These industries employ a number of methods to assess competency that include objective observational assessment by a supervisor/trainer (the most common), written and oral examinations, electronic simulations, and behavior marker systems that rate trainee behavior during a simulation. All of these methods of competency assessment used in the high-risk industries have been adapted to some degree in the assessment of surgical competency. However, what is distinctive in high-risk industries, especially aviation and nuclear power, is that technical competency is regularly assessed in simulations of both routine and emergency situations. In assessing technical competency in these high-stakes industries, declarative knowledge or knowing the technical skill is the first part of this assessment. Subsequently, knowing how to put that technical skill into safe practice in a simulation must be demonstrated to achieve competency. Consequently, these industries place significant focus on the evaluation of the impact of ‘‘soft,’’ or nontechnical, skills on the performance of technical skills. Examples of these nontechnical skills are leadership, teamwork, assertiveness, communication, and situational awareness. These nontechnical skills have been described as the underpinnings of technical proficiency, and lapses in these nontechnical skills have often been identified as the root cause of technical error [9]. A classic example of this is when, in 1988, the crew of the USS Vincennes, a guided missile cruiser equipped with the latest technology, mistakenly shot down an Iranian commercial airline with 290 passengers on

1240

TSUE

et al

board. The root-cause analysis of this deadly error identified the crew’s increased stress and fatigue that contributed to poor decision making, communication, and teamwork [10]. To measure competency in the integration of nontechnical skills during a technical procedure, behavioral marker systems have been developed and are used widely in these industries. These assessments allow a qualified trainer to identify and rate the behavior of the trainee during a simulation. For almost 25 years, the aviation industry has used a behavioral marker system called crew resource management, which has been shown to significantly improve aviation safety. In the last few years, two behavioral marker systems have been developed for training residents in anesthesiology and surgery: Anesthetists’ NonTechnical Skills and Surgeons Non-Technical Skills [11,12]. These behavioral marker systems identify elements of behavior such as communication, teamwork, situational awareness, and decision making. Currently, the use of simulations and behavior marker systems, though certainly demonstrating their value in the aviation and nuclear power industries, presents considerable cost and time challenges for inclusion in residency training.

Assessment: ideal and reality Assessment is defined as the ‘‘process of collecting, synthesizing, and interpreting information to aid decision-making’’ [13]. In medical training, assessments are used to measure progress of residents toward attainment of the program goals and objectives (‘‘formative’’ evaluation), which ultimately lead to a level of competency and beyond for program-defined outcomes (‘‘summative’’ evaluation) [14]. The assessment process includes not only the ‘‘test’’ or collection of data, but also applicable analysis and interpretation of the data, followed by communication and feedback to both evaluator and learner. This ‘‘feedback loop’’ should improve the educational efficiency for the learner, including directing him or her to priority areas of self-study and curricular focus. For the teacher, feedback should prompt pinpoint refinement of the curriculum. This process should result in a constantly improving education product or outcome. The ideal assessment method should be feasible, requiring minimal cost, time, and effort by both learner and evaluator to complete and analyze. The ideal tool should be simple, be useful for all levels of training and expertise, and assess all areas of the field. Objectivity and anonymity would be provided as well as assessment reliability (consistency or reproducibility of measurement) and validity (instrument truly measures what it is supposed to measure) in a standardized and reproducible testing environment. The assessment metric would be quantifiable, with competency lying in the middle of the scaled score. Finally, the ideal assessment would provide immediate informative feedback to direct both specific individual and programmatic changes.

ASSESSMENT OF SURGICAL COMPETENCY

1241

In reality, there is no ‘‘ideal’’ assessment that fulfills all of the above requirements. There is no assessment that evaluates all of the objectives or outcomes that need to be measured. Thus, difficult choices must be made about what can realistically be assessed. Progression through the OTOHNS residency has classically been based on the apprenticeship model, relying on the traditional graded-responsibility experienced-based model. The main feature of this model is a teacher-centered approach based on loosely structured, one-on-one supervised situations where principles are taught and the learner is assessed on the basis of the teacher’s interpretation of current standards of practice [15]. This traditional approach has helped to exacerbate the current ‘‘reality’’ of the limitations of today’s surgical competency assessment techniques. Progress and eventual graduation rely on subjective evaluations by faculty. This requires accurate evaluator recall of past intermittent and widely varied events and generally stems from an overall ‘‘gestalt’’ rather than any objective measures. Anonymity remains difficult for OTOHNS programs because of their smaller size, making concerns or threat of retaliation a reality. The number of faculty evaluators on a given rotation is even smaller, and each has a potentially different definition of competency. Additionally, the influence of resident duty-hour limitations, decreased clinical reimbursements, a continuing trend toward superspecialization, and a focus on increasing health care resource efficiency has also hampered progress toward an ‘‘ideal’’ assessment system. These influences decrease the amount of educational resources availablednamely moneyd and faculty and student time. The comparably rapid expansion of knowledge, technology, and techniques within OTOHNS not only tends to use these already limited resources at a faster rate, but also provides a moving target in terms of what needs to be evaluated.

Making it feasible Incorporating a feasible assessment system, even given the above-described constraints and challenges, is a realistic and necessary goal. Box 1 summarizes some general steps that can help with the implementation of an efficient evaluation process. Incorporating these steps within an otolaryngology residency program is discussed below. First, delineate the minimum requirements needed. The ACGME Common Program requirements delineate the necessary minimum assessment methods for residency programs, but these minimums may be influenced by the JCAHO and local credentialing requirements as well. Current recommendations include the use of an end-of-rotation global assessment tool and at least one other method. As most of the technical skill component of surgical competency falls under the patient care competency, recommendations suggest a focused assessment method such as direct observation and concurrent evaluation. Use of multiple assessment methods to measure technical

1242

TSUE

et al

Box 1. Incorporating feasible assessments 1. Know the minimum assessment requirements 2. Identify available resources and limitations 3. Adopt or adapt currently used or available assessment methods 4. Involve multiple evaluators and assess at performance milestones 5. Educate both the learner and the evaluator 6. Use the latest electronic technology available skill reduces the subjectivity of the process as well as overcoming the different limitations inherent in each particular method. Second, it is important to identify what resources are available for the assessment system. Limitations on learner and evaluator time as well as available personnel, equipment, facilities, and funds that can be dedicated to the activity need to be determined. Coordination across surgical subspecialties is an excellent way to facilitate availability of more resource-intensive assessment methods. Even mobility and sharing of techniques between OTOHNS programs are possible and certainly would add to a particular method’s attractiveness. Third, use and/or adapt assessment methods currently in use. This not only reduces the sense of ‘‘change’’ by evaluator and learner, but also saves significant implementation time and effort. Additionally, there may be proven external performance measures that are currently used by the university or department that can be easily adapted for learner assessment (eg, quality control measures, Press Ganey patient surveys). Additionally, proven assessment methodologies from other fields that also require competency evaluation of high-stakes skills are potential resources for adoption (see below). Fourth, use multiple evaluators and perform the assessment at multiple performance milestones throughout the training program. Engaging different evaluators spreads out the responsibility and should not influence the outcome of a reliable assessment method. Focusing on specific program milestones, and spreading the assessments out over the 5-year training period, should improve the usefulness of the evaluation outcomes by matching a specific skill with its assessment. If assessments can be combined with the learning activity, the efficiency of the process should be even higher. This can even extend to involving learners in the development and application of the assessment process. Learners who have achieved a level of competency can assess those still progressing toward that goal. This interaction should educationally benefit both parties. In contrast, routine assessment that is temporally based, such as after each rotation or academic year, and not related to a specific milestone level, can dilute this feedback efficacy.

ASSESSMENT OF SURGICAL COMPETENCY

1243

Fifth, educate both the evaluator and the learner about the assessment tools and processes. Providing specific objective definitions of assessment levels, such as what ‘‘competent’’ or ‘‘satisfactory’’ means, should improve the usefulness and applicability of the tool across learners. Learners are then measured against a known scale rather than against each other. This can also allow more self-assessment by the resident, as the objectives are well known and defined, potentially guiding more independent study and practice. Sixth, use the latest technology available to administer the assessment as well as collect and analyze evaluation results. Electronically administered tools are easier, especially for the technologically advanced, and can be accessed from nearly anywhere when an evaluator has time to complete the process. Less completion time required should increase compliance while simultaneously allowing faster analysis and shorter time to feedback. Closing the feedback loop The quality of an assessment is only as good as its ability to effect change. Closing the feedback loop in a constructive manner that prompts both learner and teacher to improve is difficult. The nature of the output of the planned assessment method needs to be an important factor in its selection as an evaluation tool. The output data need to be easily analyzed. Ten thousand data points are not readily understandable in raw form, and it can be time consuming to analyze the data down statistically. Assessment results that are able to point to specific skill components (ie, ‘‘competent at recurrent laryngeal nerve identification’’) are more useful than ones that give generalized results (ie, ‘‘competent at thyroidectomy’’). A quantitative measure eases comparisons and allows linear comparison in a longitudinal fashion throughout the resident’s tenure. This measure also allows peer comparisons. A graph can be helpful to visually demonstrate performance trends over time. As stated previously, aiming for a competency level in the middle of the quantitative measure is most useful. Performance feedback (formative evaluation) is an important component of the evaluation process. Confidential and organized meetings between the program director and resident are most effective, especially those done in a timely fashion after the assessment. Even if the feedback is delayed until the requisite semi-annual program director meetings, a simple, focused, and understandable presentation to the resident can still yield change. Those residents behind their peers may need more frequent meetings, and feedback should be adjusted to their needs. Prioritizing specific results (ie, ‘‘top three’’ and ‘‘bottom three’’ lists) can help a learner remember and focus on where effort needs to be applied. Recognizing positive improvement is beneficial for many obvious reasons. Residents should also be given the opportunity to provide feedback on the assessment methods themselves. An essential component of feedback for the program director is providing guidance to areas of need. Priority areas, should be accompanied by

1244

TSUE

et al

suggested solutions and approaches. Several approaches should be discussed and documented, as learning styles can differ between residents. An associated timeline for improvement and expectations of future assessment results should also be negotiated. Key to this success is mentorship and accountability between assessments. Mutually selecting a faculty member other than the program director to provide nonthreatening oversight, daily guidance, and objective progress reports is paramount for those with significant need for improvement. Some evaluation results that are a widespread problem may not only require individual feedback, but most likely will also require tangible programmatic and curricular change.

Available assessment methods Many skills assessments have been or are potentially adaptable to the assessment of the OTOHNS resident. However, data on assessment tool reliability and validity in the OTOHNS or in other surgical fields are sparse and/or premature. Some methods are more established in other fields, and some are only research tools at this time. Written examinations, such as the Board Certification and In-Training Examination, are good assessors of surgical knowledge but are not intended to measure technical proficiency. These evaluation methods do not necessarily correlate with technical skill and operative performance [16]. Thus, other methods must be employed to measure all aspects of surgical competency. Surgical technical proficiency can be broken down into dexterity, spatial orientation, and operative flow [17]. Dexterity further involves the psychomotor aspects of tasks, tissue handling/respect for tissue, and economy/fluency of movements [6]. Objective assessments should include these components to fully measure competency. The following methods are currently available assessments and are foci of contemporary practice and study: 1. 2. 3. 4. 5. 6. 7. 8. Direct observation Resident case log reports/key indicator operative procedures (KIP) End-of-rotation global assessments Objective structured assessment of technical skill (OSATS) Final product analysis Hand motion analysis Simulation Possible new techniques

Direct observation This method involves a senior colleague, usually a faculty member, observing a learner during a surgical task. The observer then documents an

ASSESSMENT OF SURGICAL COMPETENCY

1245

opinion on competency based on these observations. Feedback during the observation is possible and can be immediate and effective. This process occurs daily in every residency program. Unfortunately, unlike most of the methods described below, the judgments derived from simple direct observation rely on unsystematic and unstructured observations. Without judgment criteria, the assessment can be vulnerable to many biases, including the faculty evaluator’s own memory. The assessment is usually relayed to the program director in a retrospective and generalized manner or through a signed resident procedure competency checklist. As different faculty have varied definitions of competency, this tends to yield unreliable and imprecise assessments [18]. However, physicians continue to rely on this process as a general method, as they make their judgments from faculty input and personal interactions, and then sign Board certification exam application materials, residency graduation certificates, and hospital and insurance company credentialing forms based on this information. Hopefully, incorporating some of the more reliable and valid methods described below will help physicians to make more objective and accurate assessments. Resident case log reports/key indicator operative procedures Another universally employed measure of progress toward surgical competency is the operative case log system. The cornerstone of surgical training has been one of apprenticeship, with graduation and competency believed to be resulting from a well-documented exposure and participatory experience in a graded, supervised environment. Not only is exposure important in building competency, but this system also requires progressive levels of responsibility. As defined by the ACGME, this progression starts as assistant surgeon, through resident surgeon, to (finally) resident supervisor surgeon. The implicit understanding is that progress toward and through competency with a given procedure follows these stages. Competency is requisite before moving to the supervisory level. Although all operative experiences are recorded by each resident, the ACGME has focused on select representative procedures, referred to as Key Indicator Procedures (KIP). Classically, these numbers have been used by the Residency Review Committee (RRC) to determine a program’s ability to provide their residents adequate experience in the various OTOHNS procedures. Acceptable levels have been determined by national percentile ranking for program graduates. Unfortunately, the use of percentiles always results in some programs falling below the desired threshold for experience. The electronically derived KIP report, available to both residents and program directors, helps the learner to focus on specific procedural and subspecialty areas of needed concentration. It also gives useful information as to the appropriate and balanced graded progression of operative experience, from assistant surgeon through resident supervisor. Although only national graduate means are available, keeping annual Post-Graduate Year KIP experience averages allows

1246

TSUE

et al

comparisons to be made to a resident’s predecessors in addition to a resident’s current peers. Unfortunately, changes in present-day medicine threaten the usefulness of this competency measure. These changes include duty-hour limitations, bottom-line–driven emphasis on operative time efficiency, and a desire to mitigate against medical errors. This has noticeably limited the level of resident autonomy in the operating room and probably hindered the efficiency of his or her technical development. The assumption essential to the use of operative log numbers as an assessment tool is that ‘‘adequate’’ experience results in surgical competency. Carr, through a survey of OTOHNS program directors, identified 16 procedures in which competency is achievable in the PGY-1 through PGY-5 levels [19]. In most of these procedures, the graduating residents’ mean operative experience was higher than the number thought necessary by the surveyed program directors to achieve competency. This assumption of jumping from experience to competency is hindered by many factors. Experience numbers do indicate an educational environment adequate or inadequate for the resident to ‘‘potentially’’ obtain competency. However, this assessment method lacks validity as it does not record quality of the operative experience [20]. Technically completing a procedure as resident surgeon or even as a resident supervisor in a supervised environment does not ensure the ability to perform that procedure independently. The levels of responsibility are subjectively recorded by the residents themselves, not the supervising faculty teacher. Despite rigid ACGME definitions, interpretations still vary. Self-assessment by surgeons is notoriously variable and generally overstates ability [21]. Also, adequate experience that results in competency will differ for each resident, because learning rates differ between residents and for different procedure types (eg, open versus microscopic). This variability is also affected by supervision and teacher nurturing bias. Time taken for a procedure has also been used as a measure of surgical performance. However, time also does not necessarily reflect the quality of the procedure and is generally unreliable due to the influence of many uncontrollable factors [22]. Morbidity and mortality data are often implied as surrogate markers of surgical performance outcome, but are heavily influenced by patient factors and probably do not reflect an accurate measure of surgical competency. End-of-rotation global assessments These assessments are also common and are frequently divided into sections corresponding to each of the six ACGME core competencies. Assessment of resident surgical technical skill applicably falls into the patient care core competency. A faculty rater scores a specific resident on very broad aspects of surgical skill, ranging from surgical efficiency and progression to handling of tissues. These scores generally are on a Likert-like scale using a scaling method to measure either a degree of positive or negative response

ASSESSMENT OF SURGICAL COMPETENCY

1247

to a statement. Some global assessments give qualitative performance statement examples to correspond with the numerical scale, whereas others, like the sample available on the ACGME Web site, provide statements for just the extreme scores [23]. Satisfactory or competency is the middle score;written comments are also allowed. These ratings are generally completed at the end of a specific rotation, which could be even a few months in duration. Evaluator scores are retrospectively derived from impressions and other sources, including memory of specific interactions or clinical outcomes. These global assessments can provide a quantitative summary of overall surgical skill that can be assessed longitudinally for a given resident and also used for comparison between peer residents. The results of many raters can be averaged as well. The forms are easy to construct and readily made available electronically to shorten completion and analysis time. Unfortunately, the large number of numerical data points over a resident’s tenure can rapidly become untenable, and is not as useful in a formative fashion to help guide a given resident toward a focal area of improvement. Additionally, these scales are subject to distortion from several causes, including central tendency bias (avoiding extreme response categories), acquiescence bias (agreeing with the presented statement), and/or social desirability bias (portrayal of faculty rater’s specific rotation in a favorable light). These distortions are even more significant and the tool less reliable with untrained evaluators. The ACGME rates this method as only a ‘‘potentially applicable method.’’ ‘‘Three Hundred and sixty degree’’ (360 ) global assessments, with competency evaluations and comments from staff who work side by side with the learner, can provide a more real-life assessment but are subject to the same limitations described above. As these assessors are usually not faculty (eg, scrub nurse), and thus not fully competent performers of the skill themselves, their expertise in assessing the learner and his or her biases usually focuses only on a particular aspect of the assessed skill. This assessment can still be useful and provide the basis or confirmation of informative feedback. The ACGME does rate this method a ‘‘next best method.’’ Objective structured assessment of technical skill The OSATS technique was developed by Reznick and colleagues [24] for general surgery use and is based on the original objective structured clinical examination (OSCE) method. The OSCE is increasingly used in medical schools, National Board of Medical Examiners licensing exams, and also many international certification and licensure boards [25]. The OSATS are being considered as a standard part of many board certification examinations to demonstrate technical competency [26]. During an OSATS, each learner rotates through a series of self-contained stations within a limited predetermined time. Each station is composed of a standardized surgical task (eg, laceration repair), and participants are assessed by the same trained observer in a standardized fashion using objective criteria. Thus, examiners

1248

TSUE

et al

are observers rather than interpreters of behavior, thereby minimizing the subjectivity of the evaluation process. The use of simulated models allows standardization and avoidance of the problem of finding adequate real patients. Cadaver sections can also be used, and Dailey and colleagues [27] described the use of laryngeal specimen stations to practice and assess both open and endoscopic laryngeal procedures. Reznick assessed the learners by using both a valid and reliable checklist and a global scoring sheet. The checklist is a series of 10–30 longitudinal ‘‘yes’’ or ‘‘no’’ items based on the specific task being assessed. This list includes the essential components for an ideally performed operation, and aims to reduce subjectivity of an evaluator’s specific experience. According to the ACGME Table of Suggested Best Methods for Evaluation, this checklist assessment is one of the most desirable methods of evaluating procedural skill. The disadvantages of the checklist method include the inability of the examiner to indicate that a particular task on the checklist was performed well but at an inappropriate stage. The global scoring sheet includes 5–8 overall performance measures, such as ‘‘flow of operation,’’ ‘‘instrument handling,’’ and ‘‘technique familiarity,’’ that are scored from 1 (poor) to 5 (excellent). As this global rating is not task specific, it has broader applicability, and has generally been shown to be a more effective discriminator than the checklist [24]. A separate performance score is derived for each station, and scores are generally combined across tasks to determine a pass/fail assessment. Several stations are recommended to provide a reliable performance measurement. The OSATS are useful in assessing technical skills in terms of knowledge and dexterity, but they cannot assess surgical judgment as easily. Also, checklists are task specific and therefore must be developed and validated for each task. Global rating forms, though more flexible, also tend to have a poorer faculty completion rate [28]. The OSATS are difficult to develop and administer due to their resource intensiveness (equipment, time, and manpower), and tend to be more useful for assessing simpler tasks and thus for assessing more junior trainees [29]. Limited resources in OTOHNS programs, which tend to be on the smaller size, can limit OSATS availability. However, this form of evaluation can be more cost effective if resources and expertise are shared between programs within an academic institution or a given geographic area. The OSATS use of inanimate procedural simulation (bench top) assessments has been shown to translate to actual surgical performance in the operating room (OR) [30,31]. So deconstructing an operation into its component parts can provide a simpler and less costly bench model for surgical competency. This should not replace eventual assessment within the OR, though. Roberson and colleagues [32] developed and validated an OSAT-based instrument to measure tonsillectomy performance. Their instrument was shown to be both reliable and valid and confirmed that the global rating evaluation will probably be a more meaningful and readily applicable tool for OTOHNS. Assessing a videotaped procedure with

ASSESSMENT OF SURGICAL COMPETENCY

1249

structured criteria can also be used, possibly providing a more favorable environment for trainee feedback. Such a system allows multiple step-by-step reviews with many learners and focused identification of specific errors. This method does have a higher cost in terms of materials and editing time, and does not necessarily improve on reliability or validity [20,33,34]. In contrast, by condensing the edited video, evaluator time should be decreased, and videotaping procedures allows for better learner anonymity, eliminating gender, racial, or seniority biases [35]. Final product analysis More efficient objective assessments have been proposed that seem to correlate well with the OSATS. Datta proposed the surgical efficiency score (SES) and snapshot assessment (SS) techniques [36]. The SES combines evaluation of final skill product quality and hand-motion analysis (see below); the SS uses OSATS scoring of a 2-minute edited video of the task performance. In surgical model task assessment, both the SES and the SS showed correlation with traditional OSATS evaluations. Szalay also assessed final product quality after the performance of six bench model tasks [37]. These results also demonstrated construct validity and correlation with OSATS results. Leak rates and cross-sectional lumen area outcomes after bench-model vascular anastamoses were significantly correlated with hand-motion analysis [38]. Using a different approach to final product analysis, Bann and colleagues [39] studied the ability of trainees to detect simple surgical errors in models containing purposely made mistakes. This was a valid predictor of qualitative performance on the same bench tasks. These results suggest that these less labor- and time-intensive assessments may be as useful as the OSATS, making surgical skill assessments more feasible. More research into the quality of these metrics needs to be performed. Hand-motion analysis Efficiency and accuracy of hand movements are a trademark of an experienced surgeon’s dexterity. Hand-motion analysis during a standardized surgical task is possible using the commercially available Imperial College Surgical Assessment Device. Through the use of passive trackers on the dorsum of each hand while performing a task through a magnetic field, currents are induced in the trackers that allow hand position to be determined using Cartesian coordinates. Number of movements, path length, speed of motion, and time on task can be measured and compared as a valid assessment of skill during a standardized procedure. Streaming video allows segmental focus into specific key steps of the observed procedure. These objective measurements have been shown to be an effective index of technical skill in both endoscopic and open procedures [40–43]. They have also been shown to have a good concordance with OSATS results [44].

1250

TSUE

et al

Hand-motion analysis is generally employed for simpler standardized tasks that are components of more complex tasks. The method is limited in assessing more complex tasks in their entirety. An enormous amount of raw data is generated that can be summarized numerically and visually. These data need to be analyzed and compared, and although motion analysis can be used to compare learners and monitor individual progress, handmotion analysis is more of a summative than formative assessment tool. Specifically telling the learner the results provides little information on how to improve his or her individual scores. The challenge for the program director is to provide what practice or teaching is specifically needed to improve on a particular skill, and interpreting needed areas of focus from the provided scores also is a challenge [45]. Additionally, there is a significant cost and technical expertise need associated with this methodology that has limited its availability at present, though continued research into the overall general validity and reliability of this method as an assessment of surgical competency is increasing. Simulation Simulation methods attempt to imitate or resembledbut not duplicated real-life clinical situations. Like real cases, simulation can provide a number of options to the learner but in a safe, standardized, and reproducible testing environment that removes the worry of compromising patient safety or outcome. Without the inhibiting fear of an irreversible change from an error, feedback can be immediate, focused, and efficient. A controlled environment can allow a ‘‘cleaner’’ and more subtle assessment of performance that may not be possible in real-life situations. Simulation can simultaneously provide improved learning and assessment, and it affords the learner the opportunity of repeated practice of a noncompetent area, measuring that progress with an objective metric. Simulator metrics can provide motivation for the trainee, and eventually set standards for certification, allowing objective comparison of trainees both to each other and to a normative value. Simulation must always be considered an adjunct to competency judgments determined by expert assessment of observed performance in the OR and by measured outcome variables from real procedures. Many studies need to be done to fully validate each simulator, especially in the realm of predictive validity. Simulation involves a wide range of growing techniques as technology progresses. Most current simulators are able to distinguish between novice and competent trainees, but are not yet sophisticated enough to distinguish between the competent and the expert. Thus, simulators may be more applicable to assessing the early phases of technical learning and skills [46]. Lowfidelity simulators tend to be mechanical representations of a procedure’s smallest fundamental components. These are generally organized into timed stations and require faculty evaluators to observe the learner at each station. This method forms the core of the above-described OSATS method. Such

ASSESSMENT OF SURGICAL COMPETENCY

1251

inanimate devices (eg, sewing a Penrose drain laceration) are relatively inexpensive and made from readily available products, but still require a significant time commitment by evaluating faculty. Body part models, which can further improve the semblance to real life, are expensive. As stated above, the OSATS method using bench-top models has been shown to correlate with OR performance, but direct translation to a broader range of surgical procedures still needs to be proved [30]. Live animal models or human cadavers can further improve the simulation. Live animal models can simulate the ‘‘feel’’ of real surgery, as they are living tissue, but generally do not reflect the exact anatomic correlate as human cadaver models can. Cadaver models do lose the feel of real tissue handling, and the temporal bone laboratory is an example of this. The OSATS using bench-top models shows good correlation with both animal and cadaver models, but at a significantly higher overall cost [24,47]. Higher-fidelity simulators include mannequins that incorporate electronics to simulate normal and pathologic conditions, and have the ability to respond realistically to interventions by the trainee. Human models with high-performance simulator technology that go well beyond ‘‘resuscitation Annie’’ are now available. These are frequently used by anesthesiologists for critical-incident and team training, but can have obvious direct applications to airway situations in the OTOHNS as well [25]. Computer-based simulators are becoming increasingly available. Such ‘‘virtual reality’’ simulators also have varying degrees of fidelity. They range from using abstract graphics that measure partial task skills to full-OR simulators. Users are able to interact in real time with a three-dimensional computer database through the use of their own senses and skills. The main challenges of creating more advanced simulators include simulating realistic surgical interfaces (coupling of instrument to tissue); geometric modeling of objects and their interactions; and an accurate operative field with advanced signal processing to simulate such phenomena as texture, light, smoke, and body fluids [48]. The first virtual reality system used in surgical skills assessment was the Minimally Invasive Surgical Trainer-Virtual Reality, which was a lower-fidelity system that focused on simulating basic laparoscopic skills rather than the appearance of the surgical field [49]. It was developed as a collaboration between surgeons and psychologists who performed a skills analysis of the laparoscopic cholecystectomy. The Advanced Dundee Endoscopic Psychomotor Tester is another example that is essentially a computerized system connected to standardized endoscopic equipment [50]. Computers are now better able to replicate not only realistic organ surface image and topography, but also the instrument ‘‘feel’’ a surgeon would expect from a real patient (realistic haptic fidelity). Rapid advances in technology, and successful use in certification in many other high-stakes fields (see above), have made the availability of simulators in measuring surgical competency a reality. The major thrust of development has been in minimally invasive procedures, especially laparoscopic, because of the more

1252

TSUE

et al

straightforward surgical interfaces compared with open surgery (ie, endoscopic instruments are levers on fulcrums with fewer degrees of freedom). Fortunately, simulators that assess basic open procedures such as vascular anastamosis suturing are now being increasingly studied. Simulators are being developed and tested in many areas, including the following:           Ophthalmic surgery Colonoscopy Arthroscopic surgery Limb trauma (musculographics) Pericardiocentesis Diagnostic peritoneal lavage Interventional/endovascular procedures (CathSim Simulator) Ultrasound Cleft-lip surgery Bronchoscopy [51–53]

Additionally, full-OR simulators are being studied to increase the assessment to include other aspects of overall surgical performance, including hemorrhage control, aseptic technique, and elements of team communication [54,55]. Virtual reality simulators probably provide the most objective measurement of a technical skill in the most standardized and reproducible environment currently available. Precision, accuracy, and error metrics are easily obtained without being labor intensive for the evaluator. Studies have shown the effectiveness of simulation primarily for lower-level learners, but further larger-scale validation studies are needed [56]. The main drawback of simulators remains cost, and further study is needed to determine whether this extra investment is worthwhile. In most cases, low-fidelity simulators may be as educationally beneficial in training and assessment, and this may help ultimately to keep simulators affordable and more generally available. Although less faculty time is needed, there are increased initial and maintenance costs compared with other assessment methods. This cost is increased when the simulation is broadened to include multiple procedure types and increased complexity. Hopefully, as computing technology improves, costs will fall, making simulators more affordable and readily studied. Many low-fidelity model simulators have been designed and used to train and assess procedures in the tympanic membrane, such a tympanocentesis and myringotomy with pressure equalization tube insertion [57–60]. These simulators were well in use before any computer models were available. A virtual reality temporal bone (VR TB) simulator has been developed at the University of Hamburg (VOXEL-MAN TempoSurg Simulator). Highresolution images of the temporal bone are used to create computer-generated images that are modified in real time as the trainee drills and receives haptic feedback, such as pressure changes depending on the material being drilled. Glasses provide a three-dimensional image that can color code

ASSESSMENT OF SURGICAL COMPETENCY

1253

different aspects of temporal bone anatomy. Zirkle and colleagues [40] studied the use of the VR TB as an assessment tool for OTOHNS trainees. Cadaveric temporal bone and VR TB drilling were assessed by both expert observers and hand-motion analysis. Experts reviewed videotaped sessions and were able to distinguish novice and experienced surgeons (construct validity) on the cadaver models but only a trend toward doing so on the VR TB. Experienced trainees outperformed novices in all hand-motion analysis metrics on the VR TB and only on the time-on-task metric for the cadaveric models. This limited study of 19 trainees concluded that the VR TB is an appropriate assessment of trainees for transition from laboratory-based to operative-based learning. More research needs to be performed to confirm temporal bone simulator validity and reliability as a competency assessment tool [61]. In otolaryngology, just as in general surgery, simulation technology focuses on endoscopic approachesdmost notably endoscopic sinus surgery. For example, a low-fidelity simulator using a force-torque sensor during gauze packing in a human nasal model was able to differentiate experienced and intermediate endoscopic sinus surgeons [62]. More experience has been gained in the OTOHNS with an endoscopic sinus surgery simulator (ES3) developed by Lockheed Martin (Akron, Ohio). The ES3 comprises four principal hardware components: a simulation host platform (high-powered Silicon Graphics workstation); a haptic controller that provides coordination between the universal instrument handler and the virtual surgical instruments; a voice-recognition instructor that operates the simulator; and an electromechanical platform that holds the endoscope replica, universal surgical instrument handle, and rubber human head model. Simulated surgical tasks range from vasoconstrictor injection to total ethmoidectomy and agar nasi dissection. The ES3 has a novice mode, thought to be a good tool to assess skill competency, whereas the intermediate mode seems best suited for surgical training. The advanced mode has potential as a practice and rehearsal tool for trained learners. Fried and colleagues [63] have performed extensive construct validation studies of the ES3 to demonstrate its discriminative capabilities. It appears to be a viable assessment tool for various endoscopic skills, especially if used in the novice mode, and correlates strongly with other validated measures of perceptual, visuospatial, and psychomotor performance [64,65]. Their extensive experience observing expert performance in the ES3 has allowed benchmark criteria to be developed that will be useful in the future to establish objective levels of proficiency. Its usefulness in predicting endoscopic sinus surgery skills in the OR (predictive validity) remains to be shown. Possible new techniques Several other experimental adaptations of technology to assessment methods have been proposed or are currently being investigated. Eye

1254

TSUE

et al

tracking during procedures is possible from vestibular testing technology and may complement hand-motion analysis. Functional brain mapping is an area of early current investigation clinically, and its usefulness in surgical assessment is not far-fetched. Downloading specific patient data into simulators could strengthen the correlation between performance in the simulator and in the OR. With intraoperative CT and MRI scans entering the market, real-time updating of stereotactical guidance systems and simulators should be eventually possible. Also, operative performance metrics could be developed from the real-time results of these intraoperative scans. Intraoperative videotaping technology is also constantly improving, and cameras are available in headlights and overhead lights, making routine use of this technique more available.

Final determinants of competency Whatever the methodology used, and regardless of whether it is in the laboratory or the OR, measurements of patient outcomes by individual surgeons must still be used in the final determinant of competency. The challenge lies in simplifying the metric for such diverse outcomes as operative blood loss to malignancy recurrence rate. Additional research needs to be done linking these measurements with today’s chosen skill assessment techniques. Despite some progress induced by ‘‘pay-for-performance’’ regulations, which have different end goals than those of trainees, the surgical academic community has placed inadequate focus on this important metric. A more objective and comparable method needs to be developed to allow measurement of trainee progress, but also comparison between trainees and community norms. Unfortunately, unless physicians themselves develop these assessments, they may be imposed on the profession by regulatory agencies.

The Otolaryngology–Head and Neck Surgery Resident Review Committee pilot study The ACGME OTOHNS RRC, in the spirit of the ACGME Outcome Project, has begun to direct its efforts toward outcomes-based evaluations rather than process-based evaluations. This includes the assessment of both technical competency and overall surgical competency. Rather than waiting for the ideal assessment method, or continuing to rely on just the resident operative case logs, the RRC, in coordination with the ABOto, is piloting a project on the use of surgical checklists to aid in assessing surgical competency. The RRC and the ABOto have defined a list of approximately 40 core procedures in which all otolaryngology residents should be proficient at the conclusion of their training. At specified intervals during their training,

ASSESSMENT OF SURGICAL COMPETENCY

1255

residents will be assessed on their competency with these procedures, and this assessment maintained as part of the resident’s permanent record. This assessment approach is based on work done in the urologic and thoracic surgery communities, and includes not only technical proficiency, but also the overall understanding of the procedure, its operative flow, and requisite perioperative patient care. The resident is graded on his or her performance on a scale from novice to competency to independence. Assessments can be completed by the supervising faculty in real time, during, or at the conclusion of the appropriate procedure, or performed more globally by faculty committees on a semi-annual basis. A limited number of OTOHNS programs are currently piloting this project. Data should reveal which procedures the majority of residents are competent in during their training, as well as indicating at which training level competency is achieved. The focus of this national pilot study includes not just the technical aspects of the surgical procedure, but also the understanding of the procedure flow and interactions with other health care personnel during the perioperative period. This makes more sense than relying on numbers of procedures performed, as the number of cases required to achieve competency for any one procedure will vary with the procedure, the resident involved, and the teaching effectiveness of the faculty. It is anticipated that the completed checklists can then be provided by the program to support the competency of the individual at the time of board certification and the effectiveness of the program at the time of ACGME accreditation.

The future All assessment efforts should be focused on the goal of producing the most outstanding graduating residents in the OTOHNS possible. No single assessment will be the panacea to the struggle to prove surgical competency in the trainees; instead, a mixture of assessment tools will be required. The resident must pass each assessment in a specified longitudinal fashion, rather than having a passing average for a group of assessments. Advancement of the residents through their training should depend on these well-defined milestones of competency rather than one mostly dependent on time and experience. This may make some training periods longer for some and shorter for others. For example, technical surgical progress through the early years of residency could be assessed every 6 months on bench models of core fundamental surgical techniques. These techniques would be made up of core components of both basic and advanced OTOHNS procedures. As competency is progressively obtained and documented, the trainee is allowed to progress to a more senior status, and regular assessments with higher-fidelity bench models and, ultimately, virtual reality simulators could be integrated. Annually, each resident could participate in an annual competency fair,

1256

TSUE

et al

testing more in-depth skills using different methods with the entire resident complement (junior and senior trainees). This could all take place in parallel with objective structured observations during live or videotaped level-appropriate procedures throughout the year. Objective testing of every procedure may not be possible, but competency in defined seminal procedures that form the basis of an OTOHNS practice must be demonstrated at each level of competency-based advancement. The trainees would be required to maintain a portfolio of this stepwise structured progress in surgical technical competency, and advancement would depend on successful completion of each objective assessment. If this were standardized nationally, it could be adopted as part of the ABOto certification process. Objective documentation of the progress toward surgical competency, especially technical skill competency, can be monitored during training rather than from an ‘‘after graduation’’ certification examination, when the usefulness of feedback is less timely. This approach would make the certification of the residents’ progress to technical competency more formative rather than summative, and thus, help to further their progress toward surgical competency.

Summary Classic surgical training and assessment have been based on the apprenticeship model. The vast majority of residents are trained well, so radical changes in the methodology must be approached with caution. Technical skill remains only one component of overall surgical competency, but has been one of the most difficult to measure. Assessment methods are currently subjective and unreliable and include techniques such as operative logs, endof-rotation global assessments, and direct observation without criteria. Newer objective methods for assessing technical skill are being developed and undergoing rigorous validation andinclude direct observation with criteria, final product analysis, and hand-motion analysis. Following the example set in fields in which high-stakes assessment is paramount, such as in aviation, virtual reality simulators have been introduced to surgical competency assessment and training. Significant work remains to integrate these assessments into both training programs and practice and to demonstrate a resultant improvement in surgical outcome. Continuous assessment and subsequent real-time feedback provided by these methods are important in the structured learning of surgical skills and will prove to be increasingly important in the documentation of the trainees’ surgical competency. References
[1] The Joint Commission. Available at: www.jointcommission.org. Accessed July 9, 2007. [2] The Bristol Royal Infirmary Inquiry. The inquiry into the management of care of children receiving complex heart surgery at the Bristol Royal Infirmary. Available at: www. Bristol-inquiry.org.uk. Accessed July 9, 2007.

ASSESSMENT OF SURGICAL COMPETENCY

1257

[3] Giddings T, Gray G, Maran A, et al. Response to the general medical council determination on the Bristol case. London: The Senate of Surgery of Great Britain and Ireland; 1998. [4] Dibb CB. Medical residency: when are program administrators liable? Journal of Legal Education 2007;281:1–8. [5] Outcome Project: Enhancing residency education through outcomes assessment. Available at: www.acgme.org/Outcome. Accessed July 9, 2007. [6] Darzi A, Mackay S. Assessment of surgical competence. Qual Health Care 2001;10(Suppl II):ii64–9. [7] Yule S, Flin R, Paterson-Brown S, et al. Non-technical skills for surgeons in the operating room: a review of the literature. Surgery 2006;139(2):140–9. [8] Wright M, Turner D, Harburg C. Competence assessment for the hazardous industries. Sudbury (Great Britain): Greenstreet Berman Ltd. For Health and Safety; 2003. [9] Hamman WR. The complexity of team training: what we have learned from aviation and its application to medicine. Qual Saf Health Care 2004;13:72–9. [10] Collyer SC, Malecki GS. Tactical decision making under stress: history and overview. In: Cannon-Bowers JA, Salas E, editors. Making decisions under stress: implications for individual and team training. Washington, DC: American Psychological Association; 1999. p. 3–15. [11] Fletcher G, Flin R, McGreorge P, et al. Anaesthetists’ Non-technical Skills (ANTS): evaluation of a behavioral marker system. Br J Anaesth 2003;90:580–8. [12] Flin R, Yule S. The non-technical skills for surgeons (NOTSS) systems handbook v1.2. 2006. Available at: http://www.abdn.acuk/iprc/notss. Accessed July 9, 2007. [13] Airasian PW. Classroom assessment. 3rd edition. New York: McGraw-Hill; 1997. [14] Kern DE, Thomas PA, Howard DM, et al. Curriculum development for medical education: a six-step approach. Baltimore (MD): Johns Hopkins University Press; 1998. [15] Cosman PH, Cregan PC, Martin CJ, et al. Virtual reality simulators: current status in acquisition and assessment of surgical skills. ANZ J Surg 2002;72:30–4. [16] Scott DJ, Valentine RJ, Bergen PC, et al. Evaluating surgical competency with the American Board of Surgery In-Training Examination, skill testing, and intraoperative assessment. Surgery 2000;128(4):613–22. [17] Adrales GL, Donnelly MB, Chu UB, et al. Determinants of competency judgments by experienced laparoscopic surgeons. Surg Endosc 2004;18(2):323–7. [18] Reznick RK. Teaching and testing technical skills. Am J Surg 1993;165:358–61. [19] Carr MM. Program directors’ opinions about surgical competency in otolaryngology residents. Laryngoscope 2005;115:1208–11. [20] Grantcharov TP, Bardram L, Funch-Jensen P, et al. Assessment of technical surgical skills. Eur J Surg 2002;168:139–44. [21] Moorthy K, Munz Y, Adams S, et al. Self-assessment of performance among surgical trainees during simulated procedures in a simulated operating theater. Am J Surg 2006; 192(1):114–8. [22] Moorthy K, Munz Y, Sarker SK, et al. Objective assessment of technical skills in surgery. BMJ 2003;327:1032–7. [23] Available at: http://www.acgme.org/acWebsite/RRC_280/280_resEval.asp. Accessed July 9, 2007. [24] Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997;84(2):273–8. [25] Siker ES. Assessment of clinical competence. Curr Opin Anaesthesiol 1999;12(6):677–84. [26] Pandey VA, Wolfe JH, Liapis CD, et al. The examination assessment of technical competence in vascular surgery. Br J Surg 2006;93(9):1132–8. [27] Dailey SH, Kobler JB, Zeitels SM. A laryngeal dissection station: educational paradigms in phonosurgery. Laryngoscope 2004;114(5):878–82. [28] Gosman GG, Simhan HN, Guido RS, et al. Focused assessment of surgical performance: difficulty with faculty compliance. Am J Obstet Gynecol 2005;193(5):1811–6.

1258

TSUE

et al

[29] Darzi A, Datta V, Mackay S. The challenge of objective assessment of surgical skill. Am J Surg 2001;181:484–6. [30] Datta V, Bann S, Beard J, et al. Comparison of bench test evaluations of surgical skill with live operating performance assessments. J Am Coll Surg 2004;199(4):603–6. [31] Beard JD, Jolly BC, Newble DI, et al. Assessing the technical skills of surgical trainees. Br J Surg 2005;92:778–82. [32] Roberson DW, Kentala E, Forbes P. Development and validation of an objective instrument to measure surgical performance at tonsillectomy. Laryngoscope 2005;115(12):2127–37. [33] Naik VN, Perlas A, Chandra DB, et al. An assessment tool for brachial plexus regional anesthesia performance: establishing construct validity and reliability. Reg Anesth Pain Med 2007;32(1):41–5. [34] Hance J, Aggarwal R, Stanbridge R, et al. Objective assessment of technical skills in cardiac surgery. Eur J Cardiothorac Surg 2005;28(1):157–62. [35] Saleh GM, Gauba V, Mitra A, et al. Objective structured assessment of cataract surgical skill. Arch Ophthalmol 2007;125:363–6. [36] Datta V, Bann S, Mandalia M, et al. The surgical efficiency score: a feasible, reliable, and valid method of skills assessment. Am J Surg 2006;192(3):372–8. [37] Szalay D, MacRae H, Regehr G, et al. Using operative outcome to assess technical skill. Am J Surg 2000;180:234–7. [38] Datta V, Mandalia M, Mackay S, et al. Relationship between skill and outcome in the laboratory-based model. Surgery 2002;131:318–23. [39] Bann S, Khan M, Datta V, et al. Surgical skill is predicted by the ability to detect errors. Am J Surg 2005;189(4):412–5. [40] Zirkle M, Roberson DW, Leuwer R, et al. Using a virtual reality temporal bone stimulator to assess otolaryngology trainees. Laryngoscope 2007;117(2):258–63. [41] Taffinder N, Smith SG, Huber J, et al. The effect of a second-generation 3D endoscope on the laparoscopic precision of novices and experienced surgeons. Surg Endosc 1999; 13:1087–92. [42] Datta V, Mandalia M, Mackay S, et al. Relationship between skill and outcome in the laboratory based model. Surg 2002;131(3):318–23. [43] Datta V, Mackay S, Mandalia M, et al. The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. J Am Coll Surg 2001;193:479–85. [44] Datta V, Chang A, Mackay S, et al. The relationship between motion analysis and surgical technical assessments. Am J Surg 2002;184(1):70–3. [45] Porte MC, Xeroulis G, Reznick RK, et al. Verbal feedback from an expert is more effective than self-accessed feedback about motion efficiency in learning new surgical skills. Am J Surg 2007;193(1):105–10. [46] Reznick RK. Surgical simulation: a vital part of our future. Ann Surg 2005;242(5):640–1. [47] Anastakis D, Regehr G, Reznick RK, et al. Assessment of technical skills transfer from the bench training model to the human model. Am J Surg 1999;177(2):167–70. [48] Seymour NE, Rotnes JS. Chellenges to the development of complex virtual reality simulations. Surg Endosc 2006;20:1774–7. [49] Wilson MS, Middlebrook A, Sutton C. MIST-VR: a virtual reality trainer for laparoscopic surgery assesses performance. Ann R Coll Surg Engl 1997;79:403–4. [50] Hanna GB, Drew T, Clinch P, et al. Computer-controlled endoscopic performance assessment system. Surg Endosc 1998;12:1997–2000. [51] Moorthy K, Smith S, Brown T, et al. Evaluation of virtual reality bronchoscopy as a learning and assessment tool. Respiration 2003;70(2):195–9. [52] Henderson BA, Ali R. Teaching and assessing competence in cataract surgery. Curr Opin Ophthalmol 2007;18(1):27–31. [53] Schendel S, Montgomery K, Sorokin A, et al. A surgical simulator for planning and performing repair of cleft lips. J Craniomaxillofac Surg 2005;33(4):223–8.

ASSESSMENT OF SURGICAL COMPETENCY

1259

[54] Moorthy K, Munz Y, Adams S, et al. A human factor analysis of technical and team skills among surgical trainees during procedural simulations in a simulated operating theatre. Ann Surg 2005;242(5):631–9. [55] Moorthy K, Munz Y, Forrest D, et al. Surgical crisis management training and assessment: a simulation-based approach to enhancing operating room performance. Ann Surg 2006; 244(1):139–47. [56] Reznick RK, MacRae H. Medical education: teaching surgical skills-changes in the wind. N Engl J Med 2006;355(25):2664–70. [57] Baer S, Williams H, McCombe A. A model for instruction in myringotomy and grommet insertion. Clin Otolaryngol 1990;15:383–4. [58] Holt GR, Parel SM, Shuler SL. A model training ear for teaching paracentesis, myringotomy, and insertion of tympanostomy tubes. Otolaryngol Head Neck Surg 1983;91:333–5. [59] Hantman I. An ear manikin. Teaching and training device. Arch Otolaryngol 1968;88: 407–12. [60] Neal SL, Harris JP, Davidson TM. Artificial eardrum for instruction in myringotomy and PET tube insertion. Laryngoscope 1985;95:1008–9. [61] Sewell C, Morris D, Blevins NH, et al. Validating metrics of a mastoidectomy simulator. Stud Health Technol Inform 2007;125:421–6. [62] Kumagai T, Yamashita J, Morikawa O, et al. A new force-based objective assessment of technical skills in endoscopic sinus surgery. Stud Health Technol Inform 2007;125:235–7. [63] Fried MP, Sadoughi B, Weghorst SJ, et al. Construct validity of the endoscopic sinus surgery simulator: II. Assessment of discriminant validity and expert benchmarking. Arch Otolaryngol Head Neck Surg 2007;13:350–7. [64] Arora H, Uribe J, Ralph W, et al. Assessment of construct validity of the endoscopic sinus surgery stimulator. Arch Otolaryngol Head Neck Surg 2005;131(3):217–21. [65] Available at: http://www.acgme.org/acWebsite/resEvalSystem/reval_otolaryngology.asp. Accessed July 9, 2007.