Journal of Emergencies, Trauma, and Shock
Home About us Editors Ahead of Print Current Issue Archives Search Instructions Subscribe Advertise Login 
Users online:1190   Print this pageEmail this pageSmall font sizeDefault font sizeIncrease font size   

SYMPOSIUM Table of Contents   
Year : 2010  |  Volume : 3  |  Issue : 4  |  Page : 353-359
Tools for evaluating team performance in simulation-based training

1 Department of Psychology & Institute for Simulation and Training, University of Central Florida, USA
2 Maricopa Integrated Health System, Rochester, MN, USA
3 Orlando Health, Rochester, MN, USA
4 Mayo Clinic, Rochester, MN, USA
5 Healthcare Consulting, LLC, USA
6 Office of the Assistant Secretary of Defense (Health Affairs) TRICARE Management Activity, USA

Click here for correspondence address and email

Date of Submission07-Feb-2010
Date of Acceptance08-Jul-2010
Date of Web Publication28-Sep-2010


Teamwork training constitutes one of the core approaches for moving healthcare systems toward increased levels of quality and safety, and simulation provides a powerful method of delivering this training, especially for face-paced and dynamic specialty areas such as Emergency Medicine. Team performance measurement and evaluation plays an integral role in ensuring that simulation-based training for teams (SBTT) is systematic and effective. However, this component of SBTT systems is overlooked frequently. This article addresses this gap by providing a review and practical introduction to the process of developing and implementing evaluation systems in SBTT. First, an overview of team performance evaluation is provided. Second, best practices for measuring team performance in simulation are reviewed. Third, some of the prominent measurement tools in the literature are summarized and discussed relative to the best practices. Subsequently, implications of the review are discussed for the practice of training teamwork in Emergency Medicine.

Keywords: Simulation-based team training, simulation, team training, team performance measurement, team evaluation, team performance

How to cite this article:
Rosen MA, Weaver SJ, Lazzara EH, Salas E, Wu T, Silvestri S, Schiebel N, Almeida S, King HB. Tools for evaluating team performance in simulation-based training. J Emerg Trauma Shock 2010;3:353-9

How to cite this URL:
Rosen MA, Weaver SJ, Lazzara EH, Salas E, Wu T, Silvestri S, Schiebel N, Almeida S, King HB. Tools for evaluating team performance in simulation-based training. J Emerg Trauma Shock [serial online] 2010 [cited 2022 Aug 16];3:353-9. Available from:

   Introduction Top

Healthcare is in the midst of a variety of transformations. New technologies are being introduced that change the way care is provided (e.g., robotic surgery) and managed (e.g., electronic health records), as well as the way care providers are educated and trained (i.e., simulation). While simulation-based training (SBT) has been a mainstay educational method in other domains for decades, only recently has its potential been widely acknowledged in healthcare. This adoption of new technologies into care delivery processes and into the development and education of personnel is accompanied by new social and organizational structures as well, most notably teamwork and team training. Good measurement practices go hand in hand with effective design of practice-based learning activities; however, team performance measurement is an often overlooked aspect of designing SBT for teams (SBTT).

This article focuses on the intersection of these two new areas of interest in healthcare: SBT and teamwork training. More specifically, the purpose of this article is to provide a review of the state of the science and practice of team performance measurement in SBT. To this end, four main goals are addressed. First, we provide a review of the fundamental concepts of team performance measurement and evaluation within the context of SBT. Second, we provide a set of best practices rooted in the science of teams, training, and performance measurement to guide the development or selection and modification of measurement tools for a specific context. Third, we review illustrative examples of tools currently available in the healthcare literature. Fourth, we identify gaps in the current state of the literature and discuss future needs.

   Overview of Team Performance Evaluation Top

This section reviews key concepts and terminology that provide the groundwork for understanding team performance measurement and evaluation in SBT. Specifically, issues of purpose (i.e., why evaluate team performance?), content (i.e., what to evaluate?), and method (i.e., how to evaluate?) are provided below. First, some basic definitions are provided.

While performance refers to the actual behaviors enacted by a team (or a system of teams), effectiveness refers to the evaluation of the results of performance; that is, the degree to which these behaviors satisfy team, organizational, and/or other super-ordinate goals. [1] Through this lens, the term team performance embodies teamwork-the behavioral, cognitive, and affective processes teams engage in order to coordinate their interactions toward shared goals. [2],[3] Team performance evaluation can, thus, be defined as the application of standardized diagnostic measurement tools to assess the behaviors, cognitions, and attitudes enacted by team members in relation to clearly operationalized criteria. As such, evaluation is designed to provide information not only on what outcomes the team achieved, but also how they reached these outcomes. [4] A scientifically rooted and practically meaningful approach to SBTT is built upon a foundation of diagnostic evaluation.

Why evaluate team performance?

Team performance evaluation serves several purposes within SBT for teams. Uses of team performance in evaluation for training, assessment, and program evaluation are discussed below.

First, it provides a mechanism to guide learning through systematic, developmental feedback. [4],[5] To meaningfully reflect on and integrate simulation-based learning experiences, learners must be provided with explicit, constructive feedback detailing their current performance levels and strategies for improvement. [6] Team members need to understand their performance level relative to expectations or standards before they can address how to improve.

Second, team performance evaluation enables summative assessment; that is, it allows trainers and team members themselves to obtain a snapshot of a team's development at a particular time. Such assessments may help determine whether trainees possess the requisite team competencies for effective on the job performance upon the completion of training. Summative assessment may also be used to match current trainee skill mastery level to training objectives in order to create a more targeted and efficient training program. For example, team members may complete an initial simulation scenario to determine those team competencies most in need of refinement. In this light, team performance evaluation provides a mechanism for "individualizing" training to particular teams or constellations of potential team members, resulting in a more focused use of training time.

Third, by defining key behavioral and outcome criteria, evaluation provides a mechanism for SBT curriculum validation. Validating a curriculum answers the question, "does this training work?". [7] Evaluation is the mechanism for determining the degree to which trainees apply knowledge, skills, and attitudes learned in training to similar situations (i.e., routine transfer) and/or to novel problems and new situations (i.e., adaptive transfer). In this way, team performance evaluation in simulation scenarios can serve the purpose of understanding the general level of team functioning within an organization in a controlled and systematic manner.

What to measure?

While is it clear the evaluation is a vital component of SBTT, the question then arises what to measure? By definition, teams are composed of individuals, often having heterogeneous knowledge, skills, and attitudes, working interdependently to achieve a shared goal. [8] To achieve this goal, they must coordinate, communicate, and cooperate, each individual dynamically adjusting their own effort and input to the group based upon the effort and input of fellow team members. [9] Team performance outcomes are, thus, the synergistic product of these multiple inputs. Therefore, evaluation must not only capture the final outcome of team performance, but also how the team achieved these outcomes. To comprehensively evaluate team performance, training designers must employ multiple measurements, which capture the behavioral, cognitive, and attitudinal components of performance at the team level. This also means capturing diagnostic information on individual team member roles in order to provide targeted corrective feedback. While evaluation is not focused specifically on individual team members, providing role-specific feedback helps individual members on how to operate effectively within the specific team roles as well as helps to develop important teamwork competencies such as communication. In sum, the primary content of a team performance evaluation tool in SBT should be the teamwork competencies targeted for training: the knowledge, skills, and attitudes underlying effective teamwork. [10]

How to evaluate team performance?

As noted earlier, fully capturing the range of team processes which comprise team performance involves multiple approaches to measurement. One pitfall of many attempts to evaluate team performance involves measuring what is easy to measure as opposed to what is meaningful to measure. For example, many early SBT studies only measured trainee reactions to the training program: did trainees like it, find it useful, and so forth. However, reactions to training have demonstrated relatively weak relationships to other, arguably, more important outcomes such as learning and transfer of learned KSAs to the daily work environment. [11] For example, training which includes difficulty and uncertainty may initially be rated as less enjoyable by trainees, especially those with a performance orientation, who strive to demonstrate their present level of skills. [12],[13] However, more challenging training scenarios enable trainees to practice more advanced teamwork competencies under more realistic clinical conditions. Additionally, important, but broad, indicators of patient safety and quality (e.g., adjusted mortality rates) may not be sensitive to the components of team performance that impact patent outcomes. [14] Therefore, the methods and metrics used for team performance evaluation must be designed to capture both team process and outcomes.

Tools for team performance measurement and evaluation can be organized into three overarching categories: observational rating scales, team self-assessment, and the event-based approach to training and measurement. Observational rating scales are standardized measurement protocols, which train observers to record and rate observable team behavior. [15],[16] Teamwork rating scales may employ traditional behaviorally anchored rating scales (BARS) or other graphical observational scales. Additionally, they may provide ratings of each dimension of teamwork, a global rating of teamwork, or both. Rating scales developed for rating team performance in healthcare settings have included global Likert-scale ratings (e.g., Mayo High Performance Teamwork Scale [17] and BARS scales), [18] as well as approaches which capture both the quantity and quality of teamwork behaviors (e.g., CATS [19] ). Observational methods, of course, have weaknesses due to limitations and biases of individual raters. For example, while early research suggested that raters can rate validly up to seven behavioral dimensions at a time, [20] more recent suggestions are for limiting the number of behavioral dimensions to be rated to three to four. [21] Additionally, raters must be able to observe and rate behavior within the rich situational contexts of clinical simulations.

Event-based training and assessment techniques (EBAT) were specifically designed to deal with the complexities of rating performance during complex simulation scenarios. [22],[23],[24] In the EBAT approach, critical events which provide opportunities to perform key teamwork competencies are combined and embedded into relevant contextualized scenarios. Critical events can be routine or novel events which occur at a pre-determined point during the course of the simulation. Developing scenarios based upon critical events create both a script and timeline for the scenario, which helps raters to know explicitly what behaviors should occur, approximately when they should occur, and also provides a clear way to organize rating forms. This approach was originally developed to rate team performance during complex military simulation scenarios; [22] however, it has successfully been leveraged for ratings of teamwork during resident medical training. [24] A more detailed review of observational scales used in healthcare is presented later in this article.

While observational measures capture the behaviors teams actually engage in, observers cannot assess the implicit components of teamwork, including team cognition and implicit communication. To capture these elements of team performance, observational measures can be combined with self-assessment measures completed by team members. While self-report measures also have limitations, they offer a means for assessing unobservable components of team performance, which are no less important than observable components. For example, Sexton and colleagues [25] developed a component of their Operating Room Management Questionnaire to specifically capture team-member assessment of teamwork, during their last surgical case.

Different approaches to evaluating teamwork in simulations have associated strengths and weaknesses. Consequently, a combined approach leveraging best practices derived from the science of teamwork and human performance measurement can help to obtain a more complete picture of the complex phenomena, that is, team performance. The next section presents eight best practices for team performance evaluation based on a review of techniques currently being used in healthcare and other complex team environments.

   Best Practices Top

There is no "one size fits all" approach to measure team performance. As reviewed above, differences in purpose and content of measurement (e.g., the team competencies being trained or assessed) as well as the clinical context require different approaches. However, the fundamentals of team performance measurement and evaluation provide guidance development or selection and modification of team performance measurement tools for a given set of conditions. This section integrates recently advanced best practices in the measurement of team performance in SBT for healthcare [24] and provides practical tips for implementation. [Table 1] provides a summary of these best practices, rationales for each, as well as practical tips to guide the development of measurement tools suited for a specific context. Each of these best practices is described below.
Table 1: Best practices in team performance measurement for SBT for healthcare, description and tips (adapted from Rosen et al.[30])

Click here to view

Best Practice #1 - The content of a measurement tool should be driven by theoretically and empirically rooted competencies of teamwork

An initial and critical step in designing measurement tools for SBT for teams is to clearly define the content, [24] that is, the specific teamwork knowledge, skills, and attitudes that need to be measured and assessed. This content should be based in the best available evidence on the important characteristics of teamwork. [2],[10] Ideally, the competency model is rooted in the specific context of the target clinical area; however, most clinical specialties have not developed consensus teamwork competency models to date. Therefore, adapting more general teamwork skills (e.g., basic communication, leadership, back-up behavior, etc.) is the primary option for training and evaluating teamwork skills in healthcare.

Best Practice #2 - Measures should be linked to specific learning objectives

In training, measurement drives the provision of feedback. Therefore, the measures should have explicit and tight connections to the learning objectives-the educational goals for a given learning activity. [26] The content of the measurement tool (i.e., the "what" that is being measured) should be consistent with what is being trained.

Best Practice #3 - Capture multiple levels of performance

It is important for measurement tools in SBTT to discriminate between the individual and team levels of performance. This can be a challenging goal to meet as team and technical performance are frequently intertwined. However, performance measures should distinguish as much as possible between individual-level performance and team-level performance, no matter what the purpose of evaluation. Separating these aspects of performance is critical for both formative assessments (e.g., is there a deficiency in an individual's technical ability, or is there a problem with how the team works together?) as well as for generating feedback for learning in simulated environments. [27]

Best Practice #4 - Use scenario events to anchor measurement opportunities

Team training scenarios can be fast-paced and complex. With many learners engaged in individual and team performance, a team-based practice activity can quickly overwhelm an observer's limited attention and ability to accurately score performance. However, the control over scenario events in SBT affords a great advantage for measuring team performance (i.e., predictability of events requiring a teamwork response), which reduces the burden on observers. Specifically, observers can focus their attention on pre-determined critical events-circumstances in a scenario that require targeted teamwork skills to manage effectively. [22]

Best Practice #5 - Focus measurement on observable team behaviors and the processes of performance, not just the outcomes

Emphasizing observable teamwork behaviors has two main advantages. First, creating and maintaining inter-rater reliability is simplified when measuring objective and directly observable behaviors, when compared to more abstract team concepts (e.g., team climate) that require observers to make judgments beyond what they are actually seeing. Second, measuring observable behaviors can provide team members with actionable guidance for improvement, that is, behaviors they can use in the future to improve performance.

Best Practice #6 - Focus on "diagnosing" performance

Performance diagnosis entails uncovering the reasons by a team performed as they did-on identifying the underlying knowledge, skill, or attitude competencies that contributed to effective and ineffective team performance. This is a valuable approach to measurement in not only understanding the reasons why a team performed the way it did but also for providing the appropriate corrective feedback or remediation. [4]

Best Practice #7 - Train and monitor observers

Measurement tools are only as good as the quality of data they produce. For observation-based measurements, the quality of the data relies not just on the tool, but the observer as well. For this reason, training observers is critical. Training can be facilitated with the use of scoring guides where the different types and levels of teamwork behaviors are clearly articulate. This type of guide and associated training helps to keep all observers and trainers on the same page in terms of expectations for team performance.

Best Practice #8 - Use measurement to support post-session debriefs and remediation

Feedback drives learning, and in SBTT, facilitated debriefs are the primary method of delivering feedback. [28] Here, teams are expected to actively discuss what happened during the scenario and how this performance relates to standards. Debrief facilitators can use performance measurement tools to structure their observations and guide the team's discussion to aspects of team performance that need reinforcement or correction. [29]

   Tools for Observational Methods in Simulation-Based Team Training Top

Observation-based methods have been and remain the primary means of capturing team performance. This section provides an overview of four of the predominate tools available in the literature to date. These form different domains and are not necessarily designed for the simulation environment. However, an examination of these tools in light of the best practices for SBTT described above is useful for understanding what makes a good performance measurement tool in a team training environment. [Table 2] provides a summary of each tool, and the following section describes the types of choices in content and method, which need to be made when developing or adapting a tool for the purposes of training.
Table 2: Overview of four observation-based team performance evaluation tools

Click here to view

First, the University of Texas Behavioral Marker Audit Form (UT BMAF) [31] was derived from team training methodologies utilized in aviation [i.e., the Line Operations Safety Audit (LOSA), an observational tool that assesses the behaviors of flight crews]. The UT BMAF is a one-page behavioral rating scale with three sections (i.e., event demographics, threats to patient care, and behavioral) designed to evaluate teamwork behaviors performed during neonatal resuscitations. In addition to incorporating LOSA, focus groups, surveys, and video recordings resulted in 10 behavioral markers: information sharing, inquiry, assertion, intentions shared, teaching, evaluation of plans, workload management, vigilance/environmental awareness, teamwork overall, and leadership. Each behavior is rated on two scales: observability (i.e., how well a behavior could be observed) and frequency (i.e., how consistently behaviors were exhibited).

Second, the Anesthetists' Non-Technical Skills (ANTS) System was developed in collaboration between industrial psychologists and consultant anesthetists in a 4-year research project attempting to develop a taxonomy for structured observations of anesthetists. The ANTS System is a behavioral marking system which describes observable non-technical skills associated with good anesthetic practice. [32] The ANTS System is designed as a hierarchy, with 4 higher level skill categories (i.e., task management, team working, situation awareness, and decision making) and 15 lower level skill elements within these four dimensions. Each element is further defined with markers of good and poor behaviors to facilitate the observer in determining the presence/absence of the higher skill elements. Ratings of both the higher and lower level skill categories are made on a four-point scale to describe the quality of the observed performance (1-Poor, 2-Marginal, 3-Acceptable, 4-Good). The rating form also contains areas to note observations on performance and debriefing notes (which is encouraged after every observation session). Furthermore, the ANTS system has been empirically tested, with acceptable levels of validity, reliability, and usability, using a method previously used for NOTECHS, [33] a system from aviation.

Third, the Observational Teamwork Assessment of Surgery (OTAS) [34] was developed to evaluate technical and interpersonal skills in surgery teams. OTAS measures two main facets (taskwork and teamwork) of the surgical process in three stages: pre-operative, intra-operative, and post-operative. The first component, the clinical checklist, is completed by a surgical expert and is subdivided into categories associated with patient, equipment/provisions, or communications. Items are rated dichotomously (i.e., as "yes" or "no") indicating whether or not the behavior was appropriately completed. The second component assesses teamwork behaviors (communication, leadership, coordination, monitoring, and cooperation) and is preferably rated by an expert researcher who scores behaviors on a 7-point Likert scale.

Fourth, derived from LOSA, ANTS, and OTAS, the Communication and Teamwork Skills [19] Assessment is designed to measure communication and teamwork behaviors of medical professionals in multiple clinical settings. Ratings are made on four clusters of behavioral markers (i.e., coordination, cooperation, situational awareness, and communication), and teams are scored both on frequency and quality. Rating the frequency of behaviors provides evidence as to how often behaviors are occurring; however, there is no indication of whether or not behaviors are being performed correctly. Thus, behaviors are also rated by quality, which can assist clinicians in determining the specific areas of improvements. Furthermore, scores can be aggregated to an overall score, or they can be parsed out to specific, individual behaviors. The benefit of providing several individual and team scoring techniques allows clinicians to evaluate behaviors on multiple levels, which offers a thorough assessment of both the team as a whole and the individual team members.

   Implications Top

The tools reviewed above represent some of the best instruments available in the published literature. However, these tools were not all developed specifically for use in a training environment. Therefore, some of the characteristics of these tools do not map onto the best practices described above. This section highlights some of these considerations for developing tools for SBTT.

Issues of scoring and temporal organization are important to meet the demands of guiding learning in simulation (see best practices 4-6). Many of the tools reviewed above require observers to summate their ratings across time by providing for example, an overall rating of the quality of a specific teamwork behavior during an entire observation session. This is satisfactory if the goal is purely assessment; however, in order to provide meaningful feedback, more specific aspects of the team's appropriate and inappropriate use (or lack of use) of a behavior are more useful. At best, a global rating can provide team members with an indication of whether or not there is a problem with their performance, but it provides very little information about how to improve their performance. Additionally, the content of the measurement tool is critical if it is to support learning (see best practices 1, 2, 6, and 8). Ratings on very high level or abstract "dimensions of teamwork" (e.g., overall teamwork) do not provide the specificity necessary to guide learners in their development. However, capturing a particular example of a specific teamwork behavior (e.g., was there a closed-loop communication when a medication order was called out?) can feed directly into a learning point on post-performance debrief.

As described earlier, EBAT [22],[24] is a method for developing measurement tools that meet all of the best practices outlined above. The challenge with these tools is that each is scenario specific. However, the added investment in developing measurement tools for each training scenario pays dividends in the quality of measurement as well as the quality of the training. While general methods have been outlined for doing the EBAT process, future work is needed to develop tools to structure and facilitate this process for practitioners. [35]

   Conclusion Top

It is an exciting time to be involved in the development of clinical practitioners. New technology and new thinking on what is important for effectiveness have created an opportunity to improve the quality and safety of patient care with inventive strategies. This article has provided a brief review of the concepts and methods of team performance measurement in SBT, one component of a broader initiative to transform healthcare into a high-reliability organization. This is a new area for healthcare, and there is much work to do before good practices are widely adopted. However, the types of methods and tools discussed in this review can contribute to getting the most out of SBTT.

   References Top

1.Campbell J, McCloy R, Oppler S, Sager C. A theory of performance. In: Schmitt N, Borman W, Editors. Personnel Selection in organizations. San Francisco: Jossey-Bass; 1993. p. 35-71.  Back to cited text no. 1      
2.Mathieu J, Maynard MT, Rapp T, Gilson L. Team effectiveness 1997-2007: A review of recent advancements and a glimpse into the future. J Manage 2008;34:410-76.  Back to cited text no. 2      
3.Salas E, Sims D, Burke CS. Is there a big five in teamwork? Small Group Res 2005;36:555-99.  Back to cited text no. 3      
4.Salas E, Rosen MA, Burke CS, Nicholson D, Howse WR. Markers for Enhancing Team Cognition in Complex Environments: The Power of Team Performance Diagnosis. Aviat Space Environ Med 2007;78:B77-85.  Back to cited text no. 4  [PUBMED]    
5.Cannon-Bowers JA, E. S. A framework for developing team performance measures in training. In: Brannick MT, Salas E, Prince C, Editors. Team performance and measurement: Theory, methods, and applications Mahwah, NJ: Erlbaum; 1997. p. 45-62.  Back to cited text no. 5      
6.Salas E, Sims D, Burke CS. Is there a big five in teamwork? Small Group Res 2005;36:555-99.  Back to cited text no. 6      
7.Magee JH. Validation of medical modeling and simulation training devices and systems. Stud Health Technol Inform 2003;94:196-8.  Back to cited text no. 7  [PUBMED]    
8.Baker DP, Salas E, Barach P, Battles J, King H. The relation between teamwork and patient safety. In: Carayon P, Editor. Handbook of human factors and ergonomics in health care and patient safety. Mahwah, NJ: Lawrence Erlbaum; 2007. p. 259-71.  Back to cited text no. 8      
9.Salas E, Wilson KA, Murphy CE, King HB, Salisbury M. Communicating, coordinating, and cooperating when lives depend on it: Tips for teamwork. Jt Comm J Qual Patient Saf 2008;34:333-41.  Back to cited text no. 9      
10.Salas E, Rosen MA, Burke CS, Goodwin GF. The wisdom of collectives in organizations: An Update of the Teamwork Competencies. In: Salas E, Goodwin GF, Burke CS, Editors. Team effectiveness in complex organizations: Cross-disciplinary perspectives and approaches. New York: Routledge; 2009. p. 39-79.  Back to cited text no. 10      
11.Alliger GM, Tannenbaum SI, Bennett W, Traver H, Shotland A. A meta-analysis of the relations among training criteria. Personnel Psychol 1997;50:341-58.  Back to cited text no. 11      
12.Chen G, Matheiu JE. Goal orientation dispositions and performance trajectories: The role of supplementary and complementary situational inducements. Organ Behav Hum Decis Process 2008;106:21-38.  Back to cited text no. 12      
13.VandeWalle D, Cummings LL. A test of the influence of goal orientation on the feedback-seeking process. J Appl Psychol. 1997;82:390-400.  Back to cited text no. 13  [PUBMED]  [FULLTEXT]  
14.Horn SD. Performance measures and clinical outcomes. J Am Med Assoc 2006;296:2731-2.  Back to cited text no. 14      
15.Baker D, Salas E. Analyzing team performance: In the eye of the beholder? Mil Psychol 1996;8:235-45.  Back to cited text no. 15      
16.Black N. Why we need observational studies to evaluate the effectiveness of healthcare. Br Med J 1996;312:1215-8.  Back to cited text no. 16      
17.Malec JF, Torsher LC, Dunn WF, Wiegmann DA, Arnold JJ, Brown DA, et al. The Mayo High Performance Teamwork Scale: Reliability and Validity for Evaluating Key Crew Resource Management Skills. Simul Healthc 2007;2:4-10.  Back to cited text no. 17  [PUBMED]    
18.Wright BG, Phillips-Bute BG, Petrusa ER, Griffin KL, Hobbs GW, Taekman JM. Assessing teamwork in medical education and practice: Relating behavioral teamwork ratings and clinical performance. Med Teach 2009;31:30-8.  Back to cited text no. 18      
19.Frankel A, Gardner R, Maynard L, Kelly A. Using the communication and teamwork skills (CATS) assessment to measure health care team performance. Jt Comm J Qual Patient Saf 2007;33:549-58.  Back to cited text no. 19  [PUBMED]    
20.Thornton GC, III. Assessment centers in human resource management. Reading, MA: Addison-Wesley; 1992.  Back to cited text no. 20      
21.Thornton GC, III, Mueller-Hanson R. Developing organizational simulations: A guide for practitioners and students. Mahwah, NJ: Lawrence Erlbaum; 2004.  Back to cited text no. 21      
22.Fowlkes JE, Dwyer DJ, Oser RL, Salas E. Event-based approach to training (EBAT). Int J Aviat Psychol 1998;8:209-21.  Back to cited text no. 22      
23.Fowlkes JE, Lane NE, Salas E, Franz T, Oser R. Improving the measurement of team performance: the TARGETs methodology. Mil Psychol 1994;6:47-61.  Back to cited text no. 23      
24.Rosen MA, Salas E, Wu TS, Silvestri S, Lazzara EH, Lyons R, et al. Promoting Teamwork: An Event-based Approach to Simulation-based Teamwork Training for Emergency Medicine Residents. Academic Emergency Medicine. 2008;15(1-9).  Back to cited text no. 24      
25.Sexton JB, Helmreich RL, Glenn D, Wilhelm JA, Merritt AC. Operating room management attitudes questionnaire. 2000; Available from: [cited 2008 Feb 12]  Back to cited text no. 25      
26.Goldstein IL, Ford K. Training in Organizations: Needs Assessment, Development, and Evaluation. 4 th ed. New York: Wadsworth Publishing; 2001.  Back to cited text no. 26      
27.Deering S, Rosen MA, Salas E, King HB. Building team and technical competency for obstetric emergencies: The Mobile Obstetric Emergency Simulator (MOES) System. Simul Healthc 2009;4:166-73.  Back to cited text no. 27  [PUBMED]  [FULLTEXT]  
28.Dismukes RK, Smith GM, Editors. Facilitation and debriefing in aviation training operations. Aldershot, UK: Ashgate; 2000.  Back to cited text no. 28      
29.Prince C, Salas E, Brannick M, Orasanu J. Beyond facilitation: An improved CRM debrief for safety training. Hum Factors Aerospace Safety 2005;5:1-22.  Back to cited text no. 29      
30.Rosen MA, Salas E, Wilson KA, King HB, Salisbury M, Augenstein JS, et al. Measuring Team Performance for Simulation-based Training: Adopting best practices for healthcare. Simul Healthc 2008;3:33-41.  Back to cited text no. 30  [PUBMED]    
31.Thomas EJ, Sexton JB, Helmreich RL. Translating teamwork behaviors from aviation to healthcare: Development of behavioral markers for neonatal resuscitation. Qual Saf Health Care 2004;13:i57-64.  Back to cited text no. 31  [PUBMED]  [FULLTEXT]  
32.Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Rating non-technical skills: Developing a behavioral marking system for use in anesthesi. Cognition Technol Work 2004;6:165-71.  Back to cited text no. 32      
33.O′Connor P, Hormann JJ, Flin R. Developing a method for evaluation crew resource management skills: A European perspective. Int J Aviat Psychol 2002;2:62-86.  Back to cited text no. 33      
34.Healey AN, Undre S, Vincent CA. Developing observational measures of performance in surgical teams. Qual Saf Health Care 2004;13:i33-40.  Back to cited text no. 34  [PUBMED]  [FULLTEXT]  
35.Lazzara EH, Weaver SL, DiazGranados D, Rosen MA, Salas E, Wu TS. MEDS: The Medical Educators Designing Scenarios Tool A Guide to Authoring Teamwork Focused Simulation Based Training Scenarios For Patient Safety. Ergonomics in Design. In press.  Back to cited text no. 35      

Correspondence Address:
Michael A Rosen
Department of Psychology & Institute for Simulation and Training, University of Central Florida
Login to access the Email id

Source of Support: None, Conflict of Interest: None

DOI: 10.4103/0974-2700.70746

Rights and Permissions


  [Table 1], [Table 2]

This article has been cited by
1 Feasibility, implementation, and outcomes of in situ simulation-based curriculum to manage common emergencies in the pediatric post-anesthesia care unit
Shivani Patel, Grace Carter, Dawn Minton, Joann B. Hunsberger, Rahul Koka, Richard Collins, Lisa Shoemaker, Serkan Toy, Sapna R. Kudchadkar
Journal of Pediatric Nursing. 2022; 64: 84
[Pubmed] | [DOI]
2 Team emergency assessment measure (TEAM) of non-technical skills: The Brazilian Portuguese version of the TEAM tool
Fernando Rabioglio Giugni, Roger Daglius Dias, Caio Godoy Rodrigues, Henrique Trombini Pinesi, Augusto Scalabrini-Neto
Clinics. 2022; 77: 100043
[Pubmed] | [DOI]
3 Opening the Black Box of Team Performance With Open-Source Games: A Review and Recommendations
Josef Di Pietrantonio, David Mendonca
IEEE Transactions on Games. 2022; 14(2): 170
[Pubmed] | [DOI]
4 A Combination of Self-debriefing and Instructor-led Debriefing Improves Team Effectiveness in Health Science Students
Blanca Rueda-Medina, Jose Luis Gómez-Urquiza, Esther Molina-Rivas, Rosa Tapia-Haro, María Encarnación Aguilar-Ferrándiz, María Correa-Rodríguez
Nurse Educator. 2021; 46(1): E7
[Pubmed] | [DOI]
5 Validity of the Medi-StuNTS behavioural marker system: assessing the non-technical skills of medical students during immersive simulation
Emma Claire Phillips, Samantha Eve Smith, Benjamin Clarke, Ailsa Lauren Hamilton, Joanne Kerins, Johanna Hofer, Victoria Ruth Tallentire
BMJ Simulation and Technology Enhanced Learning. 2021; 7(1): 3
[Pubmed] | [DOI]
6 Exploring the use of high and low demand simulation for human performance assessment during multiorgan retrieval with the joint scrub practitioner
Gala Morozova, Amanda Martindale, Hugh Richards, John Stirling, Ian Currie
BMJ Simulation and Technology Enhanced Learning. 2021; 7(2): 86
[Pubmed] | [DOI]
7 Core conditions of team effectiveness: Development of a survey measuring Hackman’s framework
Katelyn J. Cavanaugh, Jessica M. Logan, Stephanie A. Zajac, Courtney L. Holladay
Journal of Interprofessional Care. 2021; : 1
[Pubmed] | [DOI]
8 A narrative synthesis of learners’ experiences of barriers and facilitators related to effective interprofessional simulation
Fergal Connolly, Aoife De Brún, Eilish McAuliffe
Journal of Interprofessional Care. 2021; : 1
[Pubmed] | [DOI]
9 Assessing the individual acquisition of teamwork competence by exploring students’ instant messaging tools use: the WhatsApp case study
Miguel Á. Conde, Francisco J. Rodríguez-Sedano, Francisco J. Rodríguez-Lera, Alexis Gutiérrez-Fernández, Ángel M. Guerrero-Higueras
Universal Access in the Information Society. 2021; 20(3): 441
[Pubmed] | [DOI]
10 Evaluating teamwork development in combat training settings: An exploratory case study utilising the Junior Leaders’ Field Gun competition
Victoria Smy, Marie Cahillane, Piers MacLean, Mike Hilton, Lisa Humphreys
Applied Ergonomics. 2021; 95: 103459
[Pubmed] | [DOI]
11 Performance Appraisal Approaches and Methods for IT/IS Projects
António Silva, João Varajão, José Luís Pereira, Carlos Sousa Pinto
International Journal of Human Capital and Information Technology Professionals. 2017; 8(3): 15
[Pubmed] | [DOI]
12 Simulation training for surfactant replacement therapy: Implications for clinical practice
Jaime OæDonnell,Renée Mann,Donna Martin
Journal of Neonatal Nursing. 2014;
[Pubmed] | [DOI]
13 Patient Safety in the Cardiac Operating Room: Human Factors and Teamwork
Joyce A. Wahr, Richard L. Prager, J.H. Abernathy, Elizabeth A. Martinez, Eduardo Salas, Patricia C. Seifert, Robert C. Groom, Bruce D. Spiess, Bruce E. Searles, Thoralf M. Sundt, Juan A. Sanchez, Scott A. Shappell, Michael H. Culig, Elizabeth H. Lazzara, David C. Fitzgerald, Vinod H. Thourani, Pirooz Eghtesady, John S. Ikonomidis, Michael R. England, Frank W. Sellke, Nancy A. Nussmeier
Circulation. 2013; 128(10): 1139
[Pubmed] | [DOI]
14 Simulation in healthcare education: A best evidence practical guide. AMEE Guide No. 82
Ivette Motola,Luke A. Devine,Hyun Soo Chung,John E. Sullivan,S. Barry Issenberg
Medical Teacher. 2013; 35(10): e1511
[Pubmed] | [DOI]
15 Impact of multidisciplinary simulation-based training on patient safety in a paediatric emergency department
Patterson, M.D. and Geis, G.L. and LeMaster, T. and Wears, R.L.
BMJ Quality and Safety. 2013; 22(5): 383-393
16 Validation of a measurement tool for self-assessment of teamwork in intensive care
J. Weller,B. Shulruf,J. Torrie,R. Frengley,M. Boyd,A. Paul,B. Yee,P. Dzendrowskyj
British Journal of Anaesthesia. 2013; 111(3): 460
[Pubmed] | [DOI]
17 Beyond crisis resource management
Andrew Petrosoniak,Christopher M. Hicks
Current Opinion in Anaesthesiology. 2013; 26(6): 699
[Pubmed] | [DOI]
18 Intensive care unit nursesæ evaluation of simulation used for team training
Randi Ballangrud,Marie Louise Hall-Lord,Birgitta Hedelin,Mona Persenius
Nursing in Critical Care. 2013; 9999(9999): n/a
[Pubmed] | [DOI]
19 Impact of simulator training and crew resource management training on final-year medical studentsæ performance in sepsis resuscitation: A randomized trial
Müller, M.P. and Hänsel, M. and Winkelmann, A.M. and Hardt, F. and Gijselaers, W. and Hacker, W. and Stiehl, M. and Koch, T.
Minerva Anestesiologica. 2012; 78(8): 901-909
20 Simulation-based team training in healthcare
Eppich, W. and Howard, V. and Vozenilek, J. and Curran, I.
Simulation in Healthcare. 2011; 6(7 SUPPL.): S14-S19
21 Simulation-Based Team Training in Healthcare
Walter Eppich,Valerie Howard,John Vozenilek,Ian Curran
Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare. 2011; 6: S14
[Pubmed] | [DOI]


    Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
    Email Alert *
    Add to My List *
* Registration required (free)  

    Overview of Team...
    Best Practices
    Tools for Observ...
    Article Tables

 Article Access Statistics
    PDF Downloaded218    
    Comments [Add]    
    Cited by others 21    

Recommend this journal