《Measurement of Effectiveness for Training Simulations.doc》由会员分享，可在线阅读，更多相关《Measurement of Effectiveness for Training Simulations.doc（14页珍藏版）》请在三一办公上搜索。
1、Measurement of Effectiveness for Training SimulationsDr. J.E. (Hans) Korteling, Dr. E.A.P.B. (Esther) Oprins, Dr. V.L. KallenTNO Human Factors, P.O. Box 23, 3769 ZG, Soesterberg, THE NETHERLANDSABSTRACTThis paper presents and discusses experimental designs, measures, and measurement methods for dete
2、rmining the effectiveness of training simulators. First, we describe experimental designs in which training effects of training simulators are compared to those of conventional training. Next, the most commonly used metrics for quantifying the potential beneficial effects of training applications ar
3、e explicated. We also present and discuss three main categories of measurement methods that may be used to assess the beneficial effects of new ways of training on transfer or training effectiveness; that is, methods based on measurement of learning performance of trainees, methods focusing on the s
4、ynthetic training device or overall training program itself, and ratings or questionnaires focusing on the subjective evaluations of trainees. All designs, metrics, and measurement methods have their specific advantages and limitations, which may make them highly complementary. In general, one shoul
5、d always be aware of the advantages and drawbacks of each method and consider the most appropriate combination of methods for each study, given the main research questions. Therefore, various types of measurement techniques should be used in combination with each other for effective results in order
6、 to meet reliability and validity requirements of training effectiveness studies. Finally, we give some examples of practical approaches and draw conclusions on best practices. 1INTRODUCTIONPeople have long used synthetic environments to simulate reality for training purposes. The first instances of
7、 these “training simulators” or “simulations” were primitive and mainly used for flight training before and during World War 1. Since then, training simulators have expanded to include transport and vehicle systems, military sensor and weapon systems, and complex processes, such as military combat,
8、genocide, and the training of first responders (Schollmeyer, 2006).Training simulators are here defined as devices generating a synthetic and interactive environment on the basis of mathematical, physical, or logical models representing (aspects of) the real (operational) world, in order to obtain t
9、raining goals. The most prominent examples of training simulators are training simulations, instructional computer games, and simulations in e-learning environments. These synthetic training environments may have many potential advantages over real-task equipment and, if adequately designed and used
10、, can substantially reduce the cost of military training and enhance training effectiveness. Since training simulators may also require substantial investment, not only in the device itself but also in instructional personnel and infrastructure, an important question concerns the degree to which the
11、se systems are beneficial and cost-effective.The benefit and cost effectiveness of training systems, however, is often only partially investigated (Cohn, et al, 2009). Many studies have been limited in their design. They focus, for instance, only on fidelity of the simulation (e.g., Allan, Hays & Bu
12、ffardi, 1986), or on trainee reactions or subjective opinions of experts or instructors (e.g., Stehouwer et al., 2005). A meta-analysis by Alliger et al. (1998) has shown that evaluations of training by trainees do not correlate strongly with subsequent performance on the job. In addition, results o
13、f many studies are confounded by characteristics of the educational context in which the simulator is embedded, including the target group, instructional practices, and preconceived opinions of personnel. In addition, not all training simulations, or simulated instructional games, seem to live up to
14、 their potential. With regard to instructional games, Hays (2005) has reviewed 48 empirical research articles on training effectiveness. This report also includes summaries of 26 other review articles and 31 theoretical articles on instructional gaming. The relevant major conclusions and recommendat
15、ions from their report are as follows: Empirical research on the instructional effectiveness of games is fragmented, filled with ill-defined terms, and plagued with methodological flaws. Some games provide effective instruction for some tasks some of the time, but these results may not be generaliza
16、ble to other games or instructional programs. No evidence exists that games are the preferred instructional method in all situations. On the basis of a meta-analysis of the literature on instructional effectiveness of computer-based games, Sitzmann (2011) has drawn similar overall conclusions. The o
17、bjective of training effectiveness measurement is acquiring knowledge concerning the degree to which a training system accomplishes the purposes for which it was developed. These purposes are usually related to real job activities and organizational goals, and thus lie outside the training game or s
18、imulation itself. In many studies, measurements focus on the experiences and opinions of users or trainees with regard to the effectiveness of a training device. Next to that, many measurements and evaluations focus on learning and trainee performance in the training device itself. Both approaches l
19、ack the measurement of real transfer and retention of training results to the workplace, i.e., the situation for which the training was actually intended. Without such measurement, one remains ignorant of the tools effectiveness or quality with regard to enhanced task performance in the operational
20、environment and its organizational impact (Cohn et al, 2009; Kirkpatrick 1959, 1998). In addition, the factors responsible for the tools success or failure will remain obscure. At present, there is no generic framework predicting either the amount of transfer of training or the cost of obtaining it.
21、 Generic knowledge is sparse on the determining factors, as affected by situational factors (such as target group or type of task). This, however, is crucial for decision makers who have to decide about the purchase and application of training simulators. Therefore, in the present paper we will focu
22、s on the designs, measures, and methods available to determine the quality or effectiveness of training simulators in comparison with more conventional training environments, such as military training in the field and/or (embedded) training using operational equipment. In the domain of learning and
23、modeling and simulation (M&S) for training purposes, the concepts of training output are often captured in the term transfer. Transfer denotes the ability to flexibly apply (parts of) what has been learned to new tasks and/or new situations, i.e., real world tasks on the job see, e.g., Baldwin and F
24、ord (1988); Detterman & Sternberg, 1993; Gielen (1995); Mayer & Wittrock, (1996). The degree to which training leads to enhancement of actual behavior on the job is the gold standard of training (Alvarez et al, 2004). In this contribution, we will discuss experimental designs, time- and cost-based m
25、easures, and evaluation instruments (questionnaires, checklists) for determining the benefit of training simulators relative to other forms of training.2EXPERIMENTAL DESIGNS FOR STUDIES ON TRANSFER OF TRAINING2.1Seven designsThis section describes designs that can be used in measuring the training e
26、ffectiveness of training simulators. The descriptions are based on the most common designs, first described by Campbell and Stanley (1963). These designs focus on the comparison of the effect of a treatment with that of no treatment in an experimental setting. Below we have translated these designs
27、into experimental designs, in which the training effects of training simulators are compared to those of conventional training. Experimental-versus-control-group methodThe experimental-versus-control-group method uses a design in which the experimental group is trained with the simulator and the con
28、trol group is trained on real-task equipment only. Afterwards, task performance is measured on real-task equipment on a predetermined criterion task resembling operational task performance. Preferably, performance is also measured before the training (pre-) to get clear data on the actual learning p
29、erformance of the trainees. In this case, the experimental-versus-control-group method is generally thought to be the most appropriate study design to determine whether the simulator has improved real-life performance (Caro, 1977). Self-control-transfer methodAccording to this method, the experiment
30、al group is also the control group. A group of subjects already receiving real-task training would train for a given time on a simulator. Data from subject performance on the real task before synthetic training started are obtained. These data are compared to data of performance obtained on the real
31、 task after synthetic training. The difference between these datasets could be attributed to the simulator. The mayor flaw in this design lies in the absence of a genuine control group. One cannot draw hard conclusions about the effectiveness of the training device because the effect of synthetic tr
32、aining is not compared to a control group that is completely trained on real-life equipment.Pre-existing-control-transfer methodThere are studies in which a concurrently trained control group might not be necessary. For instance, synthetic training can be introduced in an existing training program.
33、Learner performance data from the older or on a predetermined criterion task can be compared to data of performance by the new experimental group who trained with the simulator. This method is called the pre-existing-control-transfer method. Conclusions based on this method are tentative because of
34、time-related changes; for example, in the trainee group, in training methods or circumstances, or in the training staff.Uncontrolled-transfer methodThere are also circumstances where no control group exists. Such a condition can occur when safety plays a role; e.g., forced landing by an airplane. Wh
35、en no control group can be formed, the training effectiveness of the simulator can be established by determining whether subjects can perform the learned task on a real-life system the first time they perform this task. This is called first-shot performance and the method that is based on this kind
36、of measurement is called the uncontrolled-transfer method. Data collected from such studies are tentative, since it cannot be conclusively shown that simulator training has had an effect on the real-task operations performed by the subjects (Caro, 1977). Quasi-transfer-of-training methodBecause of e
37、fficiency (or financial) reasons, one method often applied in validation of training simulators is the quasi-transfer-of-training method (QToT). The difference between the experimental-versus-control-group method and the QToT method is that real-task training occurs in the former (until criterion pe
38、rformance is reached), while it does not in the latter. Experimental groups receive training in the simulator or with the instructional game that has to be evaluated. The control group is trained on a fully operational high fidelity simulator. Eventually, both groups are evaluated on a criterion tas
39、k in this fully operational simulator. The difference in performance reveals the contribution in learning results of the simulator to be evaluated relative to the high-fidelity simulator. Of course, the major limitation of this design is the absence training and performance measurement under real-ta
40、sk conditions.Backward-transfer methodIn a backward transfer study, an operator who has already shown sufficient performance on the relevant task has to perform in the simulator or serious game. If he can perform the task on the synthetic device, backward transfer has occurred. The assumption here i
41、s that transfer of training in the other direction (forward transfer) for learners who have been training on the simulator will also occur.Simulator-performance-improvement methodIn the simulator-performance-improvement method, the performance of a learner is measured in a number of subsequent sessi
42、ons. An essential premise of an effective synthetic training program is improvement in performance by the learners over several sessions of training. If this does not occur, there would be little expectation of improvement in executing the real task. However, the existence of learning in the trainin
43、g simulator or game does not necessarily mean that what is learned is relevant and, thus, transferred to the real, operational-task environment. In general, the assumption of transfer is only plausible if the training environment is a high-fidelity environment with a high degree of physical, functio
44、nal, and psychological fidelity (similarity) with regard to the real-task environment (Korteling, Helsdingen & Theunissen, 2012).2.2DiscussionExcept for the experimental-versus-control-group method, all other (quasi-experimental) methods may be susceptible to questions about their internal validity.
45、 This means that these methods have major limitations for drawing certain conclusions about effects on performance of training manipulations. Generally, strictly controlled experiments permit strong inferences about the effects. However, it is generally difficult to execute these experiments in prac
46、tical settings and the degree to which the results can be generalized will be lower (low external validity). While quasi-experiments may be susceptible to threats of internal validity because of less rigorous control, they allow the researcher to apply less controlled and more realistic contexts (se
47、e also discussion in Section 4.4) and, thus, have a higher external validity, i.e., better generalization, or translatability of results, to operational settings.3TIME- AND COST-BASED METRICS3.1Four metricsHere we present four metrics that have been introduced previously to quantify the potentially
48、beneficial effects of training applications, mainly by Roscoe (e.g., Roscoe & Williges, 1980). These metrics can be adopted for use in the determination of transfer of training, training effectiveness, and cost-effectiveness in training simulators. The type of experimental design needed to do that i
49、s the experimental-versus-control-group method, which was presented above as the most appropriate study design to determine whether the training has improved real-life performance (Caro, 1977). In this kind of experiment, an experimental group is trained with a simulator. After a certain period of time, the group receives additional training in the real-task environment, until the real-task performance of this group reaches a predetermined criterion level. The time needed for the experimental group to
