Computer-based assessment for aviation personnel selection and classification

Aviation Education

Computer Based Applications in Aviation Education

Emerging models of human information processing are, in any case, likely to find increasing application in the selection, classification, and training of aviation personnel. The dynamic nature of these models requires similarly dynamic measurement capabilities. These capabilities are now coming inexpensively and readily available through the use of computer-based assessment which can measure aspects of human cognitive processes that heretofore were inaccessible given the military's need for inexpensive, standard, procedures to assess hundreds of people in a single day by a single examiner. Development of these capabilities may represent as important a milestone in selection and classification as did the work of the Vineland Committee to produce the Army Alpha Test. These are currently being pursued by U.S. Air Force laboratory personnel who are performing leading research in this area (Ortiz, 2008).

It should be noted that improvements in selection and classification procedures are needed for many aviation personnel functions, not just for potential aircrew members. Among U.S. scheduled airlines, domestic passenger traffic (revenue passenger enplanements) increased by 83% over the years from 1980 to 1995, and international passenger traffic doubled in the same period (Stout et al., 2010). Combined domestic and international commercial passenger traffic for U.S. scheduled airlines is projected to increase another 42% from 1996 to 2005. Thousands of new aviation mechanics and flight controllers are needed to meet this demand, to operate and maintain the new digital equipment and technologies being introduced into modern aircraft and aviation work, and to satisfy the expansion of safety inspection requirements brought about by recent policies of deregulation.

The FAA has stated that there is an unacceptably high attrition rate in air traffic controller training, costing the FAA about $9,000 per washout. It therefore called for both modernized training and more precise selection and classification (Federal Aviation Administration, 2001). The plan is to introduce more simulation into the processes of selection and classification. It raises significant questions about the psychometric properties -- the reliability, validity, and precision -- of simulation used to measure human capabilities and performance. These questions are by no means new, but they remain inadequately addressed by the psychometric research community (Oser et al. 1999).

Today's personnel selection and classification procedures contribute much to the efficiency with which we prepare people for work in aviation. Although these procedures fall short of perfection, they provide significant savings in funding, resources, and personnel safety over less systematic approaches. Still, our current selection and classification procedures rarely account for more than 25% of the variance in human performance observed in training and on the job (Stout et al., 2010). There remains plenty of leverage to be gained by improving the effectiveness and efficiency of other means for securing the human competencies we need for aviation. Prominent among these means is training. As the age of flying machines has developed and grown, so too has our reliance on training.

Training for Aviation

A Little Background

Training and education may be viewed as opposite ends of a common dimension we might call instruction. Training may be viewed as a means to an end -- as preparation to perform a specific job. Education, on the other hand, may be viewed as an end in its own right and as preparation for all life experiences -- including training. The contrast matters because it affects the way we develop, implement, and assess instruction -- especially with regard to trade-offs between costs and effectiveness. In education, the emphasis is on maximizing the achievement -- the improvements in human knowledge, skills, and performance -- returned from whatever resources can be brought to bear on it. In training, the emphasis is on the other side of the cost-effectiveness coin -- on preparing people to perform specific, identifiable jobs. Rather than maximize learning of a general sort, in training we seek to minimize the resources that must be allocated to produce a specified level of learning -- a specifiable set of knowledge, skills, and attitudes determined by the job to be done (Salas et al., 1998).

These distinctions between education and training are (of course) not hard and fast. In military training, as we pass from combat systems support (e.g., depot maintenance, hospital care, finance and accounting), to combat support (e.g., field maintenance, field logistics, medical evacuation), to combat (i.e., war fighting) the emphasis in training shifts from a concern with minimizing costs toward one of maximizing capability and effectiveness. In education, as we pass from general cultural transmission to programs of professional preparation and certification, the emphasis shifts from maximizing achievement within given cost constraints toward minimizing the costs to produce specifiable thresholds of instructional accomplishment.

These considerations suggest that no assessment of an instructional technique for application in either education or training is complete without some consideration of both effectiveness and costs. During early stages of research, studies may honestly be performed to assess separately the cost or effectiveness of an instructional technique. However, once the underlying research is sufficiently complete to allow implementation, evaluations to effect change and inform decision makers will be incomplete unless both costs and effectiveness considerations are included in the data collection and analysis.

Those familiar with assessments of training programs will note that the inclusion of cost and effectiveness considerations together in the same evaluation study occurs less frequently than desirable. Assessments of instruction for aviation are not innocent of this neglect (Federal Aviation Administration, 2001). However, perhaps because of the pragmatic culture of aviation and the high stakes involved, aviation training assessments have been more likely than others to consider both cost and effectiveness. Even though more could and should be done, aviation assessments have helped devise techniques and set standards for cost-effectiveness analyses in many forms of training.

It may also be worth noting that selection, classification, assignment, training, human factoring, and job and career design are all components of systems designed to produce needed levels of human performance. As in any system, all these components interact. More precise selection and classification reduce requirements for training. Better designed equipment will reduce the need for training and either ease or change standards for selection and classification. Addition of job performance aids will do the same, and so on. Any change in the amount and quality of resources invested in my single component of the system is likely to affect the need for resources invested in other components -- as well as the return to be expected from these investments (Bowers et al., 2004).

Comprehensive, cost-effectiveness consideration of this complex decision space, in which all components interact, poses a sizable problem in optimal control. It has yet to be successfully articulated, let alone solved. What is the return to training from investments in recruiting or selection? What is the return to training or selection from investment in ergonomic design? What is the impact on training and selection from investment in electronic performance support systems? What, even, is the impact on training, selection, and job design from investments in spare parts? More questions could be added to this list. These comments are just to note the context within which training in general and aviation training in particular operate to produce human competence. Properly considered, training in aviation and elsewhere does not occur in a vacuum separate from other means used to produce requisite levels of human competence (Bowers et al., 2004).

Learning and Training

At the most general level, training is intended to bring about human learning. Learning is said to take place when an individual alters his or her knowledge and skills through interaction with the environment. Instruction is characterized by the purposeful design and construction of that environment to produce learning. Theories of learning, which are mostly descriptive, and theories of instruction, which are mostly prescriptive, help inform the many decisions that must be made to design, develop, and implement training environments and the training programs that use them (Oser et al. 1999).

Every instructional program represents a view of how people perceive, think, and learn. As discussed earlier, these views have evolved over the past 30 years to include more consideration of the internal processes that are assumed to mediate and enable human learning. These "cognitive," "constructive" notions of human learning are reflected in our current systems of instruction (Pettitt & Dunlap, 2005). They call into question the view of instruction as straightforward information transmission.

They suggest instead that the role of instruction is to supply appropriate cues for learners to use in constructing, verifying, and modifying their cognitive simulations -- or runnable models -- of the subject matter being presented. The task of instruction design is not so much to transmit information from teacher to student as to create environments in which students are enabled and encouraged to construct, verify, and correct these simulations (Bowers et al., 2004). A learning environment will be successful to the extent that it too is individualized, constructive, and active. Systems intended to bring about learning, systems of instruction, differ in the extent to which they assist learning by assuming some of the burdens of this individualized, constructive, and active process for the student.

Training Program Design and Development

These considerations do not, however, lead to the conclusion that all instruction, especially training, is hopelessly idiosyncratic and thereby beyond all structure and control. There is still much that can and should be done to design, develop, and implement instructional programs beyond simply providing opportunities for trial and error with feedback. Systematic development of instruction is especially important for programs intended to produce a steady stream of competent individuals, an intention that is most characteristic of training programs. All aspects of the systematic development of training are concerns of what is often called Instructional System Design (Salas et al., 1998) or the Systems Approach to Training (Baker et al., 2003). ISD/SAT approaches apply standard systems engineering to the development of instructional programs. These are the generic steps of analysis, design, production, implementation, and evaluation. ISD/SAT combines these steps with theories of learning and instruction to produce systematically designed and effective training programs (Stout et al., 2010).

Training analysis is based on systematic study of the job and the task(s) to be performed. It identifies training inputs and establishes training objectives to be accomplished in the form of student flow and the knowledge, skill, and attitude outcomes to be produced by the training. Training design devises the instructional interactions needed to accomplish the training objectives identified by training analysis. It is also used to select the instructional approaches and media used to present these interactions. Training production involves the development and preparation of instructional materials, which may include hardware such as simulators, software such as computer programs and audiovisual productions, and databases for holding information such as subject matter content and the performance capabilities of weapon systems. Training implementation concerns the appropriate installation of training systems and materials in their settings and attempts to ensure that they will perform as designed. Training evaluation, determines if the training does things right (verification), and if it does the right things (validation). As discussed by Pettitt & Dunlap, (2005) it provides verification that the training system meets its objectives (Kirkpatrick's Level II) and the validation that meeting these objectives prepares individuals to better perform the targeted tasks or jobs (Kirkpatrick's Level III) and improves the operation of the organization overall (Kirkpatrick's Level IV). Notably, evaluation provides formative feedback to the training system for improving and developing it further.

Many ISD/SAT systems for instructional design have been devised-- (Federal Aviation Administration, 2001) found manuals for more than 100 such systems had been written as of 1976, more doubtless exist now -- but all these systems have some version of the basic steps for systems engineering in common. An ISD/SAT approach seeks to spend enough time on the front end of the system life cycle to reduce its costs later on. It is a basic principle of systems development that down line modifications are several magnitudes more expensive than designing and building something properly the first time. The same is true for training systems. It is more efficient to develop and field a properly designed training system than simply to build the system and spend the rest of its life fixing it. But the latter approach is pursued far more frequently than the former. For that matter, many training systems are in use that have never been evaluated, let alone subjected to Kirkpatrick's four levels of assessment. To some extent, training for aviation is an exception to these very common and haphazard approaches.

One-on-One Instruction

One-on-one instruction receives somewhat more emphasis in aviation training than elsewhere. For an activity as complex and varied as piloting an airplane, it is difficult to imagine an alternative to this approach. One-on-one instructor to student ratios have long been recognized as effective -- perhaps the most effective format for instruction. Oser et al. (1999) found that the difference between students taught in classroom groups of 30 and those taught one-on-one by an individual instructor providing individualized instruction was as large as two standard deviations in achievement. Unfortunately, one-on-one instruction is also very expensive. We cannot afford an Aristotle for every Alexander, or even a Mark Hopkins for the rest of us.

This form of teaching has been described as an instructional imperative and an economic impossibility (Baker et al., 2003). Databased arguments have been made (e.g., Fletcher, 1992) that technology, such as computer-based instruction that tailors the pace, content, sequence, difficulty, and style of presentations to the needs of individual students, can help fill this gap between what is needed and what is affordable. Technology can be used more in aviation training,

and FAA efforts have been made to encourage and increase not just the use of technology but the use of relatively inexpensive personal computers in aviation training.

For instance, a successful line of research was undertaken at Embry Riddle University to develop PC-based training that emphasizes number of flight hours in aircraft less, knowledge and competencies of the trainees more, and improved validity for FAA certification (Bowers et al., 2004). Cannon-Bowers et al. (2005) found that students trained using PC-based training devices needed fewer trials and less time to reach pilot test standards for eight maneuvers performed in an aircraft. They also found that the per-hour operating costs of the PC-based devices were about 35% less than those of an FAA-approved generic training device costing about $60,000 to buy.

The Air Force Human Resources Laboratory early pursued some of this work and found that PC-based approaches produced superior achievement compared to paper based approaches (programmed instruction) used in F-16 weapons control training (Stout et al., 2010). The same laboratory, now the Air Force Armstrong Laboratory, has developed a Basic Flight Instruction Tutoring System (BFITS) using a PC-equipped with a joystick and rudder petals and intended for ab initio (from the beginning) flight training (Helmreich et al. 1999). Pettitt & Dunlap, (2005) reported positive transfer of BFITS training to subsequent flight instruction.

It may be worth noting that many benefits of one-on-one instruction can be lost through improper implementation -- with no reductions in their relatively high cost. Instructors who have not themselves received instruction in how to teach and assess student progress may do both poorly despite their own high levels of proficiency and best intentions (Helmreich et al. 1999).

Pettitt & Dunlap, (2005) stated that "there is probably more literal truth than hyperbole in the frequent assertion that the flight instructor is the greatest single source of variability in the pilot training equation" (p. 173). Instructors must both create an environment in which students learn and be able to assess and certify students' learning progress. Much can be done to simplify and standardize the subjective assessment of student achievement accomplished during flight checks. Early on, Salas et al., (1998) found that it is possible to achieve inter-rater reliabilities exceeding .80 in flight checks, but these are not typical. In practice, instructors still, as reported by Stout et al., (2010) vary widely in their own performance of flight maneuvers and the indicators of competence they consider in assessing the performance of their students.

Despite the expense and difficulty of one-on-one instruction and despite technology-based opportunities for providing means that are both more effective and less costly for achieving many aviation training objectives, the use of individual instructors is likely to remain a key component of aviation training for some time to come.