 |
|
Figure 1: Wearable Computer System. |
Ockerman, J. J., Najjar, L. J., & Thompson, J. C. (1999). FAST: Future technology for today's industry. Georgia Tech Research Institute Journal of Technology, 2.
Reprinted from Computers in Industry, Vol 38 , Jennifer J. Ockerman, Lawrence J. Najjar and J. Christopher Thompson, FAST: Future Technology f or Today's Industry, pp 53-64 , Copyright 1999, with permission from Elsevier Science.
Jennifer J. Ockerman, Lawrence J. Najjar and J. Christopher
Thompson
Multimedia in Manufacturing Education Laboratory
Georgia Tech Research Institute
Georgia Institute of Technology
Atlanta, GA 30332-0823
The age of downsizing is upon us. As a result, industrial workers are being asked to become more efficient, productive, and flexible with fewer resources. This paper discusses one way that wearable computers and multimedia electronic performance support systems are being used to meet these challenges. Our laboratory is working on a project we call Factory Automation Support Technology (FAST). FAST utilizes the power of multimedia to improve worker performance throughout the factory. Task-oriented drawings, schematics, sounds, animations, video, procedures, checklists, and expert advice are continuously available from the FAST system. The multimedia electronic performance support system improves job performance while promoting learning as workers do their jobs. Utilizing a novel wearable computer equipped with a head-mounted display, a wireless network link, and voice recognition software, the user is able to retrieve and enter job related information without using his or her hands while roaming throughout the plant. The FAST system provides training and support when, where, and how it is most needed in a completely unrestricted manner. This paper describes this new method of multimedia-enhanced support as well as the results from initial trials.
Multimedia offers a variety of techniques to present information, including text, graphics, sound, animation, and video. When combined with the interactive power of a computer, multimedia can provide a completely personalized view of this information, allowing the user to select the view that is preferred at any moment. To date, this technology has seen only limited use in manufacturing.
Manufacturing systems constructed thus far have focused primarily on automating the flow of information throughout the production facility. Several vendors and researchers are experimenting with the concept of the "paperless" factory whereby shop floor orders and associated engineering drawings are electronically stored, updated, and transferred in real-time via a computer network. Others have used multimedia in manufacturing control systems to allow the operator to have a first-hand look at the actual process while remaining at the controls of the process ([1], [2]). Multimedia can also be used for scheduling. Schedulers may run multimedia simulations to confirm the accuracy and sufficiency of their schedules. Collaborative design is another area being improved with multimedia ([3]). Designers in different geographical locations can design together through a distributed multimedia system.
The Factory Automation Support Technology (FAST) project in our laboratory, the Multimedia in Manufacturing Education (MiME) laboratory, focuses on the use of multimedia for two primary purposes: 1) to improve human performance in manufacturing systems and 2) to link mobile personnel with plant-wide databases and experts in real-time. Human performance is supported through timely presentation of technical information or advice necessary to operate, adjust, or repair advanced complex automation. Mobile personnel are able to continuously update and retrieve information from a central database while roaming throughout a large space. Mobile personnel may also make video connections with experts to get help with their current tasks. Significant changes are occurring in the workplace today that make these multimedia capabilities essential to future manufacturing competitiveness.
Today's workplace is continuously changing. Automation is increasing. The number of employees is being reduced via downsizing or "right-sizing." Workers are more transient in terms of companies as well as jobs within a company.
The increasing levels of automation require that employees know how to maintain and keep the automation running. More sophisticated automation often requires that the work force be better educated. When organizations downsize or right-size, machines are often used to replace manual human labor, but fewer employees are available to deal with the more sophisticated equipment. So more work has to be done by fewer people. Since automation is often distributed throughout a factory, if not throughout the world, technicians must routinely travel to the automation to do their work. Due to their mobility, these workers cannot easily be supported by traditional means, such as human experts, equipment providers, or paper-based documentation.
To meet these challenges, we need to rethink both support and training in the manufacturing context. This new workplace environment often requires more complex skills, skills which are not quickly mastered and may be infrequently used. There are several drawbacks to the way support and training are currently provided. These drawbacks are summarized in the following section.
Traditional training is both costly and time-consuming ([4]). Often, training requires employees to be absent from their jobs. This approach results in significant costs on two fronts: 1) technical training is often quite expensive and 2) productivity is reduced as a result of the missing employees. Often training requires employees to travel to a different location, further increasing the costs.
Training is not immediate ([4], [5], [6]); that is, training is done at a time other than when the employee needs to use the skill or information. Training is usually done prior to the task that is being trained and is often partially, if not entirely, forgotten by the time it is finally needed on the job. Due to schedules and conflicts, it is rare to have training directly precede its need on the job. Also, since most training is not performed in the context of the job, it is difficult for employees to transfer what they learn in class to their actual job ([7], [8], [9]). As a result, an increase in performance is seldom realized.
Training is geared toward increasing knowledge as opposed to improving performance ([6]). Training is often directed toward the goal of improving the employees' knowledge in the hope that this will improve performance. However, knowledge is not the same as skill in many occupations. Since the true business goal of training is to improve the performance of the work force, training is currently not directly serving this goal.
Training is trainer-centered as opposed to learner-centered ([10]). The trainer decides what the employee should know as opposed to the employee asking for the information that the employee needs to get the job done. With the trainer in control, training tends to be less focused on the actual job and sometimes is only indirectly related to what the employee has to do. For example, the training for software applications is often developed for employees of several organizations and is not tailored for how the software will be used at any one particular organization.
Finally, training is most often evaluated on learner satisfaction and attainment of classroom goals instead of job performance ([6]). Good job performance should be the true goal of training, but performance is not part of the training environment and is difficult to evaluate. Thus, trainers often resort to participant satisfaction as a measure of the effectiveness of their training; this is an indirect measure at best.
Electronic performance support systems (EPSSs) have been promoted as a solution to the drawbacks of traditional training ([4], [11], [12]). Table 1 compares aspects of traditional training with the Factory Automation Support Technology (FAST) which incorporates electronic performance support system concepts. The goal of EPSSs is to provide a worker with the right information, in the right quantity and detail, at the right time ([4]). This includes just-in-time training in the tasks the worker needs to do. In other words, electronic performance support systems allow less proficient workers to perform as more experienced workers. Industry is interested in this approach because EPSSs not only improve on the job performance but also reduce the time and cost of training ([4]). EPSSs support and train employees as they are performing their jobs, rather than before. This is a major shift in the way training is currently conducted.
| Traditional Training | Factory Automation Support Technology |
| Training is not integrated with everyday work environment or shop floor process. | Focus on continual learning process in the work environment; not limited to training; assistance provided at moment of need. |
| Training is done before doing the job task being trained. | Training is done while doing the job task being trained. |
| Training is focused on increasing knowledge about the job task. | Assistance is provided to improve performance of job task. |
| Training is trainer-centered; the responsibility for teaching is on the trainer or training system. | Learner is responsible for defining learning goals for getting the job done. |
| Assessment of training is based on learner satisfaction and attainment of classroom objectives. | Assessment of assistance is based on job performance. |
Until now it has been difficult to deliver a performance support system to workers who do not sit at a desk or who need to access information anywhere throughout a factory. These kinds of workers may include quality assurance inspectors, technicians, and production supervisors. One innovative way to provide a performance support system to these mobile workers is to furnish them with a highly portable, wearable computer. Such an approach enables mobile workers to take the EPSS with them wherever they go. Additionally, by utilizing wireless networking technology, workers can access centralized databases and establish live links with remote resources distributed throughout the world (e.g., through the Internet and company intranets).
The goal of the FAST project is to explore the issues associated with an electronic performance support system for a mobile industrial workforce. The following sections describe the current FAST hardware and software. Later sections discuss our current research and future plans.
FAST hardware is a wearable, multimedia-capable, voice-activated computer system. Figure 1 displays the system as worn by a user. The overall system is described in the following paragraphs.
|
|
Figure 1: Wearable Computer System. |
The first component is a head-mounted display (HMD) which allows the user to interact with the environment while viewing text, drawings, animations, and/or videos that are pertinent to what the user is currently doing. The display is limited to 256 grey levels; however, in the near future low-cost color technology is expected to become available. The position in space of the computer image can be adjusted by the user for maximum viewing comfort. The display fits comfortably over glasses and can be tilted up out of the way when it is not needed.
Next, the system includes a miniature microphone/earphone headset which provides audio information to the user as well as accepts voice input from the user. The audio information can include explanatory narrations, audio prompts or reminders, alarm signals, or diagnostic cues. Voice activation allows the user to keep his or her hands free for job-related tasks while interacting with the EPSS. For noisy industrial environments, a headset which includes a noise-canceling microphone and stereo earphones is used. The sound-muffling earphones protect the user's hearing while the noise-canceling microphone allows the user to maintain audio communication with the computer. This technology has been proven effective in environments where the ambient noise level is above 90dB.
A third component is a small video camera attached to the front of the headset. The camera enables the user to transmit his or her view of the workplace to a distant site to aid in remote collaboration. The camera may also be removed and used as a hand-held camera to access hard to see places.
The fourth component is a wireless communications device which sends and receives up-to-date information to and from a plant computer system. Utilizing commercially available wireless LAN adapters, a range of 600 ft. between stations is feasible in an open area. By connecting to a shared database, the plant management can access real-time information (such as quality assurance data) as it is gathered by mobile workers. Also, the mobile user can access additional information that is not stored on the wearable computer or connect with remote sites via the Internet.
The ultra-portable, wearable computer is the fifth component. The computer is worn on a belt at the waist allowing the user to enter and receive information as he or she moves throughout a plant. In addition to voice activation, the wearable computer is equipped with a small touch pad to move the display cursor and select objects on the display. This device is included as a backup to the voice control.
The current wearable computer we are utilizing includes a 486 75Mhz Intel processor, 24MB of RAM, a 340MB hard disk, a SVGA display controller, serial and parallel ports, a mouse port, 16 bit audio, and 3 additional PCMIA expansion slots. The current generation of the wearable computer weighs approximately 3 pounds plus batteries.
The final component is a battery pack to supply power to all the components. The battery is worn on the belt around the waist or within the same container as the computer. Currently, we are using rechargeable nickel-metal-hydride (NiMH) batteries which last five hours or more depending on how the computer is utilized. Other battery technologies, including lithium-ion (Li-ion) and Zinc-air, are also being explored as potential future power sources each offering tradeoffs between weight, size, energy density, safety, and cost. Discharged batteries can be replaced while the computer continues to function thus permitting long periods of uninterrupted operation. Remaining capacity of the batteries can be continuously monitored and displayed in the corner of the HMD, thus permitting the user to know at all times how long he or she can go before replacing the batteries.
These components work together to allow the user to access multimedia computing power equal to that normally found only at a fixed workstation. The user can also access remote resources via the wireless communications link. As with any new technology designed to interact with the human body, FAST requires us to look at not only the technical capabilities of the technology but also the equally important ergonomic issues involved. The next section describes some of the issues we are exploring.
Wearable computers in a factory setting present a number of ergonomic concerns including safety, durability, and usability and comfort. For safety, it is important that any exposed interconnecting wires cannot get caught in or on machinery. Additionally, little is known about the long term effects of using a head-mounted display on a daily basis. In the noisy facility, FAST must protect an employee's hearing while not interfering with two-way audio communication between the user and the computer.
For durability, a major issue is letting the workers do their jobs without worrying about bumping the computer, or breaking a headset. The hardware must contend with continuous use on a daily basis and not be prone to breakdowns. The system also needs to withstand small amounts of moisture and dirt.
For usability and comfort, it is important that the system be easily put on and comfortably worn for extended periods of time. This translates to minimum weight and size which must be weighed against a need for maximal computing power. The components should be adjustable to accommodate differences between users. Additionally one must decide where on the body the computer should be worn to keep it out of the way while maintaining comfort in as many postures as possible.
Our goal is to make the FAST system so convenient, comfortable, and helpful that workers will want to use the system. We are improving the system iteratively to make it lighter, faster, more powerful, and more comfortable. In an effort to address these concerns we are exploring a number of ergonomic concepts including belt, shoulder, arm, back, leg, foot, and head mounted designs. Ultimately, we would like to integrate the computer with the headset into a single piece design.
FAST enables employees to obtain multimedia task support anywhere and anytime they need it. Relying on voice-activated data entry and control, the system allows a worker to augment his or her job site with pertinent information while freely interacting with the surrounding environment. This wearable computer system complements software-based performance support systems by making them accessible to workers at all times and in all places during the work day.
The software component of the FAST system is an electronic performance support system (EPSS). A typical EPSS provides a combination of the information shown in Figure 2 ([4], [11], [13]). As shown in Figure 2 the employee is both a worker to be supported and a learner to be trained. The employee as the worker/learner accesses the provided information through the user interface.
 |
|
Figure 2: Components of an Electronic Performance Support System. |
The four most common kinds of support information are: reference material, just-in-time training, job advice, and performance support system advice. Reference material describes the task and other related tasks that the employee may need to perform. Reference material can also include data from manuals or procedural guidelines. This reference information supports employees by making immediately available information which they previously had to memorize or look for in a manual. For example, if a quality control inspector in a factory needs to check that a machine reading is within tolerances, the tolerances in both textual and graphical manner can be supplied by a performance support system. As a result, the inspector does not have to remember the tolerances or to go back to his or her desk to compare the reading to the specifications in a manual. The reference section not only makes workers more efficient, but it also allows them to learn more deeply about a given task. The reference material is always available for the worker to review and can provide explanations for the job tasks it supports.
Just-in-time, task-specific training reduces pre-job training by helping workers to learn while doing their jobs. For example, instead of looking up information in a textbook that was used six months ago in a training class, an employee can quickly access the on-line procedures for resetting a piece of machinery that has gone out of tolerance limits. This can be done while the worker is standing in front of the piece of machinery. In some cases, the worker may be able to do a novel repair without previous training in that specific repair by using training that is provided by the EPSS.
Performance support systems often contain specific advice on performing job tasks and using the performance support system to its greatest advantage. The advice may be provided by an expert system. Expert job advice aids employees in reasoning about their tasks. For example, an expert job advisor may include information from maintenance experts to help an employee troubleshoot a piece of machinery to determine why it is out of operating limits. Expert systems can also advise the user how to use the performance support system more effectively. An expert performance support system advisor provides information about the performance support system. For instance, when the outcome of a procedure does not match the expected one, the expert performance support system advisor may suggest that the employee use the expert job advisor. The expert performance support system advisor aids the employee in knowing when to ask for help and where to get the help.
Two other types of information are application help functions and automated tools for task performance. These types of information are most helpful when a supported task involves the use of a computer. For example, application help information can assist a worker with using an application necessary to perform his or her job (e.g., a spreadsheet program). Automated tools help a worker perform a high-level task by doing lower-level tasks automatically. For example, an automated tool may aid a quality assurance inspector by graphically depicting quality data trends throughout a shift and calculating the average and total of quality data at the end of a shift.
All this information is useful only if it is organized in a manner conducive to improving performance. There are two aspects to this organization: 1) task-oriented organization of information and 2) user knowledge level organization. Both of these aspects are described below.
We believe that an important aspect of a performance support system is that it is task-oriented (see Figure 3). The task that needs to be accomplished provides the organizational framework of the information and drives the design of the user interface. For example, reference material and just-in-time training are simply used to improve job performance. If an employee has trouble with a particular task, then reference material and training that is needed to complete the task are provided as the employee continues to work on the task. The employee is not required to leave the task to complete a tutorial or obtain reference material.
 |
|
Figure 3: Task-Oriented Electronic Performance Support System. |
As Figure 3 shows, the user interface of a performance support system reflects the worker's tasks. The interface provides the user access to the elements in the context of the job. For example, it is not expected that the user will go through training and then do the job. So the training is distributed appropriately across the tasks which compose the job. The system identifies the task the worker is performing with the kind of support information that is needed.
You can also view the structure of the performance support system in the way it supports the entire novice to expert continuum ([14]) (see Figure 4). It is necessary to structure the information in such a way that the expert is not hindered but the novice can get the help that he or she needs. To accomplish this we have divided our information into several layers; each supports a different level user with appropriate information technology (see Figure 4). The lower the level, the more support that is provided. The top level, expert workspace, allows the expert user to simply do the job. However, it also supports the pre-expert with a critiquer ([15], [16], [17], [18]). A critiquer is built on top of an expert system and monitors the user's actions. When the user's actions do not fit the expectations of the critiquer, the user is notified. This notion is similar to automatic error checking. The help level uses the expert system to tell the user what to do in a step-by-step fashion. The examples level utilizes a case-base ([19], [20], [21]) to provide the user with past experiences in similar situations. Finally, the training level provides the user with computer-based training on the task the user is trying to accomplish. The help, examples, and training levels are provided based on the task that is being completed. In most uses, these three levels will make use of multimedia to better explain the information that is being presented.
 |
|
Figure 4: Support of Expert/Novice Continuum. |
As can be seen from the above description, performance support systems contain a broad range of information. Multimedia is a helpful way to present this information to the user ([22], [23]). Multimedia includes text, drawings, photographs, animations, videos, sounds, and audio narration. We also use multimedia because it can help people learn more information in less time ([24], [25], [26], [27], [28], [29], [30], [31], [32]). Also, people prefer multimedia learning materials and believe that multimedia helps them to learn ([24], [25], [33], [34], [35], [36], [37]).
We are exploring the ability of the FAST system to improve performance in industrial plants. We are building two prototype industry applications, QALive and WaRP. QALive uses the FAST system to allow quality assurance inspectors to enter data directly into a plant's main computer system while the inspectors handle products and move around the facility. The system also gives the plant management real-time access to this crucial information. WaRP (Water Reduction Project) supports an environmental engineer's efforts to reduce water usage within a facility. Each of these applications are explained in more detail below. In addition to these prototypes, we have completed some evaluation activities of the both the hardware and software components. The evaluation activities are described at the end of this section.
QALive uses an electronic performance support system and a wearable computer to enable quality assurance inspectors to input inspection measurements from anywhere in a plant into a central database while their hands are busy inspecting products. The performance support system will tell an inspector how to do specific inspections, what to do if a measurement does not meet the quality standards, and how to notify supervisors when quality standards are not met. Most of this information will be communicated using multimedia. The performance support system will also automatically perform routine, error-prone calculations and integrate the quality information into daily reports.
We demonstrated a portion of the data collection system to plant management and representative quality assurance workers at a poultry plant. One quality assurance inspector used a prototype system to collect temperatures of poultry meat (white, dark, and tenderloins) in the actual noisy work environment. The technology worked flawlessly during this test. The prototype provided a data collection interface (see Figure 5), voice commands and input, and storage of the data in a database on a central computer. The inspector also enjoyed using the prototype system. We received helpful suggestions for improving the FAST system in both hardware and software aspects. Both managers and workers were very enthusiastic about the system and saw how it would improve plant operations and the convenience of the quality assurance inspectors' jobs. Figure 6 shows a quality assurance inspector wearing an early version of our wearable computer.
 |
|
Figure 5: White Meat Input Screen from QALive. |
 |
|
Figure 6: Quality Assurance Inspector Wearing Early Version of Wearable Computer. |
The second prototype, WaRP, is a performance support system that trains and aids plant personnel in conducting water reduction audits. The final system will allow an employee to walk to various sites inside and outside a poultry facility and perform specific water reduction tasks. The performance support system will use text, audio, drawings, and video to show the employee how to measure water flow, adjust water valves, and calculate water usage. We are working with environmental engineers at the Georgia Tech Research Institute who currently perform this task at various sites across the state of Georgia. The goal is to allow maintenance personnel within the poultry plants to monitor and correct their own water usage problems.
Some sample screens from WaRP are shown in Figures 7 and 8. Figure 7 represents the expert workspace. The expert can simply fill in the information that is required and the system uses the information to calculate water usage. Less expert workers can get help by either clicking on the text they want explained or using a coach accessed through the menu bar. The coach leads the user through the activity of completing the form in the expert workspace. Figure 8 is an example of a help screen which contains a sample water bill. Other media, which cannot be shown on a written page, are also employed to explain the concepts involved in a water audit. The first phase of this project has been completed and the system will be taken to a poultry plant for evaluation.
 |
|
Figure 7: Expert Workspace of WaRP. |
 |
|
Figure 8: Help Screen from WaRP. |
In addition to the projects described above, we have tested the audio and video capabilities of the remote collaboration. Technically, this system worked extremely well. However, integrating this capability into the rest of a collaborative system remains a research issue at this time. We also ran a small experiment with a demonstration EPSS which explained how to make a paper jumping frog. Results from this experiment demonstrated that the wearable computer in conjunction with an EPSS is a feasible method of aiding performance of a new task [38].
Multimedia can be a powerful tool for improving human performance in manufacturing operations. By combining an electronic performance support system and wearable computer technology, FAST uses multimedia to make manufacturing workers more productive. Learning and job support needs of workers are addressed as they perform their jobs. FAST reduces traditional training drawbacks by providing learner-centered training when, where, and how it is needed. FAST also meets the challenges of today's workplace presented by increasing automation, downsizing, and more transient and mobile work forces.
This is just the beginning of a promising new direction for the use of multimedia to improve manufacturing worker's performance. There are many issues still to be addressed, but the groundwork has been laid and initial systems have received positive feedback from the user community. This technology will have a positive impact on the manufacturing community in the coming decades.
[1] Zuboff, S., In the Age of the Smart Machine: The Future of Work and Power. 1988, New York, NY: Basic Books, Inc., Publishers.
[2] Guha, A., et al., Controlling the process with distributed multimedia, in IEEE MultiMedia. 1995. p. 20-29.
[3] Anupam, V. and C.L. Bajaj, Shastra: Multimedia Collaborative Design Environment, in IEEE MultiMedia. 1994. p. 39-49.
[4] Gery, G.J., Electronic Performance Support Systems. 1991, Boston, MA: Weingarten Publications, Inc. 302.
[5] Zemke, R. and S. Zemke, Adult Learning: What do we know for sure?, in Training. 1995. p. 31-40.
[6] Carr, C., Smart Training: The manager's guide to training for improved performance. 1992, New York, NY: McGraw-Hill, Inc. 273.
[7] Royer, J.M., Theories of the Transfer of Learning. Educational Psychologist, 1979. 14: p. 53-69.
[8] Salomon, G. and D.N. Perkins, Rocky Roads to Transfer: Rethinking Mechanisms of a Neglected Phenomenon. Educational Psychologist, 1989. 24(2): p. 113-142.
[9] Butterfield, E.C. and G.D. Nelson, Theory and Practice of Teaching for Transfer. Educational Technology Research and Development, 1989. 37(3): p. 5-38.
[10] Connor, J.J., On-the-job training. 1983, Boston, MA: International Human Resources Development Corporation. 89.
[11] Stevens, G.H. and E.F. Stevens, Designing electronic performance support tools. 1995, Englewood Cliffs, NJ: Education Technology Publications. 265.
[12] Carr, C., PSS! Help When You Need It, in Training & Development. 1992. p. 31-38.
[13] Reynolds, A. and R. Araya, Building Multimedia Performance Support Systems. 1995, New York, NY: McGraw Hill, Inc.
[14] Olsen, S.E. and J. Rasmussen, The Reflective Expert and the Prenovice: Notes on Skill-, Rule-, and Knowledge-based Performance in the Setting of Instruction and Training, in Developing Skills with Information Technology, L. Bainbridge and S.A.R. Quintanilla, Editors. 1989, John Wiley & Sons Ltd.
[15] Fischer, G., et al., Critics: an emerging approach to knowledge-based human-computer interaction. International Journal of Man-Machine Studies, 1991. 35: p. 695-721.
[16] Fischer, G., et al., The Role of Critiquing in Cooperative Problem Solving. ACM Transactions on Information Systems, 1991. 9(3): p. 125-151.
[17] Guerlain, S. Using the Critiquing Approach to Cope with Brittle Expert Systems. in Human Factors and Ergonomics Society 39th Annual Meeting. 1995.
[18] Silverman, B.G., Survey of Expert Critiquing Systems: Practical and Theoretical Frontiers. Communications of the ACM, 1992. 35(4): p. 107-127.
[19] Kolodner, J., Case-based Reasoning. 1993, San Mateo, CA: Morgan Kaufmann Publishers, Inc.
[20] Domeshek, E.A. and J. Kolodner, Toward a case-based aid for conceptual design. International Journal of Expert Systems, 1992. 4(2): p. 201-220.
[21] Williams, S.M., Putting Case-Based Instruction Into Context: Examples From Legal and Medical Education. The Journal of the Learning Sciences, 1992. 2(4): p. 367-427.
[22] Kozma, R.B., Learning with Media. Review of Educational Research, 1991. 61(2): p. 179-211.
[23] Kozma, R.B., Will Media Influence Learning? Reframing the Debate. Educational Technology Research and Development, 1994. 42(2): p. 7-20.
[24] Bosco, J., An analysis of evaluations of interactive video. Educational Technology, 1986. 25: p. 7-16.
[25] Fletcher, D., The effectiveness and cost of interactive videodisc instruction. Machine-Mediated Learning, 1989. 3: p. 361-385.
[26] Khalili, A. and L. Shashaani, The effectiveness of computer application: A meta-analysis. Journal of Research on Computing in Education, 1994. 27: p. 48-61.
[27] Kulik, J.A., R.L. Bangert, and G.W. Williams, Effects of computer-based teaching on secondary school students. Journal of Educational Psychology, 1983. 75: p. 19-26.
[28] Kulik, J.A., C.C. Kulik, and R.L. Bangert-Downs, Effectiveness of computer-based education in elementary schools. Computers in Human Behavior, 1985. 1: p. 59-74.
[29] Kulik, J.A., C.C. Kulik, and P.A. Cohen, Effectiveness of computer-based college teaching: A meta-analysis of findings. Review of Educational Research, 1980. 50: p. 525-544.
[30] Kulik, C.C., J.A. Kulik, and B.J. Shwalb, The effectiveness of computer-based adult education: A meta-analysis. Journal of Educational Computing Research, 1986. 2: p. 235-252.
[31] Najjar, L.J., Multimedia Information and Learning. Journal of Educational Multimedia and Hypermedia, 1996. 5:p. 129-150.
[32] Schmidt, M., et al., Computer-assisted instruction with exceptional children. Journal of Special Education, 1985. 9: p. 493-502.
[33] Bryant, J., et al. Humorous illustrations in textbooks: Effects on information acquisition, appeal, persuasabilty and motivation. in Speech Communication Association Convention. 1980. New York, NY.
[34] Holliday, W.G., L.L. Brunner, and E.L. Donais, Differential cognitive and affective responses to flow diagrams in science. Journal of Research in Science Teaching, 1977. 14: p. 129-138.
[35] Rigney, J.W. and K.A. Lutz, Effect of graphic analogies of concepts in chemistry on learning and attitude. Journal of Educational Psychology, 1976. 68: p. 305-311.
[36] Samuels, S.J., E. Biesbrock, and P.R. Terry, The effect of pictures on children's attitudes toward presented stories. Journal of Educational Research, 1974. 67: p. 243-246.
[37] Sewell, E.H. and R.L. Moore, Cartoon embellishments in informative presentations. Educational communication and Technology Journal, 1980. 28: p. 39-46.
[38] Ockerman, J.J., L.J. Najjar, and J.C. Thompson, Evaluation of a wearable computer performance support system in Educational Multimedia/Hypermedia and Telecommunications, 1997. Charlottesville, VA.