Ockerman, J. J., Najjar, L. J., & Thompson, J. C. (1997). Evaluation of a wearable computer performance support system. In T. Muldner & T. C. Reeves (Eds.), Educational Multimedia/Hypermedia and Telecommunications, 1997 (pp. 788-793). Charlottesville, VA: Association for the Advancement of Computing in Education.
Jennifer J. Ockerman - jojo@chmsr.isye.gatech.edu
Lawrence J. Najjar - gt4708d@prism.gatech.edu
J. Christopher Thompson - chris.thompson@gtri.gatech.edu
Multimedia in Manufacturing Education Laboratory
Georgia Tech Research Institute MaRC/ATRP
Georgia Institute of Technology
Atlanta, GA, USA
It has been proposed that performance support systems can improve the speed and quality of learning by providing the right information when and where it is needed. However, there have been no formal studies comparing a performance support system to traditional training or support methods. This paper describes an initial study which compared a computer-based performance support system with a book for learning and performing a simple task. Results of this initial study are summarized. Guidance on the design of future performance support systems and suggestions for future studies are also provided.
Performance support systems are computer information systems which are designed to improve employee performance. Performance support systems aid performance by providing only the necessary information at the right time and supporting training when training is needed [Gery 1991]. Performance support systems allow all employees to have expert-like performance. However, not all jobs are computer-based; many workers are mobile and do not use a computer to do their jobs. In these situations, using a wearable computer to deliver a performance support system can improve the effectiveness of the performance support system. In particular, a wearable computer performance support system can:
The use of a wearable computer with a performance support system is a novel concept in performance support [Ockerman, Najjar, Thompson, Treanor, & Atkinson 1995]. To determine whether a wearable computer performance support system can help people perform tasks, we performed an initial evaluation. In the evaluation we compared the use of a wearable computer performance support system to the use of a book to perform three simple tasks. To evaluate the effectiveness of the wearable computer and performance support system, we developed a wearable computer and a simple performance support system. Each of these technologies is described in the following sections.
Figures 1 and 2 show a front and back view of the wearable computer's major components. The user enters and receives information from the computer using a combination microphone/earphone and a head-mounted display. Voice recognition serves as the primary way to control the computer. The computer and batteries are worn on the waist.
Figure 1: Front view of wearable computer components.
Figure 2: Back view of wearable computer components.
Our wearable computer provides many advantages for helping a user to perform tasks. For example, the display allows the user to work while looking at text, drawings, and video. The earphone allows the user to hear explanatory audio narration while looking at information on the display or at the task environment. The microphone, voice recognition software, and our applications allow the user to control the computer via voice so the user's hands are free for other tasks. The computer allows the information designer to use the best medium (e.g., text, graphics, video, sound) to communicate information to the user. The combination of batteries, computer, wireless network, and head-mounted display allow the user to be mobile, getting information when and where the user needs it.
Performance support systems use specialized software which integrates information, tools, and methodologies to help a user perform a specific task [Gery 1991]. We built a performance support system to teach users how to do origami, the ancient Japanese art of paper-folding [Najjar, Ockerman, Thompson, & Treanor 1996]. Origami is a hands-on, slightly complex task which is unfamiliar to most people. The goal of our system was to allow users to learn to fold a simple jumping frog.
To be successful, a performance support system must organize information in a way that is obvious to someone who has not used the performance support system before [Najjar, Ockerman, Thompson & Treanor 1995]. We organized the training information into the following intuitive categories: a brief description of the training goal, the steps to follow to meet the goal, a tool that helps the user correct his or her work, and an on-line library of background information. Figure 3 shows the main menu of our origami performance support system.
Figure 3: Main Menu of Origami Performance Support System.
In our origami performance support system, we tried to use media in a way that helped the user to understand and learn the information [Najjar 1995, Najjar 1996]. For example, in the section that has step-by-step instructions, each step is illustrated with a simple static drawing and accompanying auditory instructions [see Figure 4]. To understand the fold described by the step, the user can examine the drawing for as long as desired and have the instructions replayed. However, since paper-folding is a dynamic task, we also provided an option that allows the user to see and hear a video of someone actually folding the paper [see Figure 5]. The wearable computer allows the user to look at the drawings or videos, and listen to the auditory instructions, while folding the paper.
Figure 4: Sample Static Drawing from Origami Performance Support System.
Figure 5: Sample Video from Origami Performance Support System.
This initial study compared participant performance in three simple tasks while using one of the following two learning systems.
The verbal instructions and static drawings on how to fold a jumping paper frog are identical in both systems. Participants were not given any explanation or prior practice using the learning system they were assigned.
Twenty professionals and students from Georgia Tech participated in the study. All of the participants rated themselves as `very inexperienced' or `inexperienced' in origami on a background questionnaire. The participants were randomly assigned to one of the two learning systems groups (computer or book) with 10 participants in each group.
The participants first completed a background questionnaire to determine their level of experience with origami. Then, using the learning system to which they were assigned, the participants completed three tasks: (1) fold a paper jumping frog, (2) find the term "squash fold," and (3) find the meaning of the word "origami." After completing the three tasks, the participants rated their learning system on a final questionnaire.
For each of the three tasks, both speed and accuracy were recorded by the experimenter. The experimenter recorded by hand the actions of the book users while an electronic log file was kept of each of the computer user's actions. The participants also provided subjective measures of the two learning systems by rating their learning system on a final questionnaire.
One-tailed t-tests were conducted on the speed data from all three tasks. On the first task, folding a paper frog, the book users were significantly quicker than the computer users (book mean = 241.9 sec., computer mean = 384.7 sec, p < 0.05). Although we had hoped that the computer would be more comparable to the book, this result was not surprising for several reasons. First, it takes longer to go through the computer steps, particularly if videos are viewed. It also takes longer for the user to listen to the auditory instructions than it does for a participant to read the instructions. Second, the task was simple enough that the instructions were on only one page of the book. The book users were not required to turn pages to get to the next step, whereas the computer users had to say "next" for the next step. Finally, the book users could look ahead in the instructions, and in some cases did not even read the instructions. This look-ahead ability also helped them to figure out how to make a confusing fold because they could easily compare before and after fold illustrations.
There was not a statistically significant difference between the book and computer users' performance on either of the information location tasks. For the second task, locating the term "squash fold", the book users were slightly quicker (book mean = 31.6 sec., computer mean = 44.8 sec.), while for the third task, locating the meaning of "origami", the computer users were slightly faster than the book users (book mean = 21.7 sec., computer mean = 18.3 sec.). These were surprising results to us. We expected the computer users to be significantly faster, due to the speed of the computer and not having to turn pages. However, the book was small and participants were very familiar with the typical layout of a book (e.g., table of contents, indexes, and introductions). Also, many of the computer users did not use the "Find" function provided in the computer performance support system. Even though there was a "Find" button on every page of the performance support system, many participants either did not see it or did not think it would help in their current task. In many cases the computer participants fell back on their knowledge of books and would go though the pages of the Origami library. It was while paging through the library section that some users finally noticed the "Find" button. Perhaps the use of a "Find" function makes more sense in a library than in the entire performance support system.
The second objective measure was accuracy of completing the first task, folding a paper jumping frog. Inaccuracies were broken into two categories, those inaccuracies which would have no effect on the final product and those which would affect the final product. Along with inaccuracies, we also computed the ratio of fixed mistakes to total mistakes. The last category was the number of participants who made at least one mistake of either type while completing their frog. It was expected that the computer users' performance would be better in all of these categories (i.e., have a lower number) than the book users' performance due to the ability to look at videos of each step. In all but one case we were correct as shown in Table 1.
| Category | Book | Computer |
|---|---|---|
| No effect mistakes | 7 | 4 |
| Mistakes with an effect | 3 | 5 |
| Ratio Fixes to Mistakes | 10/4 = 2.5 | 9/5 = 1.8 |
| At least one mistake | 7 | 5 |
Table 1: Results of accuracy measurements
However, only three participants even used the videos, so it seems unlikely that the ability to see a step done was the reason why computer users made fewer mistakes than book users. Also, two of the people who looked at videos made mistakes. Finally, the computer users made more mistakes that would affect their final outcome than the book users. It appears that more research is needed in this area.
There were four, five-point (1=low, 5=high) rating questions on the final questionnaire. The average ratings for the book and the computer are almost identical as shown in Table 2.
| Questions - Please rate... | Book | Computer |
|---|---|---|
| how much you liked or disliked using the learning system. | 3.9 | 4.1 |
| the effectiveness of the learning system for helping you perform the paer-folding task. | 4.3 | 4.3 |
| the effectiveness of the learning system for helping you to perform the word search tasks. | 4.4 | 4.4 |
| the likelihood that you would want to have this kind of learning system if you had to learn to do paper-folding (origami). | 4.2 | 3.9 |
Table 2: Results of subjective ratings questionnaire
These results were a little surprising since we thought that the participants would like the computer performance support system better than the book. We were surprised at the high ratings on the book. We thought that the book users might become frustrated and have no way to get around their frustrations since there were no videos. However, it appears as if the task was too simple to take advantage of the video feature of the computer performance support system.
From written comments it appears that the computer users would like the system on a desktop computer but did not enjoy using the head-mounted display. This probably affected their rating of the entire system to some degree.
This initial study highlighted many important issues with the wearable computer performance support system. These issues range from design of the human-computer interface to experimental design. We identified many ways to improve our interface, and offer these as suggestions for future designs of task support software. It is obvious that users fell back on their knowledge of paper books in order to interact with the computer-based system. It would be a good idea to take advantage of this prior knowledge in the interface design. For example, the "Find" button should be labeled "Index". Another possibility is to provide a before and after view of a step in order to confirm that the step was being done correctly. This would provide one function of a video but be much quicker for the user to access. Abilities of the computer system must be clearly marked and noticeable. Neither the "Find" button nor the "Show Video" buttons were used very extensively due to their not being noticed. Perhaps the system should suggest a video after a certain length of time or several requests for repeated instructions.
Wearable performance support systems would seem to be of most benefit to a user when the tasks to be performed are very complex or the environment in which the user must operate prohibits the use of standard materials such as operating manuals. This study did not include these mitigating factors. Consequently, the difference between using the two support systems was minimal. It is encouraging to note in such circumstances that the wearable system did not hinder performance of the user. Future study is planned to verify this hypothesis. Specifically, we will study more complex tasks as well as environmental factors which restrict the use of standard support materials such as books.
Funding for this research was provided by the state of Georgia as part of the Agriculture Technology Research Program.
[Gery 1995] Gery, G. (ed.). Special issue on electronic performance support systems. Performance Improvement Quarterly 8, 1, 1995 (entire issue). Available at http://www.cet.fsu.edu/sy2000/PIQ/PIQContents.html.
[Gery 1991] Gery, G. J. (1991). Electronic Performance Support Systems. Weingarten Publications, Boston, MA.
[Najjar 1995] Najjar, L. J. (1995). A Review of the Fundamental Effects of Multimedia Information Presentation on Learning (GIT-GVU-95-20). Georgia Institute of Technology, Graphics, Visualization, and Usability Center, Atlanta, GA. Available at http://www.cc.gatech.edu/gvu/research/techreports95.html.
[Najjar 1996] Najjar, L. J. (1996). Multimedia information and learning. Journal of Educational Multimedia and Hypermedia, 5, 129-150.
[Najjar, Ockerman, Thompson, & Treanor 1996] Najjar, L. J., Ockerman, J. J., Thompson, J. C., & Treanor, C. J. (1996). Building a demonstration multimedia electronic performance support system. Educational Multimedia and Hypermedia, 1996, Association for the Advancement of Computing in Education, Charlottesville, VA. 794.
[Ockerman, Najjar, Thompson, Atkinson, & Treanor 1996] Ockerman, J. J., Najjar, L. J., Thompson, J. C., Atkinson, F. D., & Treanor, C. J. (1996). FAST: A research paradigm for educational performance support systems. Educational Multimedia and Hypermedia, 1996, Association for the Advancement of Computing in Education, Charlottesville, VA. 545-550.