Ockerman, J. J., Najjar, L. J., & Thompson, J. C. (1997). Wearable computers for performance support: Initial feasibility study. In Digest of papers: First International Symposium on Wearable Computers. (pp. 10-17). Los Alamitos, CA: IEEE Computer Society.
Jennifer J. Ockerman
Lawrence J. Najjar
J. Christopher Thompson
Multimedia in Manufacturing
Education Laboratory
Georgia Tech Research Institute
Georgia Institute of Technology
Atlanta, GA 30332-0823
jojo@chmsr.isye.gatech.edu
gt4708d@prism.gatech.edu
chris.thompson@gtri.gatech.edu
Abstract
We have suggested that wearable computers combined with a performance support system could be a useful resource for mobile workers. Literature provides little guidance as to whether this is a feasible solution for aiding mobile workers in the performance of their tasks. In order to evaluate the feasibility of this concept a small, initial study was conducted. The initial study compared a voice-activated, computer-based performance support system with a book for learning and performing three simple tasks. The study demonstrated the feasibility of a performance support system on a wearable computer, but did not show it to be superior to the book in this context. Guidance on the design of future performance support systems and suggestions for future studies are provided.
Keywords: Wearable computers, Performance support systems, Evaluation
As we have suggested [6], combining wearable computers with the concept of performance support systems [2] extends the applicability of performance support systems to mobile workers. Until recently, mobile workers could not take advantage of computer-based automated and aiding systems. Performance improvements due to technology had been confined to desk jobs, where a computer was already available. With wearable computers, the advances made with desktop computers in supporting the jobs of office workers can be provided to mobile workers as well.
One of the most recent advances in office technology is the concept of performance support systems. Performance support systems are computer information systems which are designed to improve employee performance. Performance support systems aid performance by providing only the necessary information at the right time and supporting learning when training is needed [1, 2]. Providing a performance support system on a wearable computer can:
To determine whether a wearable computer performance support system might help mobile workers perform tasks, we performed a small, initial evaluation with a demonstration performance support system we had built. In this first evaluation we were simply looking at the use of a head-mounted display and voice commands to aid a person in performing an unfamiliar task for the first time. In the evaluation we compared the use of a wearable computer performance support system to the use of a book to perform three simple tasks. We chose to compare the computer performance support system to a book because we wanted to know how performance might be changed by going from paper-based instructions to wearable computer-based instructions. For this experiment the participants performed the tasks sitting at a table.
In order to perform this evaluation and to learn about performance support systems, we developed a very simple demonstration performance support system in Macromedia's Authorware 3.5. Authorware is especially designed to facilitate the creation of educational multimedia systems. The performance support system runs on any Windows computer and for the evaluation the performance support system was running on a portable computer with a head-mounted display. The performance support system and the head-mounted display are each described in the following sections.
Performance support systems use customized software which integrates information, tools, and methodologies to help a user perform a specific task [2]. We built a performance support system to teach users how to do origami, the ancient Japanese art of paper-folding [5]. Origami was chosen because it is a hands-on, dynamic task which is unfamiliar to most people. The goal of our system was to allow users to learn to fold a simple jumping frog.
To be successful, a performance support system must organize information in a way that is obvious to someone who has not used the performance support system before [5]. We organized the training information into the following intuitive categories: a brief description of the training goal, the steps to follow to meet the goal, a tool that helps the user correct his or her work, and an on-line library of background information. Figure 1 shows the main menu of our origami performance support system.
Figure 1: Main Menu of Origami Performance Support System
In our origami performance support system, we tried to use media in a way that helped the user to understand and learn the information [4, 3]. For example, in the section that has step-by-step instructions, each step is illustrated with a simple static drawing and accompanying auditory instructions [see Figure 2]. To understand the fold described by the step, the user can examine the drawing for as long as desired and have the instructions replayed. However, since paper-folding is a dynamic task, we also provided an option that allows the user to see and hear a video of someone actually folding the paper [see Figure 3]. The head-mounted display allows the user to look at the drawings or videos, and listen to the auditory instructions, while folding the paper.
Figure 2: Sample static drawing from Origami Performance Support System
Figure 3: Sample video from Origami Performance Support System
A picture of the Kopin headset used for this evaluation is shown in Figure 4. The user enters and receives information from the computer using a microphone, an earphone, and a head-mounted visual display. The Kopin headset has a 640x480 monochrome visual display with an integrated microphone and earphone. Voice recognition software (Verbex Listen for Windows 95) serves as the primary way to control the computer. We are currently working with a Computing Devices International 75 MHz 486 wearable computer. However, for this study mobility was not an issue. Instead, we focused on understanding the utility of a hands-free performance support system to support a procedural task. Consequently, the headset was connected to a laptop during the tests to facilitate observer data collection and monitoring.
Figure 4: Kopin headset used for evaluation
Our wearable computer provides many advantages for helping a user to perform tasks. For example, the display allows the user to work while looking at text, drawings, and video. The earphone allows the user to hear explanatory audio narration while looking at information on the display or at the task environment. The microphone, voice recognition software, and our applications allow the user to control the computer via voice so the user's hands are free for other tasks. The computer allows the information designer to use the best medium (e.g., text, graphics, video, sound) to communicate information to the user. The combination of batteries, computer, wireless network, and head-mounted display allow the user to be mobile, getting information when and where the user needs it.
This initial study compared participant performance on three simple tasks while using one of the following two support systems.
The verbal instructions and static drawings on how to fold a jumping paper frog are identical in both systems. In order to determine the ease of using a wearable computer performance support system and also not to bias the results in favor of the performance support system, participants were not given any explanation or prior practice using the learning system they were assigned.
Twenty professionals and students from Georgia Tech participated in the study. All of the participants rated themselves as 'very inexperienced' or 'inexperienced' in origami on a background questionnaire. The participants were randomly assigned to one of the two support systems groups (computer or book) with 10 participants in each group.
The participants first completed a background questionnaire to determine their level of experience with origami. Then, using the support system to which they were assigned, the participants completed three tasks: (1) fold a paper jumping frog, (2) find the term "squash fold," and (3) find the meaning of the word "origami." After completing the three tasks, the participants rated their support system on a final questionnaire.
For each of the three tasks, both speed and accuracy were recorded by the experimenter. The experimenter recorded by hand the actions of the book users while an electronic log file was kept of each of the computer user's actions. The participants also provided subjective measures of the two support systems by rating their assigned system on a final questionnaire.
One-tailed t-tests were conducted on the speed data from all three tasks. On the first task, folding a paper frog, the book users were statistically significantly quicker than the computer users (book mean = 241.9 sec., computer mean = 384.7 sec, p = 0.05). Although we had hoped that the computer would be more comparable to the book, this result was not surprising for several reasons. First, it takes longer to go through the computer steps, particularly if videos are viewed. It also takes longer for the user to listen to the auditory instructions than it does for a participant to read the instructions. Second, the task was simple enough that the instructions were on only one page of the book. The book users were not required to turn pages to get to the next step, whereas the computer users had to say "next" for the next step. Finally, the book users could look ahead in the instructions, and in some cases did not even read the instructions. This look-ahead ability also helped them to figure out how to make a confusing fold because they could easily compare before and after fold illustrations.
There was not a statistically significant difference between the book and computer users' performance on either of the information location tasks. For the second task, locating the term "squash fold," the book users were slightly quicker (book mean = 31.6 sec., computer mean = 44.8 sec.), while for the third task, locating the meaning of "origami," the computer users were slightly faster than the book users (book mean = 21.7 sec., computer mean = 18.3 sec.). These were surprising results to us. We expected the computer users to be significantly faster, due to the speed of the computer and not having to turn pages. However, the book was small and participants were very familiar with the typical layout of a book (e.g., table of contents, indexes, and introductions). Also, many of the computer users did not use the "Find" function provided in the computer performance support system. Even though there was a "Find" button on every page of the performance support system, many participants either did not see it or did not think it would help in their current task. In many cases the computer participants fell back on their knowledge of books and would go though the pages of the Origami library. It was while paging through the library section that some users finally noticed the "Find" button. Perhaps the use of a "Find" function makes more sense in a library context than in the context of receiving task instructions.
The second objective measure was accuracy of completing the first task, folding a paper jumping frog. Inaccuracies were broken into two categories, those inaccuracies which would have no effect on the final product and those which would affect the final product. Along with inaccuracies, we also computed the ratio of fixed mistakes to total mistakes. The last category was the number of participants who made at least one mistake of either type while completing their frog. This category was intended to determine if there were just a few participants making most of the mistakes. It was expected that the computer users' performance would be better in all of these categories (i.e., have a lower number) than the book users' performance due to the ability to look at videos of each step. In all but one case we were correct as shown in Table 1.
However, only three participants even used the videos, so it seems unlikely that the ability to watch a dynamic motion presentation of a step was the reason why computer users made fewer mistakes than book users. Also, two of the people who looked at videos made mistakes. Finally, the computer users made more mistakes that affected their final outcome than the book users. It appears that more research is needed in this area.
Table 1: Results of accuracy measurements
Category |
Book |
Computer |
| No effect mistakes | 7 |
4 |
| Mistakes with an effect | 3 |
5 |
| Ratio Fixes to Mistakes | 10/4 = 2.5 |
9/5 = 1.8 |
| At least one mistake | 7 |
5 |
Table 2: Results of subjective ratings questionnaire
| Questions - Please rate... | Book |
Computer |
| how much you liked or disliked using the learning system. | 3.9 |
4.1 |
| the effectiveness of the learning system for helping you perform the paper-folding task. | 4.3 |
4.3 |
| the effectiveness of the learning system for helping you to perform the word search tasks. | 4.4 |
4.4 |
| the likelihood that you would want to have this kind of learning system if you had to learn to do paper-folding (origami). | 4.2 |
3.9 |
There were four, five-point (1=low, 5=high) rating questions on the final questionnaire. The average ratings for the book and the computer are almost identical as shown in Table 2.
These results were a little surprising since we thought that the participants would like the computer performance support system better than the book. We were surprised at the high ratings on the book. We thought that the book users might become frustrated and have no way to get around their frustrations since there were no videos to explicitly show how a fold is made. However, it appears as if the task was too simple to require the dynamic display capabilities of video.
From written comments it appears that the computer users would like the system on a desktop computer but did not like using the head-mounted display. This probably affected their rating of the entire system to some degree.
We also looked at the usage patterns of the participants; that is, the process that each participant used to complete the tasks. Usage patterns can provide insight into how users are thinking about the tasks and their expectations of the system that they are using. Using this information can be helpful in improving the interface of any computer system.
The book users followed predictable patterns of use. For the first task of folding the jumping frog, eight of the ten book users went to the table of contents first and then to the page with the instructions. The other two simply misread the table of contents and went to the wrong page. For the second task, finding the term "squash fold," seven of the ten book users started with the table of contents then went to the index of folds and then to the page with the term "squash fold" described. Two users went directly to the index of folds and to the page with "squash fold" described. The third task, finding the definition of "origami," had a lot of variability. Users either determined from the table of contents that the Introduction was the place to find this definition or they just started going through the book by flipping pages. However, everyone who flipped pages started at the beginning of the book possibly indicating that they thought that type of information should be near the beginning of the book. In all cases it was apparent that the book users expected the definition of "origami" to be in the introduction, because once there they had to read to the second paragraph to find the definition and everyone took the time to read it, even though there was a preface before the introduction which they did not read.
It was apparent from these usage patterns that people have very specific expectations of books. They know that tables of contents will help them find one kind of information while the index is useful for other types of information. They also have expectations about where different types of information should be found within a book and use that to guide their searches.
In many ways the computer performance support users were also predictable and appeared to be using their knowledge of books to some extent. Much of the variability of the computer users processes was probably due to their unfamiliarity with the capabilities and organization of the computer performance support system. For the first task, folding a paper jumping frog, none of the computer users went straight to the steps. Like the booker users, the computer users went to other sections first (e.g., seven went to "Give Assignment" and three went to "Origami Library"), looking for table of contents type information. When it was apparent that the there was only one set of instructions, the computer users would proceed to the "Provide Steps" section. While doing the steps the computer users took limited advantage of the enhanced directions. Only three computer users watched at least one video, while four of the computer users asked to have the instructions repeated. However, most of the computer users waited for the automatic auditory instructions to finish before attempting to start work on a particular step.
For the second task, finding the description of the term "squash fold," there was a tremendous amount of variability. We expected the computer users to use the "Find" button. However, only one computer user did this immediately. Eventually, seven of the ten computer users used the "Find" button, but they went many other places before selecting the "Find" button.
For the third task, finding the definition of "origami," there was much less variability. The computer users used one of two strategies: (1) four computer users used the "Find" button and (2) six computer users went through the Origami Library.
It took the computer users an average of 25.5 steps to do all three tasks, with a minimum of 17 steps and a maximum of 38 steps. The minimum number of steps that could be used was 13. This indicates that on the whole the computer users did not get lost in the system, even though no operational training was provided ahead of time.
The overall usage patterns were pretty similar for the book and computer. For the first task, both groups of users went to an information source to find out how to get to the specific task directions that they needed. For the second task, the book users were more efficient and used their prior knowledge of books to go directly to the index to find the term "squash fold." Unfortunately, the computer "Find" button was not perceived to be helpful in finding the term. However, after some searching through pages most of the users did use the "Find" button. For the third task, finding the definition of "origami" the book users and a majority of the computer users used a similar tactic. They decided where they thought the definition should be and went searching for that place. Even though the word "origami" was included in the "Find" button list, a majority of the computer users decided to find the word themselves in the reference information in the Origami Library section. Overall, the book users and computer users used similar tactics, although the computer users had more variability due to unfamiliarity with the system. The computer users might have been less variable if the performance system was more similar to a book or another form of information with which they are familiar.
This initial study highlighted some important issues with the wearable computer performance support system. These issues range from design of the human-computer interface to new experiments to be conducted. On the interface issues end, we once again found that the design of the interface is extremely important to the performance of a system. Despite prior formative evaluations of the interface design and the input of a human factors professional, we have still found ways to improve the interface. Some of these items might not be issues if the first-time user is given a walk-through or tutorial on using the performance support system. Starting with the most obvious, all user interactive options (i.e., buttons, menus, icons, etc.) must be apparent both visually and in purpose to the user. Many of the computer users never noticed that there were videos available and a few of the computer users never noticed the find function. Both of these buttons were placed at the bottom of the screen with the previous, next, and quit buttons, which all computer users accessed. It seems likely that given the time pressure to complete the tasks as "accurately and quickly" as possible, the computer users did not feel they had the time to explore functions the system might provide, and the tasks were simple enough not to force them to. However, in more complex systems it might make sense to give the system some 'intelligence' and have the system recommend less obvious functions if it detects that the user is struggling. For example, the system could suggest the user look at the video after a certain amount of time or several requests for repeated instructions.
Another improvement would have been to have 'before' and 'after' static drawings on a single screen. 'Before' and 'after' drawings illustrate what the user should end up with after completing a step. Videos perform a similar function, but require the user to watch the entire process and invest a larger amount of time. The book users seemed to take advantage of the fact that they knew what they should have when they finished the step they were on. The computer users did not have this luxury and could not check their accuracy until they went to the next screen. This cost them some time if they were incorrect and were forced to backtrack. We suggest that 'before' and 'after' drawings are good practice for all computer systems which aid in a physical task, provided the screen is sufficiently big to display it.
Time pressure is an important aspect of the task that is being supported and should be carefully considered in future designs. If the task to be supported is very time critical then it will be necessary to make the enhanced information more noticeable or automatically supply it when it is determined that the user is not making sufficient progress. We are thinking about addressing this issue by applying Carroll's minimalism concept [7] for teaching software skills to the teaching of physical skills.
A last issue on the interface design side is the metaphor which is used for the system. In this case, it was obvious that users fell back on their knowledge of paper books in order to interact with the computer-based system. It should be remembered that the participants in this study had no prior instruction in the use of the computer-based performance support system and needed to ground themselves in some way. Thus, it might be a good idea to take advantage of this prior knowledge of the user in the interface design if minimal operational training is a goal. For example, in our software, the "Find" button might be labeled "Index." Also the organization should be clearly indicated such as the Table of Contents does for a book. The book metaphor is an obvious one, and has been used extensively in computer-based instruction. However, it has not always been used to an advantage so more research is indicated.
As for future experiments, we have many ideas generated by this experiment. We would like to look at how providing training/learning opportunities in a support system can alter its use, and what this means on a system wide basis. In other words, if a performance support system also offers training to users what does this mean to job productivity overall? When will a short drop in productivity due to learning be offset by later increased productivity? In addition, we would like to look at the usage patterns when there is no time pressure to complete the task. It seems that time pressure (although realistic in many situations) can have a large impact on how someone uses a support/learning tool. A third issue we would like to investigate is user interface metaphors and designs. Different organizations of information and interface metaphors can make a tool more or less helpful and intuitive, and thus affect everything from user acceptance to user productivity. It also seems likely that metaphors that make sense for desktop computers may not have the same impact on wearable computers.
On a larger scale, we intend to study real world work tasks in live work settings. It seems probable that there will be issues which are critical to the long-term use of a wearable computer performance support system that are not important or obvious during short-term use. Additionally, we plan on examining the use of wearable computer performance support systems to support more complex tasks which are difficult to understand from static two-dimensional pictures. We will also look at the mobility issues from both the physical design of the hardware of the wearable computer and also from the stand point of software design and organization.
Wearable performance support systems would seem to be of most benefit to a user when the tasks to be performed are very complex or the environment in which the user must operate prohibits the use of standard materials such as operating manuals. This initial study did not include these mitigating factors. Consequently, the difference between using the two support systems was minimal. However, it is encouraging to note in such circumstances that the wearable system did not hinder performance of the user.
Funding for this research was provided by the state of Georgia as part of the Agriculture Technology Research Program.
[1] Gery, G. (ed.). Special issue on electronic performance support systems. Performance Improvement Quarterly 8, 1, 1995 (entire issue). Available at <http://www.cet.fsu.edu /sy2000/PIQ/PIQContents.html>.
[2] Gery, G. J. (1991). Electronic Performance Support Systems. Weingarten Publications, Boston, MA.
[3] Najjar, L. J. (1996). Multimedia information and learning. Journal of Educational Multimedia and Hypermedia, 5, 129-150.
[4] Najjar, L. J. (1995). A Review of the Fundamental Effects of Multimedia Information Presentation on Learning (GIT-GVU-95-20). Georgia Institute of Technology, Graphics, Visualization, and Usability Center, Atlanta, GA. Available at <http://www.cc.gatech.edu/gvu/research/techreports95. html>.
[5] Najjar, L. J., Ockerman, J. J., Thompson, J. C., & Treanor, C. J. (1996). Building a demonstration multimedia electronic performance support system. Educational Multimedia and Hypermedia, 1996, Association for the Advancement of Computing in Education, Charlottesville, VA. p. 794.
[6] Ockerman, J. J., Najjar, L. J., Thompson, J. C., Atkinson, F. D., & Treanor, C. J. (1996). FAST: A research paradigm for educational performance support systems. Educational Multimedia and Hypermedia, 1996, Association for the Advancement of Computing in Education, Charlottesville, VA. pp. 545-550.
[7] Carroll, J. M. (1990). The Nurnburg Funnel: Designing minimalist instruction for
practical computer skill. MIT Press, Cambridge, MA.