The primary objective of this task was to conduct simulation studies that would allow us to evaluate the warning interfaces using a driving simulator located at the HRL Laboratories, LLC (HRL), in Malibu California. As part of the task, a fully interactive, fixed base driving simulator was to be developed and equipped with a model of a collision avoidance system. After the completion of the simulator, human subjects would be tested and asked to evaluate various warning systems that they experienced during their test runs. Both objective and subjective data would be collected and analyzed, and a final report discussing the experimental results was to be written.
A secondary objective of the task was to select 2 side-zone and 2 forward-zone warning systems that would be used later during the Task 6.4 closed course testing. As part of task 6.3 work plan, various aspects of the closed course testing were to be designed and a vehicle test plan describing the closed course testing was to be prepared.
The research consisted of two separate and individual simulation studies. The first study dealt with a side zone warning system that was designed to assist drivers making lane change maneuvers in traffic. The warning system provided information to the driver as to whether or not another vehicle was located in their blind zone as they tried to safely negotiate a lane change. The second study consisted of a forward zone warning system that was designed to detect potential head on collisions while traversing through traffic. The warning systems provided information to the driver, warning of other vehicles that the driver could potentially collide with. During each phase of the study only a single warning system was used, and the subjects only needed to concentrate on the specific system being tested. Additionally, the scenarios were designed so that the focus was on the system being tested and the critical events that occurred were designed to specifically activate the intended system
To achieve our goals during this endeavor, a systematic approach had to be employed where the eventual success of each step was partially based on the previous steps taken. These steps included:
Since there were 2 phases to the simulation study, a test plan had to be devised for both. For the side zone tests, the warning systems were designed to inform a driver that there was a vehicle located within a certain zone next to their vehicle. Therefore, the simulation study needed the capability to have vehicles enter and exit the side zones while the test subject was trying to execute a lane change maneuver. The forward zone warning systems were designed to detect a threat that the driver could potentially run head on into. Thus, the simulation study had to be designed so that vehicles could blindly pull in front of the subject's vehicle so that the subject's reactionary responses could be measured. From these definitions numerous critical events that would put the subjects into the desired situations were defined and eventually used in the simulator (all events are shown in the Task 6.3 - Experimental Results document).
In Task 6.1, studies were performed to determine which combinations of warning system components would best convey the desired information to the driver. From this work, a final set of warnings was determined for both phases of the simulation testing. The necessary hardware was purchased and integrated into the simulator and all software that was required to control the warnings was written. In the end, there were a total of 8 warning systems that would be tested during each phase of the study. There are some common components such as a Head Up Display (HUD), auditory tone, and tactile feel that appear in multiple combinations. These were the best components and they have been matched in different combinations to see how well they work together. For the forward systems, an additional component was added that dealt with false alarms. Since we did not want to mention to the subjects that there would be false alarms, we dubbed these cases as using a sensitive sensor that would pick up more activity than the normal sensor would. The false alarms that were used were randomly generated false positive readings. They were issued during simulation runs based on a mean time and deviation from a standard distribution. The warning systems used were list in Table 3.17.
Table 3.17: Warning Systems Used in the Simulation Runs.
| Side Warning Systems: | Forward Warning Systems: |
| Head Up Display (HUD) | HUD + Audio + Normal (sensor) |
| Visual icons in side mirrors | HUD + Tactile feel + Normal |
| Visual icons in rear view mirror | HUD + Audio + Tactile feel + Normal |
| Visual icons in both mirrors | Audio + Tactile feel + Normal |
| HUD + Audio | HUD + Audio + Sensitive |
| Visual icons in side mirrors + Audio | HUD + Tactile feel + Sensitive |
| Visual icons in rear view mirror + Audio | HUD + Audio + Tactile feel + Sensitive |
| Visual icons in both mirrors + Audio | Audio + Tactile feel + Sensitive |
Each of these warnings was built into the simulator's architecture and could easily be called from a menu displayed on the main computer console.
In the initial stages of development certain criteria needed to be met in order for the simulator to elicit the driving responses desired and to immerse the test subjects into as real a driving environment as possible. The following components were deemed necessary in order for the simulator to adequately mimic the desired driving tasks:
Based on these general requirements, the driving simulator was built and is comprised of several different independent components acting in unison. A complete simulator configuration is shown in Figure 3.46, and includes a complex arrangement including multiple computers, image generators, driving buck and software. All of these components as well as the image database were employed to generate the driving environment.
Figure 3.46: Final Driving Simulator Configuration
To provide for interactive controls and appropriate surroundings a complete driving buck was designed and built. This included steering wheel, throttle and brake pedals, and turn signal indicators that the subjects used to enter their control inputs. These inputs were then sent to a high fidelity vehicle dynamics model so that the vehicle behaves consistently with the subject's inputs. Additionally, the seats and mirrors were fully adjustable allowing the subject to configure the buck for optimum comfort and viewing.
To display the graphics in real time and coordinate the entire simulator's activities, 2 Silicon Graphics Incorporated (SGI) computers were used that communicated with each other using an Ethernet connection. The nature of the side zone study dictated that the subject be able to view as much of the scene as possible. Therefore, 6 separate views of the database were displayed during each simulation frame. Three of these views were used to present the forward scene that is viewed outside the windshield. This provided a 165-degree field of view and was displayed using high-resolution overhead projectors projecting onto a toroidal screen. The remaining 3 views were used to display images that could be viewed in the vehicle's mirrors. The images were displayed using monitors positioned so that the images were reflected using the driving buck's mirrors.
Since the basic simulator was built on a fixed platform, the subjects received no motion cues during operation. Other cues were provided to heighten the driving experience. The first was roadway sound effects. These effects included engine rpm noise, providing the subjects with some audio feedback based on speed and acceleration. In addition, a torque motor was added to the steering system so that as the subject steered the vehicle the steering wheel would respond and the subject would receive propioceptive feedback through their arms. This provided an additional cue based on how the vehicle was cornering.
Before any driving could be done, a few more things were needed, a database to drive through, a software package to display the visual scene, and software to control all of the interactive portions of the simulator. The database was designed to provide a complex and multi-facetted scene to drive through. The final database design contained 4 distinctly different roadway sections that provided urban, suburban, rural, and freeway driving. This allowed the subjects to experience all types of driving conditions. The database also contained a wide variety of scenery, traffic control devices and roadways. Scenery included buildings in the urban section, houses and trees in the suburban section, farmhouses and trees in the rural section, and bridges in the freeway section. Traffic control devices such as stop signs and signal lights, and roadway signs can be found throughout the database. The roadways that were built into the database include 2, 4, and 6 lane sections.
After the final database configuration was determined an independent company that specialized in 3D modeling and database development was hired to build the final database. Unfortunately, they took much longer than expected to deliver the final model and multiple iterations had to be created before the final database was accepted. This created some delays in the testing schedule because the final scenario development could not be completed until the scenarios could be tested on the final database.
In order to display the database on the screen, a software package was needed. After researching the various options for the computer system that would be running the software, a decision was made to use the EasyScene software package. This decision was made in part because EasyScene was supposed to be compatible with another software package, ScenarioBuilder that could control the interactive elements of the simulator (signal lights and vehicles). Unfortunately, there were several problems with this configuration and once again the simulation test schedule slipped.
The most serious simulator problems involved a latency between the host vehicle and other vehicles in the roadway display. This latency was caused by a communications problem between the ScenarioBuilder and EasyScene software packages. The manifestation of this problem was that the interactive vehicles that were being displayed would appear jittery (not smooth from one frame to the next) and thus would be visually unpleasant and possibly affect the data being collected. To correct this problem HRL personnel obtained source code from the vendors and made modifications to offset the effects. However, although these modifications did greatly reduce the effects, they did not eliminate them completely. Other problems also occurred such as non scripted vehicles appearing in the display scene, scripted events not occurring on cue, and some data collection glitches. All of these were continuously handled as the need arose.
In addition to the problems with the simulator and its software, there were a couple of general problems that had to be overcome in order for the study to be conducted. The first was obtaining the subjects that would be tested, second was dealing with a large percentage of dropouts due to simulation sickness problems, and third deals with data collection and analysis.
The initial intention was to obtain as many subjects as possible from with the HRL facility. Unfortunately, because of scheduling conflicts, the response from HRL personnel was very low. Next, college students were recruited from local colleges, but this too did not result in as many subjects as was need to complete both studies. Finally, ads were placed in the newspaper in hopes of drawing additional subjects. We were able to obtain the number of subjects desired, but had great difficulty getting them to show up on schedule if at all.
When subjects did show up, they were trained and then allowed to drive the simulator. While driving the simulator many subjects became ill due to the simulator's configuration and lack of motion. In some cases the subjects were able to continue and eventually finish an entire set of runs, and in other cases, they had to be dropped from the program. This resulted in lost simulator time and caused large delays in the test schedule. To help accelerate the schedule, subjects that participated in the side zone study were also asked to participate in the forward zone study. A total of 12 subjects completed both studies.
The final major obstacle that had to be overcome dealt with the data collection and analysis. During the simulation runs, approximately 10 to 20 percent of the critical events failed to be activated. Therefore, when all of the data was collected, there were holes caused by the missing data. This meant that additional analysis tricks had to be employed to try and make sense out of the objective data that was collected. In addition, the data files that were created contained a data stamp from every frame of the simulation run, thus creating data files that were around 4 MB (compressed, uncompressed about 50 MB) in size. This required a great deal of time and effort in order to extract the critical data and analyze it. To help speed the data process, software was written that would go into the data files and attempt to extract the critical information that we were looking for. Unfortunately, due to the unpredictability of human drivers, the data analysis could not be easily automated and therefore a good portion of it had to be analyzed by hand.
Even with the problems that were being experienced, a decision was made to run some preliminary tests and see if using the simulator was still a reasonable undertaking. During these tests, subjects were enthusiastic about the possibility of these systems being available in the future and did not seem to have any major concerns about the simulator's problems. Therefore, the full simulation testing program was launched. Details about the test matrices, scenarios, subject training and paperwork can all be found in the Task 6.3 Experimental Results document.
The actual side simulation tests began in February and ended in June of 1997. A total of 36 subjects (20 male and 16 female) were tested. Their ages ranged from 18 to 58 years and there were representatives from a wide range of age groups. Since gender and age effects were not being considered, the use of more male than female subjects and the predominance of middle aged subjects was not a concern.
Subjects were required to complete 9 separate drives through the roadway database, with each run consisting of a different scenario and warning condition. During each run data about the vehicle's position and motion and the subject's reactions was collected. Additionally a questionnaire requesting subject ratings and comments was also completed at the end of each individual run. All of this data was then analyzed and some conclusions were drawn.
Using the objective data that was collected directly by the simulator, a multivariate analysis was performed to see if the warning conditions that the subjects were experiencing had any significant effect on the way they reacted. For the multivariate analysis the independent variables were the warning condition and the critical event, subjects were treated as a repeated measure. The initial dependent variables that we were planning to investigate included, Time To Collision (TTC), Time To Avoidance (TTA), reaction times and collisions. However, due to problems with the simulator and the interactive vehicles it was controlling, none of these variables were used. TTC and TTA (both based on range and range rate) were unreliable because of the jittery motion of the interactive vehicles as they approached the subject's vehicle. From one frame to the next, the TTC and TTA would go from minuscule to infinity, because the vehicle would be 2 meters away, then 5 meters away, then 2 meters away. Reaction times were also desired but for the side zone study they were inconclusive due to the nature of the events. Only a couple of events required braking, and subjects were not forced to steer so they could take as long as necessary to change lanes, thus making reaction times meaningless. The number of collisions were inconclusive due to non-scripted vehicles appearing in the driving scene and subjects occasionally colliding with them. However less than a couple of dozen crashes did occur during testing which was low and usually the crash involved a vehicle in front of the subject, not in the blind zone.
Since none of the initial dependent variables provided useful data 2 new ones were tried. The first was the minimum range at lane change, and the second was simply the minimum range. The minimum range at lane change was a measure of how far apart the subject's vehicle and the blind zone vehicle were when the subject fully committed to the lane change. This is described in more detail in the Task 6.3 report. The minimum range was simply the minimum range between the 2 vehicles while the warning alert was active.
The multivariate analysis showed that the critical event that was used had a highly significant effect on the data, and failed to show any statistically reliable effects due to warning condition. This can be easily seen in Figure 3.47, where we see the mean minimum range to lane change values as a function of critical event and warning condition. The mean values follow a definite trend based on the critical event, but for each critical event, the warning conditions are scattered sporadically. This is probably due in large part to the drivers' responses to the simulation scenarios, which was reasonably evident while observing the experiments. The side zone scenarios did not contain explicit hazards that the subjects must respond to. Instead, the subjects had the option of observing situations in their mirrors, and not maneuvering until adjacent traffic was clear. Therefore, these results do not give a clear indication of a human factors distinction between the alarm conditions. A complete data analysis is contained in the Task 6.3 report.
Even though the objective analysis did not provide significant results, the subject data that was collected did. The subjective data was comprised of 2 parts. First the subjects responded to general questions about each individual warning system (the complete questionnaire can be found in the Task 6.3 report). In general these questions asked the subjects to rate the various aspects of the warning system on a scale from 0 to 6. The second part of the subjective data was obtained at the completion of each subject's participation in the study. At this point they were asked to rank the various systems from 1 (best) to 8 (worst) and to provide any comments on the individual systems.
The subjective data illustrated that there was a definite tradeoff between system effectiveness and system annoyance. In general the systems that were rated the most effective also were rated the most annoying, and vice versa. When the subjects were asked to rank the systems from best to worst, the systems that tended to be the visually least annoying were rated best. These results are shown in Figure 3.48 where the average results from several different questions are displayed. In general, these results show that the side systems were effective at getting the subject's attention, especially with the auditory warnings (warning conditions 5-8). But at the same time, the subject responses show that the systems could be annoying and that the systems with the auditory warnings were the most annoying. It is interesting to note that in general the less effective a system seems to be the less annoying it is.
At the completion of the study, subjects were asked to rank the various systems from best to worst. Using these rankings and a weighting scheme that assigned a value to each individual rating, a total composite rating was determined. These final ratings showed that all 4 conditions with the auditory warnings were rated best and the system using icons in the side mirrors combined with the auditory was rated best. With warning systems where no auditory tone was used, the system with icons in the side mirrors was rated best.
Some general comments from the subjects explained that the side view mirrors were the best visual warning because they did not appear in the driver's sight lines and were therefore less distracting than the icons in the rear view mirrors or the HUD. Unlike the HUD, they also provided a clear indication of which side of the vehicle that the threat was located on. The auditory warning was definitely the most effective component at getting the subject's attention, but was also the most annoying component and had the potential to become extremely annoying on long drives and in heavy traffic.
Figure 3.47: Mean Minimum Range to Lane Change.
Figure 3.48: Mean Subject Responses to Post Run Questions
Although the objective data collection and analysis did not yield fruitful results, the subjective data based on post run questionnaires provided data that was very useful. The following conclusion were drawn based on the data that was collected:
A secondary objective for this study was to choose the 2 warning systems that would be used in the Task 6.4 Closed Course Testing portion of the contract. Based on the data that was collected and analyzed, the following 2 systems were chosen:
The condition with icons in the side mirrors with auditory alert was an easy choice because it finished first overall in the subject ratings and there was a distinct margin between it and the next 3 systems. The second choice of icons in the side mirrors was a little more difficult. The top 4 ranked systems all included an auditory alarm, however it was only triggered when the turn indicator was used and therefore may have been less annoying. In the field test vehicle, the auditory alarm will always be present whether the turn indicator is used or not. Since there was concern over whether having the system on all the time would be too distracting, it was decided that one system would have an auditory alarm and one would not. Since the side mirror system was ranked best with and without the auditory alarm it was the logical choice.
The forward simulation tests began in May and ended in August of 1997. A total of 35 subjects (18 male and 17 female) were tested. Their ages ranged from 17 to 55 years and there were representatives from a wide range of age groups. Since gender and age effects were not being considered, the use of more male than female subjects and the predominance of middle aged subjects was not a concern.
Subjects were required to complete 9 separate drives through the roadway database, with each run consisting of a different scenario and warning condition. During each run, data about the vehicle's position and motion and the subject's reactions was collected. Additionally a questionnaire requesting subject ratings and comments was also completed at the end of each individual run. All of this data was then analyzed and some conclusions were drawn.
The forward testing had a couple of additional problems that were not experienced in the side zone study. First, for some reason that could not be readily explained (except for the previously mentioned simulator problems), close to 25 percent of the data either did not get recorded or the events did not occur. This was a much larger percentage than was seen in the side zone study. Second, of the events that did occur, approximately 25 percent of the time, the warnings systems did not issue any alerts. Once again, like the side simulation study, these 2 problems created holes within the database and required additional work to be performed in order to try and make sense of the data collected.
Using the objective data that was collected directly by the simulator, a multivariate analysis was performed to see if the warning conditions that the subjects were experiencing had any significant effect on the way they reacted. For the multivariate analysis the independent variables were the warning condition and the critical event, subjects were grouped into bins based on similar conditions and thus treated as a repeated measure. The independent variables that were investigated included minimum Time To Collision (TTC based on range and range rate), minimum range, peak steering response, and throttle, brake and steer reaction times. The number of collisions were also desired but like with the side zone study, they were inconclusive due to non scripted vehicles appearing in the driving scene and subjects occasionally colliding with them.
The multivariate analysis showed that the critical event that was used had a highly significant effect on the data, and failed to show any statistically reliable effects due to warning condition. This can be seen in Figure 3.49, where we see the mean throttle time responses for each critical event and warning condition. For each critical event there seems to be a trend in the direction that the throttle times go, but the warning conditions seem to be scattered fairly randomly and were therefore not statistically significant. The unreliability of the data was attributed to the subjects being able to anticipate the coming events and therefore react before any warning could be issued. This also caused the warnings to not be issued because in their anticipation, subjects were able to completely avoid some critical conditions. This observation is evident by looking at the throttle responses shown in Figure 3.49. A negative throttle time indicates that the subject took their foot off of the throttle before a warning was issued. For the majority of the cases, the throttle times were negative meaning that subjects were no longer accelerating and most likely anticipating a potential problem. The complete data analysis with plots and tables is contained in the Task 6.3 report.
As was the case with the side zone study, the objective analysis did not provide significant results, however the subjective data that was collected did. Once again, the subjective data was comprised of 2 parts. First the subjects responded to general questions about each individual warning system (the complete questionnaire can be found in the Task 6.3 report). In general, these questions asked the subjects to rate the various aspects of the warning system on a scale from 0 to 6. The second part of the subjective data was obtained at the completion of each subject's participation in the study. At this point they were asked to rank the various systems from 1 (best) to 8 (worst) and to provide any comments on the individual systems.
Once again, as was the case with the side zone study, the subjective data illustrated that there was a definite tradeoff between system effectiveness and system annoyance. In general, the systems that were rated the most effective also were rated the most annoying, and vice versa. This is illustrated in Figure 3.50 where mean subject responses to a question about the systems effectiveness at getting the subjects attention and how annoying the system was, are shown. Looking at the 2 plots, you can definitely see that the least effective systems were also the least annoying, and some of the most effective systems were the most annoying. When subjects were again asked to rate the systems from best to worst, 2 of the top 3 choices were the least annoying even though their effectiveness at getting the drivers attention may not have been rated best. In addition, the systems that included false alarms were rated very poorly by the subjects.
In general, comments that were submitted by the subjects, the HUD was the component that was liked best by the subjects. This was because the warning was provided without the subjects having to take their eyes off the road. However, the HUD icon received mixed reviews. Subjects stated that the auditory alert was the best at getting their attention and was also the most annoying part of the system. The tactile feel was liked by a majority of the subjects and when combined with the HUD display it was considered very effective. As for the false alarms, the subjects thought they were confusing and stressful and reduced the overall effectiveness of the system.
Figure 3.49: Mean throttle response time
Figure 3.50: Mean Subject Responses for Effectiveness and Annoyance
The primary objective was to investigate how subjects responded to various warning systems and to generate feedback from them about how the warning systems functioned. A statistical analysis of the objective data was performed and overall the warning condition did not reliably influence the data. From all of the data that was collected, the following conclusion were drawn:
A secondary objective for this study was to choose the 2 warning systems that would be used in the Task 6.4 Closed Course Testing portion of the Project. Based on the data that was collected and analyzed, the following 2 systems were chosen:
The condition with the HUD and tactile feel was chosen because it was rated best in the overall ranking that the subjects submitted. In addition, it was rated as the least annoying during post run questioning, and it has been shown a couple of times that subjects seem to prefer less annoying systems even if they lose some effectiveness.
As for choosing the HUD with tactile feel and auditory alert, this was a little more complex. It was apparent from the data that the second system would be chosen from either the HUD with tactile feel and auditory, or the HUD with auditory. They tended to be rated fairly evenly by the subjects and finished ahead of the system without the HUD. In the overall rankings, effectiveness ratings and annoyance ratings, they finished almost identical. However, in the distraction category the HUD with tactile feel and auditory was definitely rated better and therefore was chosen. In addition, since the number 1 system included the HUD and tactile, this allows the same system to be tested but with the auditory component added in.
While the scenarios were being designed and tested, a concurrent effort was also conducted to design the events and scenarios that would be used in the Task 6.4 Closed Course Testing portion of the project. During this effort, the test track was chosen, and the critical events that would occur on the test track were designed. In addition, the test matrices and all paperwork that the test subjects would need to complete were also defined. A subject review board approved the final test configurations, thus allowing human subjects to be used during testing. From this effort the Task 6.3 - Vehicle Test Plan report was written and delivered as required. The details about the closed course testing setup can be found in this report.
During the performance of Task 6.3, many different problems and issues were encountered, and handled. The following major accomplishments and program objectives were realized:
All of the desired objectives of this task were met. Unfortunately, because of the various simulator problems and the scenarios not being constrained enough, the objective data that was collected did not provide statistically significant results. This is a phenomenon that is not uncommon when dealing with human subjects who are very adept at adapting to their given situation. In this regard we can say that based on the initial expectations, less was accomplished than was expected. However, the subjective results that were obtained from the subjects provided incredible insight into the system usage and should be considered a major benefit and accomplishment for this task.
In hindsight, the experimental results show that several things could be improved upon in order to obtain more consistent and reliable data in future studies. Improved communication between software packages would have provided a more reliable simulator and would have allowed all of the desired data to be collected. Also, an improved secondary task that forced the subjects to take their eyes off of the roadway scene for an extended period of time would have added additional realism and would have put the subjects into the types of dangerous situations that the warning systems were designed to eliminate. Both of these should be considered in future simulation tests.
The major emphasize of this study was to see how drivers reacted to a warning that they were given during dangerous driving situations. The data collection and reduction focused primarily on how the subject reacted and their overall opinions of the system's effectiveness and annoyance. Now that we have some idea in these areas, additional work needs to be performed to see if subjects can "blindly" recognize the warnings that they are given. In addition, work needs to be performed in order to determine the warning cues that are best at conveying the necessary information to the driver when the driver has had no training and knows nothing about the system.