The focus of this task was to provide vital information for the development of driver-vehicle interfaces which will evoke appropriate and timely responses from the drivers in order to avoid collisions. The approach was to develop software models of generic forward-looking and side looking sensors that were designed to be used in a real time, interactive driving simulator.
The primary objective of the task was to develop software modules that accurately modeled radar sensors developed by GMR and Delphi-DE. A variety of software modules representing generic MMW-based Forward-looking Radar (FLR) and side-Near Object Detection System (NODS) sensors were to be developed. Included in this task were several issues that must be met in order to achieve the primary objective. Developing the radar models themselves was not enough. The models also had to be integrated into the Multiple Target Tracker (MTT) and Collision Avoidance Processor (CAP) architecture and thoroughly tested to insure that they functioned as desired. In addition, since the primary objective of the simulator was to perform a human factors study to determine how drivers would react to the systems, time was spent adapting the sensor models to the database and developing driving scenarios to insure that the radar modules would properly detect objects and the simulator would activate the warning systems.
Due to the closely related nature of the sensor simulation and the driving simulator, there was a great deal of cooperation between Task 6.2 and Task 6.3. Consequently, this effort was a collaborative effort between the ACAS consortium members of STI, DE/AED, and HRL. By working closely on this complex project, historically thorny integration issues were kept to a minimum. Changes in interface requirements did not cause major delays and technical assistance was freely exchanged between members. At the conclusion of Task 6.2, the following goals were anticipated:
The STI sensor simulation was developed specifically for use in the HRL Driving Simulator (HDS). The purpose of the overall driving simulator was to investigate the relative efficacy of various types of alert systems intended to notify a driver of a potential collision. The sensor model requirements included a generic detection model, real-time run capability interfacing with multiple real-time processes, and conversion of perfect information about simulated objects into quantized and sometimes misinterpreted information about radar targets. This work was attempted in order to create models that would provide the type and quality of information a real radar sensor would. The STI sensor model has met and exceeded these requirements. While there has been considerable emphasis on simplification to provide speed for the real-time simulation, the underlying structure of the model is sound enough that, with minor modifications, this package could also be used as part of a radar evaluation simulation instead of just a human factors simulation.
A functional block diagram of the simulation processes is shown in Figure 3.44. The top row of blocks compose the HDS, the center row is the STI sensor simulation process, and the bottom 2 rows represent the DE/AED MTT and CAP programs.
Figure 3.44: Functional Driving Simulator Block Diagram
In the HDS blocks, the driver inputs are transduced by the buck and monitored by a number of Silicon Graphics workstations. These computers also run the vehicle model, update the roadway environment based in part on a scripted scenario, and provide feedback to the driver. The driver feedback consists of the roadway display scene, force-feel on the steering wheel, and auditory cues such as engine noise and vehicle warning sounds. The HDS processes fill a region of shared memory with information about the roadway environment and host vehicle position. This memory is read by the STI sensor process that is running with the MTT on a separate computer. The sensor converts the roadway objects into a list of radar targets and writes this list to shared memory. The MTT reads this list and converts the targets into a list of tracks and then the CAP examines these tracks and determines which of them are "in-path" and whether they represent a threat to the host vehicle.
When the simulator is initialized, all of the objects that the sensor can detect are created and stored in memory for later use. Next the simulator sets up an interrupt driven timing routine that will wake the sensor model up at the correct sample frequency. This way the sensor model acts at the same rate as a real world sensor and will not cause problems with the overall simulation timing. When activated, a function that determines object detection is then executed. Figure 3.45 shows the process that the function goes through each time it is called.
Figure 3.45: Sensor Processing Steps
The first step in the process is to copy the shared memory data structure written by the HDS to a local memory pool that the sensor uses. After the memory is copied, a multi-stage visibility check is performed. Objects that do not fall within the sensor's field of view are ignored. Radar return signals are then calculated for objects that meet the visibility requirements. If an object's return signal is greater than a threshold level, that object is then converted to "sensor space." This process involves determining which of the sensor's beams strike the object. Those that do, create a range bin to store the object's range, range rate, and signal level. If a range bin already contains information for another object at the same range, the information is added together, weighted by signal strength. Once all the visible, detected objects have been converted to detections in sensor space, the process of merging begins. The goal here is to examine the individual detections based on proximity and differential velocity and combine them into targets. These targets are then listed. If more than the maximum number of targets have been identified by the sensor, it will perform a very rough threat assessment and ignore the least likely threats. At this point the target information is written to shared memory for use by the MTT process. Details about how the beams were modeled and how the objects were detected, can be found in the first Annual Report.
At the beginning of the project, one of the objectives was to create the model so that it could easily simulate different sensors. In order to accomplish this feat, the sensor model used a generic process for generating a beam pattern and for obtaining the signal return from the object being detected. These routines were generic and took the form of specific sensors when various parameters that were specific to a desired sensor were defined. These parameters were specified in a parameter file that was read during the simulation's initialization process. Parameters included range, azimuth angle, number of beams, beam width, etc. The above description details the final process that was used to model the sensors within the HDS. However, many problems and technical challenges had to be overcome before this final fully functional architecture was developed.
During the early stages of the ACAS Program, the challenge was to determine how the suite of algorithms that comprised the MTT and CAP would be implemented. DE/AED already had proprietary routines that were developed for use in their FCW system and implemented on various demonstration vehicles and these routines had previously been evaluated and validated. The original program focus was to have STI create the entire sensor package including the FLR, side NODS, MTT and CAP. This seemed to be a redundant task that would only cause additional problems to occur. After several meetings between STI, DE/AED and HRL personnel, it was decided that rather than have STI try to duplicate the DE/AED proprietary system, DE/AED would modify their existing software and work with STI and HRL personnel so that it would run in the HDS. Although the effort required of DE/AED was considerable, this solution worked well and has assisted in the validation of the real world system. By taking this approach several benefits were derived from this division of software module activities:
Some of the technical challenges encountered were very subtle. Once the simulation was fully integrated, a slight timing irregularity was discovered. The overall simulation consists of many separate programs running at the same time. While theoretically these processes all update in a synchronized way, this is impossible to achieve on a real system. The sensor, the vehicle dynamics model, and the scenario generator are all run on different machines and at different update rates. If any one of these programs iterates for longer or shorter than expected, the data processed by the sensor would be incorrect. The symptom of this problem was erratic velocities reported by the FLR sensor to the MTT process. These irregularities, caused by the fact that the sensor calculates object speed as differential position, resulted in the MTT program behaving unexpectedly. Two changes were made to solve this problem. First, time-stamps were included with the data reported by the driver model and scenario generator. Second, object speed was calculated using a second-order, digital low pass filter.
An important area of investigation for the ACAS Program focuses on how the sensor warning system's false alarm rate will influence drivers. Knowing the point at which drivers begin to ignore the system will set guidelines for minimum system performance. It was, therefore, necessary for the sensor simulation to reliably create false alarms. It was originally thought that the most natural way to create false alarms would be to raise the sensor simulation noise coefficient and create a false sensor detection. Further analysis revealed that increasing the sensor noise level will produce more detections from the sensor. However, these false detections did not necessarily create false alarm warnings for the driver because the DE/AED MTT and collision warning algorithm suite processes were sufficiently robust so as not to propagate spurious detections identified by the sensor. Therefore, it was decided that the best way to generate false alarms was to simply have the HDS system randomly activate collision warnings at an average specified rate based on a normal distribution.
In order for the simulation to function correctly, the sensor model would have to be able to detect objects in the roadway scene. In order to do this the HDS passes the sensor simulation a list of objects that are capable of being detected. The basic shape of these objects is a rectangular box. It was originally anticipated that once the new database became available, there would be many types of sensible objects including extensive objects such as guardrails. However, a rectangular box shape could not be used to describe something as complex as a curving guardrail. The problem with specifying detectable objects was that they had to be pulled out of the static database and converted into dynamic objects in order to be detected. This approach proved to be extremely time consuming and did not deliver the intended results. Therefore the only objects that were detectable during the simulation runs were interactive vehicles with their basic shapes remaining a rectangular box.
Most of the discussion up to this point has focused primarily on the FLR system. In addition to the FLR a side-looking sensor interface was also developed. The intent of the side-NODS sensor was to alert a driver to the presence of target vehicles traveling in the driver's blind zones. Unlike the FLR sensor package, the side-NODS system had a much simpler architecture. The side-NODS system provides very simple MTT and CAP features. This sensor merely provides a true or false value indicating the presence or absence of a blind spot threat. From a radar perspective, the side-looking sensor is modeled closely on the forward-looking sensor (i.e. objects are turned into range bin detections that are then grouped into targets). When the sensor created a target list an additional stage of processing was used to remove all stopped and oncoming objects. If any targets remained in the list, the side-sensor set a flag in shared memory that instructed the HDS to present a warning alert to the driver.
The final issue that had to be resolved before the simulation testing under Task 6.3 could occur was to make sure that the simulation's critical events capable of being detected and thus a warning would be presented to the driver. These critical events would have to be designed so that the sensor would be able to detect them and that the MTT would register them as threats, thus triggering the warning. Furthermore, since the critical situations for the side zone and forward zone warnings were completely different, different scenarios were needed for both.
The critical events were designed to produce specific driver responses and included:
Side Zone:
Forward Zone:
A complete description of each critical event can be found in the Task 6.3 " Experimental Results" report. Between 4 and 6 of these critical events were combined with a handful of benign events and other interactive events (traffic signals, stop signs and normal traffic) to form a complete scenario. For both the side and forward zone studies a total of 12 scenarios were created. The experimental design was setup so that only 9 scenarios would be used but because of possible problems with the simulator, 3 additional scenarios were created in case subjects had to be re-run.
During the process of creating the critical events, each individual critical event was tested by itself using on "off-line" procedure provided by the scenario development software. When it looked like the event was performing as planned, the next step was to drive "on-line" through the event using the simulator. Finally when a scenario was completed, a final drive through the entire scenario was performed. During each step of this process iterations on the event's performance were made, until finally they performed as desired. The amount of time that was required to create these scenarios was greatly under estimated (creation and checkout took more than 8 months) and this caused a huge delay in the simulation testing schedule. Before the actual testing could be performed, some preliminary runs were conducted with test subjects and final adjustments were made to the scenarios based on subject comments.
At the completion of the scenario development, the sensor parameters were set and all of the critical events did a good job of producing a warning. However because of the scenario software architecture, the critical events had to be designed around a relatively constant speed and were based on the speed limit in the area of the database where the event was occurring. Subjects were instructed to try and drive the speed limit, however, if a subject travels very fast or slow through the desired area, the events may not work as well as they were designed too. This is an unfortunate occurrence that could not be avoided.
During this task, many different problems and issues were encountered, and handled. The generic sensor models that were created are general enough that they can be used in other applications besides this one. Therefore, this phase of the contract can be considered successful and we were able to get more out of it then initially expected. The following major accomplishments and program objectives were realized:
The sensor simulation was created to perform human-factors studies using a real time, driver-in-the-loop simulator. Therefore, some assumptions have been made to simplify and speed processing. Even so, the changes required to turn this simulated sensor model into a more accurate, batch mode simulator would be minor. As future revisions are made to the MTT and CAP, this sensor model could be used to simulate object detections before the systems are actually implemented on a vehicle.
Incorporating the STI sensor package into an existing simulation should be quite simple from an interface point of view. Since it is designed to run as a standalone process, reading data from shared memory, it requires virtually no support during operation. Instead of using shared memory, the sensor could read data from a global object pool structure that the simulation had written. This data could even be created ahead of time and written to disk. The output of the sensor simulation is currently an array of targets suitable for processing by the Delphi-DE/AED MTT program. The actual structure used to contain this information is proprietary to Delphi-DE, but there should be no problem creating a suitably different way to pass the sensor output targets from the sensor to the simulation. Other changes that would add accuracy to the simulation include integration of a beam lobe-shape function across the sensed object silhouette, and more complete coordinate transform that allows for target objects to pitch and roll.