Field Outcomes

Survey data collection by the Federal government requires prior approval by the Office of Management and Budget (OMB). Before submitting the formal request for data collection to OMB, NHTSA published a Notice in the Federal Register soliciting comments on the information collection. The Notice appeared in the Federal Register, 63:50, pages 12858-12859, March 16, 1998. The closing date for comments was May 15, 1998. No comments were received in response to the Notice. NHTSA then submitted the request for data collection to OMB June 29, 1998. OMB approved the information collection September 9, 1998, assigning it the OMB number 2127-0596 with and expiration date of December 31, 1999.

The field interviewing for the study commenced on November 5, 1998, following training of the field interviewers, and was completed on January 12, 1999. This is approximately the same time period in which the 1996 Occupant Protection Survey (November 4, 1996 to January 5, 1997) and the 1994 Occupant Protection Survey (October 5, 1994-December 11, 1994) were conducted. Status of cases as of the end of the field period are reported using the categories defined below.

For survey Version 1 - Seat Belt Usage Issues, a total of 30,620 randomly selected telephone numbers were sampled within a geographically stratified national sampling frame for both sample components (the cross-section of youth and adults age 16 and older and the oversample of persons age 16-39):

• 59% of the numbers were not active residential phone numbers, including 32% Pre-screened NIS/DIS; 10% NIS/DIS/Change#/Wrong# after the number was dialed; 7% SSI Pre-screened Business Numbers; and 3% computer or fax tones;

• 7% of the numbers were no answers (despite repeated attempts) and 2% were answering machines; and

• 2% were households in which the designated respondent was not interviewable (not available; away for an extended period, incapacitated, or deaf) and an additional 1% were non-interviewable due to language barriers (non-Spanish).

At the close of the field period, only 521 cases (2%) were in callback status.

The participation rate represents one of the most critical measures of potential sample bias because it indicates the degree of self-selection by potential respondents into or out of the survey. The participation rate is calculated as the number of completed interviews (including respondents who screen out as ineligible) divided by the combined total number of completed interviews, terminated interviews, and refusals to interview. (The inclusion of screen outs in the numerator and denominator is mathematically equivalent to discounting the refusals by the estimated rate of non-eligibility among refusals.) The participation rate for Version 1 is based on the following elements:

• 4,094 completed interviews;

• 2,436 cases in which someone in the household completed the household screen, but no one in the household was found to be eligible for the full interview (2362 were age-related screen-outs among the age 16-39 oversample); and

• 1,627 refusals to be interviewed (including 1220 second refusals) and 43 terminated interviews.

Based on the standard calculations of participation rate, as defined by the Council of Applied Statistical Research Organizations (CASRO), the participation rate for Version 1 was 79.6%. This formula treats the numerator as all respondents who participate by completing required survey questions, while the denominator includes those who complete required questions, those who begin but terminate before completing all required questions, and those who refuse entirely.

The Final Summary Disposition of the Version 1 sample is given in Table 3. The table includes breakouts for each survey component (national youth and adult cross-section and the age 16-39 oversample). The average interview length for Version 1 was 22.7 minutes.

For survey Version 2 - Child Safety Seat Issues, a total of 29,592 randomly selected telephone numbers were sampled within a geographically stratified national sampling frame for both sample components (the cross-section of youth and adults age 16 and older and the oversample of persons age 16-39):

• 59% of the numbers were not active residential phone numbers, including 31% Pre-screened NIS/DIS; 10% NIS/DIS/Change#/Wrong# after the number was dialed; 8% SSI Pre-screened Business Numbers; and 3% computer or fax tones;

• 6% of the numbers were no answers (despite repeated attempts) and 2% were answering machines; and

• 3% were households in which the designated respondent was not interviewable (not available; away for an extended period, incapacitated, or deaf) and an additional 1% were non-interviewable due to language barriers (non-Spanish).

At the close of the field period, there were 235 cases (1%) in callback status.

The participation rate for Version 2 is based on the following elements:

• 4,121 completed interviews;

• 2,693 cases in which someone in the household completed the household screen, but no one in the household was found to be eligible for the full interview (2,614 were age-related screen-outs among the age 16-39 oversample); and

• 1,538 refusals to be interviewed (including 1024 second refusals) and 40 terminated interviews.

Based on the standard calculations of participation rate, the participation rate for Version 2 was 81.2 percent.

The Final Summary Disposition of the Version 2 sample is given in Table 4, on the next page. The table includes breakouts for each survey component (national youth and adult cross-section and the age 16-39 oversample). The average interview length for Version 2 was 16.3 minutes.

Sample Weighting

The characteristics of a perfectly drawn sample of a population will vary from true population characteristics only within certain limits of sample variability (i.e., sampling error). Unfortunately, social surveys do not permit perfect samples. The sampling frames available to survey research are less than perfect. The absence of perfect cooperation from sampled units means that the completed sample will differ from the drawn sample. In order to correct these known problems of sample bias, the achieved sample is weighted to certain characteristics of the total population. Each of the survey samples was weighted separately.

The weighting plan for the survey was a multi-stage sequential process of weighting the achieved sample to correct for sampling and non-sampling biases in the final sample. The first stage in the sample weighting procedures was designed to correct the cases in the completed sample for known selection biases in the sampling procedures. At the household selection stage, a random digit dialing process will give households with more than one telephone number an unequal likelihood of selection. Nationally, about 18% of households selected by random digit dialing will have more than one telephone number. This selection bias was corrected by giving each household a first stage weight equal to the inverse of the number of different telephone numbers in the household, up to a maximum of three phone numbers..

The second step in the weighting process was to correct for selection procedures that yielded unequal probability of selection within sampled households. Although the survey was designed as a population survey, only one eligible person per household could be interviewed (because multiple interviews per household are burdensome and introduce additional design effects into the survey estimates). A respondent's probability for selection is inverse to the size (number of other eligible adults) of the household. Hence, the second stage weight was equal to the number of eligible respondents within the household.

The next step in the weighting process was to correct the study design for deliberate disproportionate selection of younger population subsets in the sample design. The survey included both a cross-sectional sample of 3,000 respondents, aged 16 and older, and an oversample of 1,000 persons, aged 16 to 39 years old. Hence, the total achieved sample yielded a disproportionate sample distribution by age. A third stage weight was used to correct the achieved sample for disproportionate sampling by dividing the expected population distribution, based on Census projections, by the achieved sample distribution on the stratification variables. Specifically, the third stage weight corrected the sample to the cell distribution of the population for six age cohorts (16-24, 25-34, 35-44, 45-54, 55-64, and 65 or older) by gender, using the Census Population Projections for Age, Sex and Race for 1998. After these corrections were made, no further weighting by other Census characteristics (e.g., race) was considered necessary or desirable.

The final step in the weighting process was designed to correct for the fact that the total number of cases in the weighted sample was larger than the unweighted sample size because of the use of the number of eligibles weight. In order to avoid misinterpretation of sample size, the total number of cases in the unweighted sample was divided by the total number of cases in the weighted sample to yield a sample size weight. When this weight is applied, the size of the weighted sample is identical to the size of the unweighted sample.

The final weight (WEIGHT6) incorporates all of the intermediate weighting steps described above. The final weight adjusts the total completed interviews in the achieved sample to correct for known sampling and participation biases, while maintaining the unweighted sample size.

Precision of Sample Estimates

The objective of the sampling procedures used on this study was to produce a random sample of the target population. A random sample shares the same properties and characteristics of the total population from which it is drawn, subject to a certain level of sampling error. This means that with a properly drawn sample we can make statements about the properties and characteristics of the total population within certain specified limits of certainty and sampling variability.

The confidence interval for sample estimates of population proportions, using simple random sampling without replacement, is calculated by the following formula:

The sample sizes for the surveys are large enough to permit estimates for subsamples of particular interest. Table 5, on the next page, presents the expected size of the sampling error for specified sample sizes of 8,000 and less, at different response distributions on a categorical variable. As the table shows, larger samples produce smaller expected sampling variances, but there is a constantly declining marginal utility of variance reduction per sample size increase.

However, the sampling design for this study included a separate, concurrently administered oversample of youth and young adults (age 16-39). Both the cross-sectional sample and the oversample of the youth/younger adult population were drawn as simple random samples; however, the disproportionate sampling of the age 16-39 population introduces a design effect that makes it inappropriate to assume that the sampling error for total sample estimates will be identical to those of a simple random sample.

In order to calculate a specific interval for estimates from a sample, the appropriate statistical formula for calculating the allowance for sampling error (at a 95% confidence interval) in a stratified sample with a disproportionate design is:

Although Table 5 above provides a useful approximation of the magnitude of expected sampling error, precise calculation of allowances for sampling error requires the use of this formula. To assess the design effect for sample estimates, we calculated sampling errors for the disproportionate sample for a number of key variables using the above formula. These estimates were then compared to the sampling errors for the same variables, assuming a simple random sample of the same size. The two strata (h¹ and h²) in the disproportionate sample were all respondents age 16-39 and all respondents age 40 and over respectively. The proportion for the 16-39 year old stratum (w¹) was 45.7 percent while the proportion for the 40 and over stratum (w²) was 54.3 percent.

As shown in Table 6 below, the disproportionate sampling increases the confidence interval by about 2 percent, compared to a simple random sample of the same size. This means that sample design introduces almost no measurable loss in sampling precision for total population estimates, while increasing the precision of sampling estimates for the target population aged 16-39 years old. Since the difference in sampling precision between the stratified disproportion sample and a simple random sample is less than one tenth of percentage point in each case, the sampling error table for a simple random sample will provide a reasonable approximation of the precision of sampling estimates in the survey.

Estimating Statistical Significance

The estimates of sampling precision presented in the previous section yield confidence bands around the sample estimates, within which the true population value should lie. This type of sampling estimate is appropriate when the goal of the research is to estimate a population distribution parameter. However, the purpose of some surveys is to provide a comparison of population parameters estimated from independent samples (e.g. annual tracking surveys) or between subsets of the same sample. In such instances, the question is not simply whether or not there is any difference in the sample statistics which estimate the population parameter, but rather is the difference between the sample estimates statistically significant (i.e., beyond the expected limits of sampling error for both sample estimates).

To test whether or not a difference between two sample proportions is statistically significant, a rather simple calculation can be made. Call the total sampling error (i.e., var (x) in the previous formula) of the first sample s1 and the total sampling error of the second sample s2. Then, the sampling error of the difference between these estimates is sd which is calculated as:

Any difference between observed proportions that exceeds sd is a statistically significant difference at the specified confidence interval. Note that this technique is mathematically equivalent to generating standardized tests of the difference between proportions.

An illustration of the pooled sampling error between subsamples for various sizes is presented in Table 7. This table can be used to indicate the size of difference in proportions between drivers and non-drivers or other subsamples that would be statistically significant.

Statistical Comparisons Between Samples

In order to permit statistical comparisons between the two samples, the data sets from the two separate samples were merged together on like questions. The sample versions (1 for Safety Belt Usage and 2 for Child Safety Seats) were crosstabulated with each of the survey questions which had been asked in an equivalent fashion in the two samples. A chi square test was conducted for each of these crosstabulations to test for the independence of samples.

An exact test of independence was calculated to test the differences between the two samples. Pearson's chi square is a widely used statistic to test the hypothesis that the row and column variables are independent. It is calculated by summing over all cells the squared residuals divided by the expected frequencies. The calculated chi-square is compared to the critical points of the theoretical chi-square distribution to produce an estimate of how likely (or unlikely) this calculated value is, if the two variables are in fact independent. This probability is also known as the observed significance level of the test. If the probability is small (usually less than 0.05), the hypothesis that the two variables are independent is rejected.

No statistically significant difference (at the .05 level) was found between the two samples on most demographic characteristics, including race, ethnicity (Hispanic), educational attainment, and marital status. There is no difference between samples on most vehicle characteristics (e.g., airbags, type of seatbelts) or driver behaviors (e.g., drive everyday, always wear seatbelts).

There are, nonetheless, a limited set of differences large enough to be statistically significant with samples of this size. The proportion of van/minvan drivers in Version 2 is lower (8.2%) than in Version 1 (10.8%). The proportion of primary vehicles with airbags is higher in Version 1 (53.9%) compared to Version 2 (50.6%). The proportion of those reporting they had been injured in an accident was higher in Version 2 (25.4%) than in Version 1 (22.9%); and those who said their injuries prevented them from performing activities for at least one week were higher in Version 1 (55.5%) than in Version 2 (49.8%). The reported income between the two survey versions was also found to be significantly different. All these differences are likely due to the large sample size.

Finally, although the proportion of past month drinkers is the same in the two surveys (48.8% to 47.6), the proportion who report driving after drinking in the past month is significantly lower in the Child Safety Survey (20.9%) than the Seatbelt Survey (27.3%). This last difference probably reflects a contextual effect (willingness to report drinking and driving after an intense discussion of child safety in cars) rather than sample differences.

TOP | NEXT | TABLE OF CONTENTS |