Evaluate and Review Your Material

Although this is the last step listed in the Guidelines, measuring the effectiveness of your material should be part of your development strategy right from the start. Evaluation can help you track your outreach efforts, discover dissemination problems early, make corrections, and find out whether your material has the desired effect. The results can help you improve future materials. Regardless of your budget, some level of evaluation can and should be done.

Determine the Type of Evaluation You Will Do

The type of evaluation you choose will depend on your resources, staff availability, the requirements of your organization, and your overall goals in developing and disseminating the material. In general, there are two ways to evaluate the material:

Process Evaluation
Process evaluation assesses how a material is being distributed, whether it is reaching the targeted audience, and whether you are receiving requests for more materials. It helps you determine what percentage of the audience actually receives the material. Is it more or less than expected?

Impact Evaluation
Impact evaluation assesses a material’s effectiveness in producing a desired change in the knowledge, attitudes, or behavior of the target audience. For example, was there an increase in the number of parents who are aware of the law requiring that children ride in booster seats? Impact evaluation determines how well a program is changing knowledge, attitudes, or behavior. Such changes seldom occur if a material is used in isolation without being part of a larger injury-prevention program, and furthermore, such changes are often short-lived and do not lead to measurable longer-term reduction in injuries. In a full-scale evaluation of a program, one would ideally want to measure the program’s impact on knowledge, attitudes, and behavior. Expecting more than a short-term behavior change (or even being able to detect a change) from an isolated material, such as a brochure, would be unrealistic. However, if the materials are part of a larger program targeting a specific community, an impact evaluation might be able to assess the effects of several different components of the program (e.g., educational materials, increased law enforcement activity, etc.) simultaneously.

Determine the Goal of Your Evaluation

The next step is to define exactly what you are going to measure. The following are examples of goals you might want to achieve with your material and questions you can ask to determine whether you are actually achieving them.

• Goal: To determine whether you are reaching the target audience
  • Is the material available at locations accessible to the target audience?
  • How many places distribute your material (e.g., schools, health care providers, library story hours, and auto dealers)?
  • Has the material been placed in a visible location?
  • How many members of the target audience do these distribution points serve?
  • Are providers complying with the planned distribution of material? Do they routinely distribute to everyone or to just a few people as time permits?
  • How many units of the material have been distributed?
  • Have you received requests for more materials? How many?
    
    
• Goal: To determine how the target audience responds to the material
  • Do audience members remember the material?
  • Do they read the material thoroughly or do they skim it?
  • Do they bring the material home or do they discard it?
  • Do they pass the material along to other members of their community?
  • Have they talked to a child or other family member about it?
  • Do they feel that the material is culturally appropriate?
  • Do they feel that the material comes from a credible source?
  • Do they understand the contents of the material?
    
    
• Goal: To determine the total cost of producing and distributing the material
  • How much staff time was devoted to community meetings, focus groups, pilot testing, and dissemination?
  • How much staff time was used to research and write the content and develop the graphics?
  • What was the cost of writers, designers, and other professionals?
  • What in-kind services were provided and what is their monetary value?
  • What additional funds came from the community?
  • What were the mailing costs?
    
    
• Goal: To determine whether those who have received the material have increased their knowledge, changed their attitudes, or changed their behavior
  • Did audience members learn new information about the causes of traffic injury or steps they can take to prevent injury, deaths, tickets, and/or arrests?
  • Did they learn about new contacts or resources?
  • Did the material change their attitudes about the causes of injuries or how to prevent them?
  • Do they report their intention to change their behavior (e.g., put child in a booster seat or cross the street in a crosswalk)?

Keep in mind that providing one-time, stand-alone educational materials that are not part of a larger strategy is seldom effective in changing attitudes and behavior. Short-term knowledge gain and/or self-reports of change are probably the most that you can expect from an isolated education material.

Once you know the questions you want answered, you are ready to develop tools and procedures to collect this information.

Collect Data for the Evaluation

Process Evaluation
Process evaluation depends on accurate record keeping. For example, it’s important to record the number of copies distributed to various organization as well as requests for additional copies. You might also want to conduct brief telephone interviews with the people in the organizations distributing the material, recording responses on a written form you create for this purpose. Another method of collecting data is to hold focus groups of individuals who are distributing or receiving materials. Without incentives, mail surveys usually generate few responses unless there is active follow-up via telephone, mail, or e-mail.

To determine the level of interest in your materials, consider distributing it in waves to organizations that will be providing it to your target audience. Send a limited number to these organizations and keep track of subsequent requests, so that you can precisely gauge how many units are being used.

Impact Evaluation
To assess knowledge, attitudes, and self-reported behavior, you can use short surveys, actual observations of behavior (e.g., use of child safety seats), or more complex pre-tests and post-tests. Choosing an appropriate method is critical for determining the impact of your material. Too often, unrealistically high hopes about the impact and poorly conceived approaches lead to a finding that there is no detectable impact. If you think you may need a consultant to assist with the evaluation because your staff lacks experience or time, involve the consultant in the process as early as possible. Choose a consultant who has performed these types of evaluations before, such as one of the following:

• An author of an evaluation paper in the literature
• A health education researcher from an academic setting
• An agency that has implemented a program similar to yours
• A professional organization such as the American Evaluation Association
• The director of your State or local health department’s injury-prevention program (See www.stipda.org and www.naccho.org.)
• Your NHTSA regional office

Test the data-gathering instrument before you begin officially collecting your information.

Report the Findings

After you have compiled and analyzed the data, write a report describing the results of the evaluation. This document could be as simple as a few pages for a process evaluation, or as extensive as a 100-page report or journal article for a large-scale impact evaluation. Having a written report will be helpful if you plan to update the material in the future, if others use or adapt the material, or if you need to decide whether to print more copies.

Whether you find that your material was effective or ineffective in influencing your audience’s knowledge, attitudes, or behaviors, that information can be useful for you and your colleagues. You have an opportunity to contribute to the traffic safety and injury prevention fields by sharing your results. In Traffic Safety in Communities of Color, Gantz et al. (2003) say that “ongoing data collection and analysis are necessary to inform interventions and efforts. More research is specifically needed to understand past traffic safety successes so that these successes can be extended.”

Continue to Review Your Material for Accuracy and Relevance

To ensure that your material stays up-to-date, you should:

• Have traffic safety experts periodically review information about laws, new technology, and new audiences in your community of interest.
  
  
• Collect feedback from your users.

Be prepared to revise your material if warranted based on the feedback you receive from experts or from your audience.

For resources on evaluation, see appendix B. Step 9.