Back to Table of Contents

Part Three: Program Evaluation General Evaluation is the means by which the program's activities are monitored and judgments are made about its impact. A thorough evaluation process will provide essential information about the content, conduct, progress, and outcome of the effort, and allow management to objectively judge the effectiveness of a program. Evaluation of the Hot Sheet enforcement program should, at a minimum, address four fundamental questions: 1. What was the program's intent? (who, what, when, where, how) 2. What level of activity did the program achieve, when compared to preprogram activity? 3. What were the costs involved in operating the program? 4. What was the impact or outcome of the implemented activity? The program evaluation should be based on the premise that managers and administrators thoroughly understood the nature and extent of the problem. If the problem is not understood or defined, it is impossible to formulate effective strategies and countermeasures. Also, it's impossible to measure the effect or impact of the program. Vague or ambiguous program objectives cannot be objectively evaluated. Program evaluation is an objective process that reveals both program strengths and weaknesses. It requires both resourcefulness and commitment in order to provide a complete picture of what is happening, or has happened, with the enforcement effort and is the best guide for decisions about future enforcement efforts and strategies. Administrative Evaluation The administrative evaluation provides a complete description of activities and costs. In the administrative evaluation, consideration should be given to the following: What activities or strategies were used? Where, when, how were they deployed? What was the level of supervision? Who were the supervisors? What was the level of activity? How did this compare to preprogram activity levels? Was this level of activity enough to affect the problem? What was the cost of carrying out the activity? Usually, administrative evaluations are monitoring processes using existing data sources. If so, they can be completed using available personnel. Impact Evaluation It is recommended that agencies do an evaluation of the program to determine the impact the enforcement strategy had. This effort should determine the degree of influence the program had in resolving the suspension or revocation problem. Conducting an impact evaluation can be more complicated than an administrative evaluation because it involves quantitative analysis. At times, outside factors (i.e., extended adverse weather conditions, a seasonal local event that distracts enforcement efforts) may affect the outcome whether independently or combined. An impact evaluation must account for external factors and requires the skills of someone with experience in program evaluation and quantitative analysis. Individuals with these skills can be found on the staffs of large municipal governments or at colleges or universities. Data Collection For both the administrative and the impact evaluations, collection of data is critical. Data must be available for managers to monitor who, what, where, when, and how, the project is being done, and for analysts to decide how these data relate to preprogram data. Data must be available to monitor how the problem changes over time and to show the costs involved in carrying out program activities. Reliable, accurate data are essential for showing the extent to which the enforcement activities influenced the operating after suspension or revocation problems.

Back to Top of Page

Back to Table of Contents