Program Evaluation Memo

22May06

Measurement, Evaluation, and Ethics in Research

The following assignment was a little bit different from most of the other standard essays that I’ve had to write. Essentially we were asked to prepare a detailed memo. The assignment requirements were as follows:

  • You are working for an early childhood program that has not been evaluated. Your supervisor, knowing that you are studying research methods, asks you if you believe that the program should be evaluated. Indicate to the supervisor the objectives of program evaluations.
  • When you finish telling your supervisor the objectives of program evaluations, he asks you how program evaluations are done. Briefly explain to the supervisor the types of program evaluations.
  • Finally, the supervisor asks you how one conducts a program evaluation. Describe to your supervisor the steps to conducting a program evaluation.

The following is my response to these questions.


Dear Sir,

Thank you for your interest in my academic work. Program evaluations can be very valuable, especially if they are well implemented. For this reason, I have prepared a brief memo presenting some elements of program evaluations which may help you determine whether you should invest resources in such an evaluation.

Part 1: Types of Program Evaluations and Program Evaluation Goals

There are several program evaluation models and several different objectives of program evaluations which should be considered before proceeding to conduct a formal evaluation. Evaluations can focus on the program methods and the perspective of the participants, they can try to assess the program’s effectiveness of meeting objectives, or they can be designed to give management tools to make more informed management decisions. Additionally, they can be an opportunity to bring in an expert in the field and they can be a venue where differing views of the program can be analyzed productively.

Ultimately, the goal of an evaluation is to help reinforce the current position of the program’s design. It should be able to help identify which elements of the program are effective, which elements need to be changed, and whether the range of services offered should be modified. Evaluations consider whether needs are being adequately served by the program, whether the program is being implemented as it was originally designed, and whether the program is effective at achieving its originally designed goal without any ill side-effects. Program evaluations also help identify the relative value or effectiveness of each element of the program’s design and give program managers the tools to help them maintain and improve programs.

Considering that this preschool program has not yet been evaluated, there are certain aspects which may be valid program evaluation candidates. Currently, a program evaluation method that helps either identify the effectiveness of the program of meeting its goals or an evaluation designed to help identify the value of the program may be a prudent idea from a managerial and quality control perspective. Before you decide to begin a program evaluation, however, it is important that you first understand the different types of evaluations that we can conduct.

Part 2: Four Types of Program Evaluations

While there are different types of program evaluations, all types can be categorized as either formative or summative evaluations. Formative evaluations can broadly be viewed as evaluations where the findings are used to promote the design and implementation of new programs. Summative evaluations, on the other hand, look at the overall effect of the program on its participants. With that in mind, here are the four types of program evaluation: (1) needs-assessments, (2) process evaluations, (3) outcome evaluations, and (4) efficiency evaluations.

Since programs are designed to serve a need, a needs-assessment program evaluation looks at the level of need within a program. It also looks at the prevalence and persistence of the problem—in other words, how often does the problem occur and is the problem a persistent or a temporary one? A needs-assessment evaluation gathers data from a lot of outside sources which help determine the needs for a particular demographic. Needs-assessments are very helpful when trying to either develop or restructure a program.

Process evaluations focus on the operation of the program in comparison to how it was designed. These evaluations help determine whether the program is adequately serving the originally targeted population as it was expected to. Because of its nature, process evaluations need to consider both the program’s description and its ultimate form of implementation. Because of the need for accurate information about the actual operation of the program, evaluators have to work with program managers to develop effective and reasonable methods of summarizing and sharing records of services offered by the program.

Outcome evaluations are frequently used for program evaluations. These forms of evaluations typically develop summary statements about the merits of the program. Outcome evaluations look for information which can identify (1) whether program participants have improved, (2) how participants have changed compared to a control or quasi-control group, (3) whether the results for this program are typical and can be generalized, and (4) whether there is a clear level of causality between the services provided by the program and the outcomes observed in the participants.

Finally, efficiency evaluations consider the cost effectiveness of the programs. In considering whether a program is cost effective, both the fixed costs (for example, rent) and the variable costs (for example, supplies) of operating the program need to be considered. Efficiency evaluations may also include a cost-benefit analysis comparing the input costs of the program to the gains which are ultimately attributed to an individual’s participation in the program.

As you have probably determined from the above presentation, outcome evaluations and efficiency evaluations may be a worthwhile investment since they can probably both help increase enrollment at this preschool as well as point out cost-cutting measures that can be made. For outcome evaluations, since testing is a somewhat unreasonable expectation for our young participants, we will probably have to rely on extensive qualitative data from both program staff and from parents. We do not have a control group, but there is data available which looks at the academic achievement of kindergarten students. A literature review of the value of preschool in easing the transition into elementary school can be used in conjunction with our qualitative data to help determine the value of this program.

Efficiency evaluations are also especially important for us. As you know we have a range of enrollment options including full-time, part-time, and extended-day students. Some students come to our program as many as five days each week or as few as two days each week. An efficiency evaluation will help determine whether we are adequately staffed throughout the week to account for the number of students we expect on any given day. This is also an important consideration for legal reasons since we want to ensure that the program has enough credentialed teachers working each shift keeping the teacher-student ratio at the state requirement. Additionally, this will help budget for supplies, snacks, and other variable costs involved with this preschool. An efficiency evaluation can also help identify whether our fixed costs are appropriate for the program’s long-term objectives. If for example, enrollment has been chronically lower than normal and is not expected to increase, it may be advisable to either find a smaller, lower cost place to operate the preschool or consider consolidating classrooms and leasing spare rooms to compensate for lower revenue from participants. If, on the other hand, the efficiency evaluation sees a rising trend in enrollment and decreasing efficiency due to over-crowded classrooms and so on, exploration of alternative real-estate would probably be necessary.

Part 3: Steps Involved in a Program Evaluation

Once the objectives and the most appropriate type of program evaluation have been determined, the program evaluation process can begin. For the implementation of an evaluation, one must establish the evaluation’s boundaries, determine the best method of evaluation, conduct the collection and the analysis of information, and report the findings of the evaluation.

Looking at these steps in a little bit more detail, when we are establishing the evaluation’s boundaries, we are essentially determining why the evaluation is taking place and what the evaluation will focus on. Any potential barriers to the evaluation and methods of dealing with those barriers are also identified at this point. After the boundaries are established, the evaluation methods—for example the data collection procedures, methods of analysis, and sampling methods—are determined. These determinations are also based on the form of the evaluation since qualitative and quantitative evaluations will require different evaluation methods. Qualitative evaluations, for example, require that the evaluator considers the effect of the program from the perspective of the participants. Quantitative evaluations, on the other hand, may consider at pre-experiment data, or may compare the results of the program’s participants with a control group or a quasi-control group.

When collecting the data, a clear and reasonable time-frame must be developed. The time-frame must be designed in such a way that everyone responsible, for example program managers and program evaluators, can collect and organize their data in a way that allows for effective data presentation and analysis. How the data is analyzed is a determinant of several factors including the form of the program evaluation (for example whether it is qualitative or quantitative) and how the program was designed (for example if it was an experimental design or whether it included a control or quasi-control group). Additionally, the program evaluator needs to determine whether there were any validity threats and look at how the threats were dealt with.

While all of the above deal with the collection of information for the program evaluator, the evaluator’s work is not yet done. The final step in conducting a program evaluation is the reporting of the findings. Among the things that need to be considered by the program evaluator is the target audience and the method of delivery. In some cases they may have to present the same findings to many different parties and consequently may require the preparation of several different reports.

From the perspective of this preschool, a program evaluation may be beneficial for several reasons. As mentioned before, an evaluation can help with management decisions regarding the cost effectiveness of the program’s operation. If an outcome evaluation is to be conducted, it would be important to carefully design the qualitative surveys. The results of these surveys, while they will probably be of most use to us internally for improving the program’s offerings, will probably also be usable as advocates for the preschool when recruiting new clients. Results can be presented to a range of audiences. The results can be shared with the program staff to reinforce the value of the work that they are doing, and can be shared with existing parent clients to reinforce that we are concerned about the quality of the care that we are providing their children.

Please note that it would probably also be important to make the reasons for the evaluations clear to the families enrolled at our preschool. It could also be important to clarify that no personal information will be collected ant that their privacy will be respected. Also, they may be assured that the program will be targeted towards evaluating the program, and not at evaluating their children.

Again, thank you for your interest in what I am doing outside of work, and I hope this helps you make your decision about whether you should invest in evaluating your program.

Advertisements


No Responses Yet to “Program Evaluation Memo”

  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: