FASD and TRC Call to Action 34.4: A Consideration of Evaluation Methods

Limitations

The lack of accessible evaluations presents a barrier in meeting the TRC CTA 34.4 in that there is an inability to easily compare the details of the evaluations between existing programs. Programs might be engaged in evaluation or quality assurance, but the information is not necessarily released for public consumption. Having this information readily available would assist greatly in creating momentum toward appropriate evaluations. Final products could be analyzed, adapted, and improved through a collective effort.

There are various challenges in conducting evaluation. Some of those challenges already mentioned include deciding on the appropriate approach (quantitative/qualitative), selecting the proper tool within that approach, accessing the participants needed to gather the information, and ensuring an ethically rigorous process.

Some other limitations that might surface are the common issues of time and money. Ongoing evaluation is important to acquire a holistic understanding of a program’s success, but this can be time-consuming and utilize abundant human resources. One potential solution to the lengthy process of evaluation is using continuous feedback loops of information. For example, if you want to know how a program is working, you might consider asking your clientele after each service provided to give feedback or rate their experience. This will provide data, though it also comes with considerable limitations. For example, collecting information after each service can not only deliver skeletal information but also require the need to interpret why any given response was provided. This type of data also comes with limited amounts of baseline data in early stages. However, once more data is gathered, follow-up can be conducted through select interviews to gather more information about the success of service.

Another limitation of evaluation is that it can be quite expensive, depending on the length of evaluation, the number of people needed to conduct the evaluation, supplies needed (paper, recording devices, computers, art supplies, etc.), and the creation of final products/reports. There are sometimes grants available to conduct undertakings such as evaluation; though to secure such funds requires well developed proposals detailing what the evaluation is, how it will be conducted, a timeline, and needed resources. As such, the following considerations could be useful in crafting effective evaluation strategies.

Evaluating a program inadequately (i.e., not examining the program from all angles or not asking the right questions) or evaluating a program without the input of the aforementioned representatives, is likely to result in moot findings. Moreover, evaluation efforts should include this wide array of individuals from imagining and designing the evaluation, conducting the evaluation, to producing results. Evaluation mechanisms need to be accessible. Such accessibility issues that might be considered are:

These are a few examples, and not an exhaustive list of items that need to be explored. As these would have to be individual, community, and circumstantially specific, these are only four possible accommodations.