FASD and TRC Call to Action 34.4: A Consideration of Evaluation Methods

Conclusion

The authors hope that the reader has a better background and understanding about the broader contexts associated to FASD after reading this document. And that these truths can help inform changes in programs and policies to better meet the needs of individuals with FASD alongside ideas on how to evaluate these changes. Further, the authors also hope that it is clear that there are many different methods to evaluate changes in programs and policies but that these methods must include the voices of those most impacted: people with FASD. This will be challenging work but there are proven methods to effectively include individuals with FASD which is critical for reconciliation, truth, and justice. To conclude, the research questions will be re-visited, offering a summary response to each.

  1. What evaluation mechanisms have been used by existing FASD programs?

This piece offered a short summary of some of the evaluation mechanisms that exist when looking at FASD programming. This includes programs and practices inside and outside the criminal justice sector, and also a summary of promising practices and the role of collective wisdom in programming and evaluation. Unfortunately, publicly available evaluation plans and final reports are scarcely available. Based on the few publicly available reports, the authors’ extensive experience in the field of FASD and evaluation, and existing documents on program design and evaluation, this paper has offered a plethora of evaluation techniques and strategies. Some of these include qualitative approaches (interviews, oral histories, photovoice), arts-based approaches (memory boxes, drawing, music, dance, etc.), and quantitative approaches (surveys and administrative file reviews).

  1. What are some promising practices that exist in the evaluation of FASD programming?

Qualitative and arts-based methods show promising possibilities in making evaluation processes accessible to those with FASD and their communities. These methods can also capture rich information that is sometimes lost in quantitative data. Quantitative methods are also significant in the evaluation of current programs. These methods provide necessary empirical data and allow for a greater number of subjects to be studied. It is important to consider the option of co-design, where all stakeholders, including those with FASD, are involved in the process of evaluation to ensure the programs are responsive to their needs. This would involve creating evaluation processes that are accessible and building sustained relationships with stakeholders. Knowing this, a combination of methods could be most appropriate in the evaluation of programs for individuals with FASD.

  1. What would be appropriate mechanisms to evaluate FASD programs in the criminal justice system?

As was discussed, there are a range of options to choose from when it comes to evaluating FASD criminal justice programs. The methods chosen will depend on the community that the program exists in. The authors of this document strongly advocate for community involvement at all stages. In addition, if programs are being designed specifically for Indigenous participants this work must be done through relationship building and consultation. All programs and evaluations of this nature should be community-driven and co-designed with Indigenous communities, agencies, and families.

  1. What are hallmarks of programs and evaluation practices that are appropriate (for example being culturally-responsive and not perpetuating stigma)?

Many key elements have been identified as hallmarks of appropriate evaluation, such as: including those most directly affected, ensuring person-centered approaches, adapting all methodologies to best support and include those with FASD (slowing down, taking breaks, reducing distractions, accounting for sensory sensitivities, etc.). The “Best Practices for FASD Service Delivery: Guide and Evaluation Toolkit” is one resource to consider looking at for modification when engaging in program design and evaluation. Also included at the end of the document are further readings (in Annex) with information on evaluation methods. Unfortunately, there is a limited amount of data that is publicly available about how agencies engage in evaluation; an issue that will hopefully change in the future. In the end, each program will be unique. Each program should be responsive to local needs and to the best of an agency’s ability, program design and evaluation should be co-designed with key stakeholders. That said, the suggestions in this document are general guidelines to consider. The ideas discussed are anchored in the literature and evidence while also being grounded in collective wisdom and lived experience of individuals with FASD and their families.

The issue of precarious access to housing and community supports as well as other structural and institutional barriers should be part of program design and evaluation. These factors influence the rate at which individuals with FASD are involved with the criminal justice system. It is crucial to account for them in program design and evaluation. To not do this would risk missing a critical piece of the structural inequalities that surround the disability, that often lead directly to criminal justice system involvement. While it is challenging to build in these broader structural issues, it is ethically important to do so.

  1. What existing evaluation tools could be revised in response to 34.4 that would be useful for frontline service delivery?

Adopting appropriate evaluation mechanisms to measure the effectiveness of community, correctional, and parole programs for individuals with FASD is a complex task. The Call to Action indicates the need for collaborative involvement at all levels of government. This action also demands a comprehensive understanding of the structural, institutional, and historical contexts that surround FASD. As indicated, there is a need for more FASD training and capacity across the subfields of justice. Those conducting evaluation should also have FASD-informed expertise which comes with a broader contextual understanding of the disability. This will necessarily inform the programs and evaluation practices that take place. The methods used should reflect an awareness of these nuanced contexts for the evaluation methods to be appropriate. Further, more explicit and accessible communication about what evaluations have been done and the results, would be beneficial. Sharing evaluation methods and outcomes will build program and evaluation capacity. This capacity building is critical to address TRC Call to Action 34.4 in a robust and sustainable manner. This also speaks to broader ethics in evaluation.

While few evaluation tools are publicly available, there are some resources discussed in this document that might be revised in response to TRC Calls to Action 34.4, such as the Best Practices Guide, and Mental Health Evaluation, and the Yukon Wellness Court evaluations. While the Guide and these previous evaluations are valid and served their purpose, there are adaptations that could be made to ensure greater accessibility to those with FASD. These resources should be consulted as a starting point in crafting evaluation strategies, as they are already rooted in FASD research and expertise.

Programs developed with evaluation methods built into the design could transform the nature of the program and the range of data to evaluate. This does not mean programs should be designed in such a way that evaluation is guaranteed to render particular results. This entails designing programs with plans for evaluation built in, so that data can be collected from the start and avoid the challenges presented by retrospective data collection, as it can result in an uneven story about a particular program. Moreover, if the program is designed with evaluation in mind, the correct types of consultations can take place in advance to assist with co-design.