FASD and TRC Call to Action 34.4: A Consideration of Evaluation Methods
FASD Programs and Environmental Scan
The following is not intended to be an exhaustive list of FASD-related justice programs and evaluations. Not all programs have a public or online footprint. Very few programs disclose their evaluation methods and it is possible other programs may have been evaluated, but the findings have not been released. What is listed is a sample of some programs that have been evaluated, with the aim to offer guidance when thinking about creating evaluation for justice programs for people with FASD. We begin with a brief discussion of an environmental scan conducted in 2015 on FASD and justice programs in Canada, followed by a discussion guided by the research questions that directed this project.
In 2015, Dr. Stewart and her research team at the University of Regina prepared an environmental scan that examined FASD and justice programs across Canada (Stewart 2015). This document aimed to detail programs and practices that serve both youth and adult offenders with FASD throughout the country. Nationwide, a total of 13 programs were identified, with 8 programs for youth offenders with FASD and 5 programs for adult offenders with FASD. As indicated by the environmental scan, there are a limited number of programs across Canada that serve adult and youth that are involved in the criminal justice system and who are diagnosed with, or suspected to have, FASD. Many current programs have been created by local FASD organizations, associated foundations, and governmental organizations. Some of the programs include intensive supports in residential settings, while others primarily act as a place for case management and referral. Several of the current programs offer a range of supports, education, training, and outreach services.
As noted earlier, “research indicates that a disproportionate number of people involved in the criminal justice system have FASD (diagnosed, suspected, or otherwise) which places pressure on this system to address the needs of these clients—and raises questions about where needs can be best met” (Stewart 2015, 3). The environmental scan found a “lack of FASD training and education amongst frontline justice personnel (police, judges, lawyers, corrections, and court staff), which can impact equal justice for persons living with FASD” (Stewart 2015, 3). Further, those with FASD face barriers to justice due to a lack of appropriate supports before they have contact with the criminal justice system, and they are then met with even fewer options once they enter the criminal justice system. As the majority of assessment tools within the criminal justice system have been designed with youth in mind, adults are frequently underserved. Additionally, the report found a lack of services and procedures designed to adequately support persons with FASD, which can further complicate encounters with the criminal justice system.
Many services, policies, and programs do not take aspects of the disability into account. An individual with FASD can have sensory processing disabilities, and become distressed and distracted in courtroom settings while in custody, to the point of not hearing or understanding what is being said. Some courtrooms have made adjustments (such as lowering lights) which is a promising practice and opens the door to other much needed changes. Stewart (2015) also indicated a “… growing recognition within the court system that FASD is a relevant and timely matter…” and that “… it can impact a wide range of justice outcomes including sentencing practices and supervision in the community” (Stewart 2015, 26-27). While there have been some developments since the report, such as extended pilot programs and the new FASD Court in Manitoba, there is still an urgent need for improved access to court and community supports that are FASD-informed and culturally appropriate. What is also clear from the environmental scan is that there needs to be strong, relevant evaluation of the programs across Canada that are addressing FASD and the criminal justice system. Those evaluations need to be made publicly available for greater accountability, transparency, and to communicate best practices with others offering similar services. These evaluations also need to include the perspectives of those with lived experience.
It is critical to involve individuals with FASD in program evaluation as they are the consumers of the service. Best practices in evaluation indicate the need to use plain language when developing surveys for the public. An appropriate guideline in the field of FASD evaluation would be to write at grade five literacy level. This can be achieved by using MS Word features to measure the literacy rate when authoring surveys. Research also shows that a best practice would be to allow “reasonable” time in conducting interviews and allowing for adequate breaks. When needed, participants can be asked periodically for their consent to allow for ongoing data collection at different times. Further, research also tells us that shorter interviews with a simple consent form allows for informed consent with “brain breaks” built in as needed. Data collection should not exceed 30 minutes if possible.
An effective practice with a project in British Columbia included team interviews with a trusted worker, researcher, and individual(s) with lived experience all participating in a guided discussion. This method can produce a lot of qualitative data. It allows for there to be a trusted individual in the room as well as individuals to “translate” the question as needed, to best ensure informed consent as well as understanding of the questions. Examples of this method can be found in a different FASD program evaluation that was conducted by Muhajarine et al. (2013) who evaluated government-funded FASD programs in Saskatchewan which included interviews with agencies as well as clients who received services. The letter of invitation to clients was written in clear language, notes the time commitment, and the rights to participate or to decline participation. The full report is available online and is also listed in the Further Reading section.
There are dedicated dockets and specialized courts across Canada, some of which have existed for well over 30 years. Some of these courts, more recently, have also placed a focus on FASD, including the recently opened FASD Court in Manitoba that started hearing matters in Spring 2019. Prior to this entirely dedicated docket that branches off of youth justice, the Wellness Court in the Yukon, Gladue Court in Ontario, and dedicated Mental Health Courts in Saskatchewan have all placed specialized attention on using the strengths of a therapeutic court, to try to bring about better justice outcomes for individuals with complex disabilities like FASD.
The Yukon Wellness Court started in 2007 and was evaluated in 2011 (Hornick, Kluz, and Bertrand 2011). The court evolved from a pilot project to a long-standing and nationally recognized alternative justice practice. While not FASD-specific, the court does handle clients with FASD. The evaluation of the court included a variety of methods designed for a summative analysis (focused on outcomes) that tested the overall effectiveness of the court and whether the court was being implemented as planned.
The team recognized a number of limitations and challenges in the evaluation process including the lack of a control group, limited client information, and a low number of clients. The evaluation strategy was composed of several components, with only one part (interviews with court clients) relating directly to the purposes of this paper. The report indicates interviews were conducted with clients, though does not explicitly state any detail of how those interviews were conducted, and more specifically if any accommodations were made to increase accessibility to those with FASD. A promising practice was to engage with the clients that did stay involved with the courts and to include their perspectives. Of note, all clients were thanked for their participation, including those that participated by sharing their stories. This is an example of an FASD-informed approach to evaluation, where all participants are valued equally. All survey tools were shared and could be modified in other evaluations to become more FASD-client friendly and accessible.
There are now two dedicated mental health dockets in Saskatchewan that serve clients with FASD. Each of these courts were evaluated a short time after implementation. In 2015, Barron, Moore, Luther, and Wormith released “The Process Evaluation of the Mental Health Strategy” which focused on the results of the opening year of operations of the court that included clients with FASD. The team used an evaluation matrix, with a breakdown of their qualitative methods for conducting interviews with key stakeholders, including questions and indicators. Additionally, quantitative methods were used to conduct a closed file review. Some modifications were made to evaluation approaches to accommodate individuals with FASD, such as shorter interview times with opportunities to take breaks. Additionally, the evaluation aimed to streamline (remain consistent) and simplify language where possible, which would also work to accommodate individuals with FASD.
One year later “The Mental Health Disposition Court - A Formative Investigation” was released by Stewart and Mario (2016) which focused on the dedicated docket in Regina, Saskatchewan. The researchers used mixed methods in this evaluation. Qualitative methods included participant observation and analysis of pre-court and in-court practices as well as semi-structured interviews with professionals and clients. Quantitative methods included case file review and analysis of court outcomes including statistical analysis of demographics and breakdown of charges. A new questionnaire tool was developed with an advisory group to better include the experiences of those who had participated in the court. The questionnaire allowed for participants to use a visual aid to illustrate their experience, by allowing them to use poker chips to indicate areas they felt they needed help in. This allowed for a unique form of communication, and a concrete way of understanding. The authors indicated that while there were a limited number of participants, use of this tool and similar modified tools should be revisited.
The Asante Centre in British Columbia runs a peer-to-peer mentoring program where two adults with FASD run group sessions for youth with FASD to help them learn about their disability and focus on their strengths. While not justice-related, this program’s evaluation strategies are worth mentioning for future adaptation to the criminal justice context. This is a peer mentoring program meant to help youth not only make strong pro-social relationships, but also understand their disability. This program has been evaluated (2018) and is being evaluated again (2019) through a collaboration that brings together researchers, agency staff, and mentors with FASD that deliver the program. The team collaboratively co-designed parts of the program as well as the evaluation tool. The mentors have been using the “Playing To Our Strengths Toolkit” (see further information: improvenabled.ca) developed by Dr. Michelle Stewart and Dr. Rebecca Caines since May 2018. The Toolkit is a community-based project and came about following consultation with caregivers and individuals with FASD. The toolkit responds to a need for strengths-based resources and research in the field and has been evaluated with a focus on how the peer mentors modified it for their purposes.
The evaluation strategy focused on the program goals to derive the evaluative questions. The evaluation method was weekly face-to-face semi-structured interviews with the peer mentors. As such, the first round of evaluation was aimed at understanding the usefulness of the Toolkit from the mentor’s perspective. It was acknowledged that while the Toolkit was originally developed for people with FASD, it was not fully considered if the Toolkit was user friendly, to be delivered by people living with FASD. This round of evaluation set out to measure the usefulness of the Toolkit for use by people living with FASD. Currently evaluation has transitioned to focus on youth interpretations of the mentoring sessions through arts-based evaluations. Arts-based evaluation methods can potentially improve accessibility so that more individuals participate in evaluation. For the purposes of the Toolkit evaluation, participants used a memory box. The youth placed items inside the box each week that reminded them of the weekly lesson. In the following weeks, the youth revisited their memory boxes to discuss the meaning of the weekly and overall mentoring group, and to create audio and visual content for evaluation.
Critically, the authors want to draw attention to the possibility of looking at programs that are co-designed, meaning they are created within the community for individuals with FASD. The need for the participation and insights of individuals with FASD, and the need for evaluation to consider the impact of what co-designed practices can look like, cannot be emphasized enough. While there are supports and services available within the criminal justice system, the first place to look for best practices is in the community; the first place to invest in individuals is in the community; and the first place to develop and foster strengths within the individual and the family is in the community. Co-designed programs and evaluation open the door to creating non-stigmatizing tools for evaluation and tools that are culturally-responsive as well as mutually-beneficial. Dreise and Mazursk (2018) note that the co-design methods address colonial legacies in which research and evaluation was done upon Indigenous communities rather than with communities. Just as arts-based evaluation methods can assist more individuals in becoming involved in evaluation, so too can co-design offer new insights. By engaging in co-designed methodologies, the evaluation or research team works collaboratively to identify items of interest, which in turn will impact the types of questions that are asked about a particular project. When Indigenous peoples are engaged with directly, and the evaluation design is collaborative there is an increased likelihood for more robust and potentially meaningful findings. There is also the potential to develop long-term relationships and shared capacity building, which can assist in the feasibility of the evaluation itself.
- Date modified: