Aboriginal Justice Strategy, Summative Evaluation

3. Methodology


3. Methodology

Several lines of evidence have been used to address the evaluation issues and questions listed in Appendix A.

3.1. Document review

Primary sources of information, provided by the Aboriginal Justice Directorate, were reviewed. The list of these documents included AJS annual reports, internal briefing notes, AJS official documentation, program data, as well as samples of contribution agreements, memoranda of understanding, activity and financial reports.

Secondary sources of information were also reviewed, including statistics from the Canadian Centre for Justice Statistics, as well as recent studies and court decisions addressing issues relating to Aboriginal offenders. These documents are referenced throughout this report.

3.2. Case studies

In 2006, the Department of Justice's Evaluation Division conducted case studies with 10 communities that have established community-based justice programs through AJS funding and that volunteered to participate in this process. The selected case studies include a diverse mixture of programs that serve different types of communities (including on- and off-reserve communities). As part of these studies, documents for each of the selected communities were reviewed and five individuals for each of the case study programs were interviewed, including justice coordinators, police officers, victims, offenders, justice committee members, city officials, elders, prosecutors, probation officers, and defence counsel. A total of 63 individuals were interviewed.

The method known as Photovoice© was also used to explore the impacts of community-based justice programs.[7] In a process spanning one to two days, participants were given cameras and asked to take pictures based on the following themes:

These individuals subsequently met, either individually or in a small group, to put their words and stories to the pictures. A total of 41 people participated in the Photovoice© exercise. It should be noted that findings from these case studies cannot be viewed as representative of all AJS funded community-based justice programs. Pictures and quotes from case study participants are included in this report for illustrative purposes only. A summary of the case studies is included in appendix B.

3.3. Key informant interviews

During the months of October and November 2006, a total of 28 semi-structured interviews were conducted with representatives from the Aboriginal Justice Directorate, other federal departments, provinces and territories. These interviews were conducted either in-person or over the phone. All key informants received an interview guide prior to the interview itself.

3.4. Surveys of stakeholders

Two surveys were conducted as part of this evaluation:

3.5. Recidivism study

In July 2006, the Department of Justice's Evaluation Division completed a study on the impact of the AJS on rates of re-offending (hereafter referred to as the recidivism study). A summary of this study is included in appendix C. The criminal behaviour of individuals who participated in an AJS program was compared with the criminal behaviour of individuals who were referred to, but did not participate in, an AJS program. In total, the study considered 4,246 offenders (3,361 AJS program participants and 885 comparison group members) from nine programs across Canada. Close to 60% of offenders in the total sample were tracked for at least four years. In order to determine whether AJS program participants and comparison group members differed in their likelihood of re-offending, a series of survival analyses were used.

As a result of methodological limitations, the interpretation of findings from the recidivism study must be made with caution. Among other things, it was not possible for the study to use random assignment to either the program participant group or the comparison group; to use data from offenders who systematically went through the mainstream justice system; or to use a more representative sample of AJS participants. Despite these limitations, and as further discussed in this report, the consistency of the findings provides helpful insights on the impacts of AJS on Aboriginal offenders.

3.6. Cost analysis

A cost analysis was undertaken to explore the cost implications of the AJS. To this end, activity and financial reports were reviewed for nine AJS programs located in Ontario, Manitoba, Saskatchewan, and British Colombia. In addition, justice spending in Newfoundland and Labrador, Ontario, and British Colombia was reviewed, covering court expenditures (including prosecution) and legal aid, but excluding policing and costs associated with carrying out sentences, including those incurred by correctional facilities. These last items were excluded on the basis that policing work is needed in both AJS and non-AJS scenarios, and that the ratio of offenders who would end up sentenced, and the type of sentence they would be given, could not be established. Only court expenditures relating to summary offences were considered for this study, as these offences most closely resemble those for which an Aboriginal offender could be referred to a community-based justice program. A number of critical factors and methodological limitations for this methodology are identified in the discussion of the cost-effectiveness of community-based justice programs in Section 4.0.