Aboriginal Courtwork Program, Formative Evaluation

1. Introduction

1.1. Background

The Aboriginal Courtworker (ACW) Program began in the early 1960s to address the unique justice challenges of Aboriginal people.  Several studies had revealed the particular challenges faced by Aboriginal persons charged with criminal offences: a sense of alienation from the administration of justice in Canada, a feeling of futility, and a limited knowledge of their rights and obligations, of court procedures and of the resources available to them.  Additionally, officials often failed to understand Aboriginal people and issues.  Federal financial support began in 1969 and by 1978; the ACW Program was an ongoing cost-shared program between the federal and provincial/territorial governments.  In 1987; the program began providing services to Aboriginal youth.  ACW programs currently operate in eight provinces (all but P.E.I. and New Brunswick) and in all three territories.

1.2. Description of the Aboriginal Courtwork (ACW) Program

1.2.1. Program Objectives

The ACW Program seeks to ensure that Aboriginal people charged with criminal offences receive fair, equitable, and culturally sensitive treatment by the criminal justice system. This objective is achieved by: 

Aboriginal courtworkers prepare and accompany the accused to court and, as “Friends of the Court,” also provide critical background information on the accused, make the court aware of alternative measures and options available in the Aboriginal community, and ensure that the accused comprehends the court process.  Furthermore, courtworkers mobilize the community to become more aware of and address emerging Aboriginal justice issues.

1.2.2. Delivery Approach

In the jurisdictions with ACW programs, all Aboriginal (First Nation, Inuit or Métis) people in conflict with the law are eligible for courtworker services regardless of status, age or residency.

In all but three jurisdictions, courtworker services are provided by Aboriginal Service Delivery Agencies (SDAs) under agreement or contract with the provincial/territorial government. Currently over 200 Aboriginal courtworkers are employed by 20 different SDAs across Canada. In Manitoba courtworkers are provincial employees, in NWT they are employees of the NWT Legal Services Board and the Government of Northwest Territories and in Nunavut they are currently hired by the legal aid clinics.  (This arrangement may change now that Nunavut has a courtworker coordinator.)

Because Aboriginal courtworkers are uniquely placed in the justice system and in their communities, they are becoming increasingly involved in community-based approaches and in working with service partners to address the needs of their clients.  The role of courtworkers increasingly involves:

1.2.3. Governance

The ACW program is delivered through the collaboration of service delivery agencies (SDAs), provinces/territories (P/Ts), the Department of Justice and a Tripartite Working Group.

Service delivery agencies provide direct services to the Aboriginal accused through annual contracts with provincial and territorial governments, and funds are based on a schedule of eligible costs. In all but three jurisdictions, SDAs are Aboriginal organizations that are accountable to their communities and, where applicable, to their boards of directors and Aboriginal government organizations.

Canada and the P/T governments enter into multi-year contribution agreements to support the ACW Program. Under these agreements each P/T ministry establishes the overall framework for the ACW program within its jurisdiction.  Generally, the court services divisions within the P/T justice ministries administer the program, and some jurisdictions include the ministry responsible for Aboriginal affairs to ensure there is a consistent approach to services available to Aboriginal people. 

The P/T governments are responsible for:

The territorial Access to Justice Services Agreements that integrate federal funding in support of Aboriginal Courtwork as well as Legal Aid and Public Legal Education and information services, give the territories the flexibility to allocate federal funds among the eligible services as they see fit.  In return, the territories agree to maintain certain minimum levels of service in providing services to eligible persons.  Each territorial government agrees to provide the Department with annual program reports, policies, and procedures that show its programs are consistent with the agreement. Aboriginal courtworkers in the territories also handle civil and family matters, although the nature of that work varies by jurisdiction. 

The Department of Justice Programs Branch administers the ACW Program by entering into contribution agreements that include how much will be contributed to each province or territory and the timing of the payments.  

The ACW Program is guided by the Tripartite Working Group (TWG), which serves as a forum to address program and operational issues. The TWG reports as needed to the committee of Federal/Provincial/Territorial (F/P/T) Deputy Ministers responsible for Justice.  The TWG, co-chaired by one federal and one provincial/territorial representative, comprises up to three federal representatives and one provincial/territorial official and one SDA representative from each jurisdiction.  Each year the TWG meets in person at least once and holds at least three national teleconferences.  The mandate of the TWG is presented in Annex B.

1.2.4. Resources

The program is cost-shared with provincial governments through contribution agreements.  In 2005-06, federal funding totaled $5.5 million.  In 2006-07, the total federal funding will also be $5.5 million.

1.2.5. Program Logic

The program activities and outcomes are delineated below. Annex A contains two logic models developed for the ACW Program.  The first, from the Results-Based Management and Accountability Framework (RMAF), is for the federal and P/T governments.  It encompasses the TWG activities, outputs and outcomes.  The second, from the Performance Measurement and Reporting Strategy Guide, is used by the SDAs and courtworkers and includes the details of their activities.  It is a national logic model that was created in collaboration with all program stakeholder groups.

Activities
Immediate Outcomes
Intermediate Outcomes
Ultimate Outcome

Aboriginal accused receive fair, equitable and culturally sensitive treatment before the courts:  By ensuring that the Aboriginal accused understands the process and that court officials respond appropriately, it is anticipated that the accused will receive fair, equitable and culturally sensitive treatment before the courts.

1.3. Evaluation Purpose and Scope

This evaluation is a Treasury Board Secretariat requirement. The RMAF developed for the ACW program in 2003-04 noted that a comprehensive evaluation of the ACW program had not been done since 1985 and indicated that a summative evaluation would be completed in 2007-08.

The purpose of this formative evaluation was to determine if the program is being implemented as intended and whether the performance information required for the summative evaluation is being collected. The primary audiences for this study are the Department of Justice and the TWG. 

The findings of this national formative evaluation will be useful not only for the DOJ managers of the ACW program, but also for all the program partners, including the provinces, territories, Service Delivery Agencies and courtworkers.

1.4. Evaluation Issues

The key evaluation issues were:

The evaluation issues and questions are shown in Annex C.

1.5. Evaluation Methodology

The evaluation methods included a document review, in-depth interviews with DOJ representatives, representatives of P/T governments, SDA representatives, community justice officials with a stake in the ACW program and a survey of courtworkers.  Table 1 summarizes the evaluation methodology.  Although a data file review was initially planned, data was not available from the P/Ts within the timeframe of the evaluation.

The timeframe for this evaluation was as follows:

An Evaluation Advisory Committee was created to provide advice on the evaluation process and data collection.  The committee consisted of federal, provincial/territorial and SDA representatives.

Table 1 – Evaluation Methodology
Method Description Sampling
Document Review Thorough review of documents from the DOJ, the P/T governments, the SDAs, the Aboriginal courtworkers and others as appropriate.  Included were background documents, decision records, agreements, progress reports, meeting minutes, and all other relevant documents. The document review served three key purposes:
  • helped develop a better understanding of the program and its environment;
  • provided background information on issues; and
  • provided direct evidence to decide on some issues
  
In-depth Interviews Telephone interviews were completed with 31 individuals.  Interview guides were sent in advance of the scheduled interviews.  The interviews were between 30 minutes and three hours in length, and each was completed in the respondent's choice of official language. Because of the small number of potential interviews, limited sampling was required.  The specific sample sizes for each line of evidence were:
  • 3 DOJ representatives (total universe of 4; purposive sampling of most knowledgeable / experienced)[1]
  • 10 P/T representatives (total universe of 11; 1 refusal.
  • 13 SDAs (total universe of 15)
  • 5 community justice officials (total universe unknown, sampling frame of 25; sampling methodology, stratified – maximum of 1 per P/T, random)
Survey of Courtworkers A total of 130 courtworkers who are currently involved in the ACW program were surveyed.  Two completed the survey by mail or fax; all other interviews were completed by telephone in the official language of respondent's choice.  The interview was approximately 15 minutes in length.  The survey was voluntary, and all questions were optional. The survey questionnaire was pre-tested with 8 courtworkers prior to the actual survey, and was changed as a result of the test; therefore, these responses were excluded from the analysis. An attempt was made to interview all 184 courtworkers identified.  Responses were as follows:
  • 133 completed interviews (72.3%)
  • 8 used in pre-test (4.3%)
  • 42 could not be reached during the survey period (22.9%)
  • 1 no time (refusal) (0.5%)
Response rates were high thereby reducing the risk of non-response bias.  Additionally, the sample size provides results that are statistically reliable (to within +/- 4.7%, 19 times out of 20).

The approaches and sample sizes provided sufficient evidence to reach a conclusion on most issues.  However, the evaluation methodology had some limitations.  For example, data was not available for analysis because P/T performance reports were not required until December 31, 2006.  This meant that a number of the evaluation questions could not be addressed. These included questions on client satisfaction and the intermediate outcomes of the program. Additionally, some courtworkers surveyed had been involved with the program for a very limited time, and only a very small number of community justice officials were interviewed.

[1] The responses of all three DOJ representatives interviewed are aggregated and reported as DOJ to prevent attribution to particular DOJ officials.