Evaluation of the Legal Aid Program

3 Evaluation Methodology

To guide the evaluation, an evaluation matrix (evaluation questions, indicators, and data sources) was developed during the evaluation scoping process and further refined with the EWG. The scope of the evaluation focussed on relevance and performance (effectiveness and efficiency) and included the following nine evaluation questions:

Issue #1: Relevance

1.1. Does the Legal Aid Program continue to serve public interest and need?

Issue #2: Effectiveness

2.1. To what extent have the Legal Aid Program components provided services – directly (state-funded counsel) or indirectly (contribution funding to the provinces and territories for criminal and I&R legal aid) – to eligible vulnerable persons, including those who are economically disadvantaged?

2.2 Have emerging issues and the current context impacted legal aid service delivery in the federally funded areas? How have the legal aid systems responded?

2.3 What impact has federal funding had on improving access to legal services for vulnerable populations?

2.4 To what extent have legal aid systems contributed to a fair and efficient justice system?

2.5 In what ways has the Program contributed to collaboration in legal aid policy and program development?

Issue #3: Efficiency

3.1. How efficient is the Legal Aid Program in managing the contribution agreements and state-funded counsel?

3.2 To what extent have the LAD and the PWG contributed to effective and efficient delivery of the Legal Aid Program?

3.3 Considering the legal aid system as a whole, are there more efficient ways of delivering the components of the Program and of meeting its objectives?

Four lines of evidence were used to address the evaluation questions: a review of program and administrative documents and data; key informant interviews; case studies; and journey mapping. Each of these methods is described briefly in the following sub-sections.

3.1 Document and Data Review

The document and data review provided descriptive information on the activities of the Legal Aid Program and legal aid plans, as well as information responding to most evaluation questions. This review was ongoing throughout the life of the project and included the following types of documents:

3.2 Key Informant Interviews

A total of 31 in-depth interviews were conducted with 48 individuals, representing the following groups:

Most of the provincial and territorial government representatives and legal aid plan representatives were members of the PWG or invited to participate in the interview by one of its members. Three key informants opted to provide written responses.

3.3 Case Studies

Three case studies were conducted to highlight and provide further depth to the analysis of key areas of legal aid service delivery, including services provided, challenges, costs, and innovative practices. The case studies also considered how legal aid supports and is impacted by other stakeholders in the justice system. The case studies were selected in consultation with the EWG and the jurisdictions listed in Table 2 agreed to participate. Summaries of the case studies are included in Appendix A.

Table 2. Case Studies
Topic Jurisdiction(s) Number of Interviewees Rationale
I&R legal aid QuebecOntario 35 The case study explored how I&R legal aid is delivered and what the differences are in terms of services and delivery in each selected jurisdiction. The case study considered costs, challenges, and promising practices.
Youth criminal legal aid Yukon 4 The case study considered service innovations and challenges for providing youth criminal legal aid.
Issues and challenges related to remand AlbertaSaskatchewan 22 The case study considered the impact of remand on legal aid, the justice system and the accused, as well as how legal aid plans have approached the issue of remand (e.g., duty counsel services).

3.4 Journey Mapping

Sessions were conducted with three jurisdictions to map the journeys of legal aid clients through the justice system. The mapping of hypothetical legal aid clients’ journeys through the justice system explored how legal aid supports clients at different steps in the process, what options might be available for clients, what steps might be particularly challenging for an accused person, and what might be the impacts of legal aid representation on court efficiency and client outcomes.

The journey mapping process was informed by a EWG meeting devoted to considering the best approach. As a result, it was determined that, rather than establishing a single hypothetical legal aid client for mapping in each jurisdiction, the participating jurisdictions should be given the opportunity to select the types of journeys that would be explored. As a result, the evaluation mapped eight client journeys with three jurisdictions.

Table 3. Journey Mapping
Jurisdictions Journey Map Scenarios Number of Participants
Nova Scotia Adult male African Nova Scotian criminal legal aid client who proceeds through the traditional court process in Halifax or Dartmouth Seven Nova Scotia Legal Aid staff
One male youth criminal legal aid client who proceeds through Restorative Justice (diversion) – rural office Four Nova Scotia Legal Aid staff
One female Indigenous criminal legal aid client who proceeds through a specialized court – Halifax based Six Nova Scotia Legal Aid staff
Manitoba Adult criminal legal aid client in Thompson Three Legal Aid Manitoba staff
Youth criminal legal aid client in Winnipeg Two Legal Aid Manitoba staff
Adult criminal legal aid client in Winnipeg Mental Health Court Three Legal Aid Manitoba staff and one Crown counsel
Northwest Territories Adult criminal legal aid client in a remote community who goes through the traditional court process in a circuit court location Four Northwest Territories Legal Aid Commission staff
Adult criminal legal aid client in Yellowknife who attends the Wellness Court

3.5 Limitations, Challenges, and Mitigation Strategies

The evaluation encountered a few methodological limitations or challenges.

Table 4. Summary of Limitations, Challenges, and Mitigation Strategies
Line of Evidence Limitation or Challenge Mitigation Strategy
Document and data review In regards to the legal aid statistics, for some indicators, there are three years of data (FYs 2016-17 to 2018-19), so an analysis of trends is premature. The evaluation relied on the Legal Aid in Canada reports prepared by Justice Canada’s Research and Statistics Division and the LAD. Prior to FY 2016-17, annual reports were prepared by Statistics Canada and during the transition period (FY 2015-16), no report was prepared. Additionally, this evaluation occurred before the report for FY 2019-20 was available. Where possible, information was gathered from 2019-20 claims submitted by the jurisdictions, some of which were still drafts.
Key informant interviews Challenges included potential response biases from the sampling approach (selective, non-random), the voluntary nature of participation, self-reporting (reporting on own activities). The evaluation used multiple lines of evidence and triangulation to confirm results.
Case studies The case studies were each intended to include two jurisdictions in order to support a comparison in approaches. However, for one case study (youth), ultimately only one jurisdiction volunteered. None needed. While a comparison in approaches was hoped for, the case study was still able to provide examples of youth criminal legal aid services. In addition, the document review included a review of any other noteworthy youth criminal legal aid service delivery methods.
Each case study was to include interviews with legal aid plan representatives and other provincial and territorial justice stakeholders. Legal aid plan representatives did not participate in one case study (remand) due to lack of time or capacity to participate. To the extent possible, the evaluation relied on available relevant documentation to provide the legal aid perspective, although the documentation was limited.
As part of each case study, the evaluation sought to include relevant documents and data. The evaluation was not successful in obtaining data from the provincial and territorial governments or plans for most of the case study jurisdictions. Data was either not available or it was collected but no interviewees were able to provide the data or indicate where the evaluation might be able to obtain the data. To the extent possible, the evaluation relied on publicly available documentation and data.
Journey mapping The purpose of the journey mapping sessions was to seek information related to potential system efficiencies (e.g., the number of appearances, time in court, ability to negotiate release on bail), to which monetary amounts could be attributed if reliable information was available to support that analysis. This type of information was not available. The evaluation used qualitative information to the extent possible.