Evaluation of the Legal Aid Program

3. Methodology

The evaluation made use of multiple lines of evidence in order to support robust findings. The methodology included five lines of evidence: a document, file, data and literature review; key informant interviews; interviews with justice professionals; interviews with clients; and case studies.

The design of the methodology began with a facilitated session with the Evaluation Advisory Group, which included representatives of three areas within the Department: the LAD, the PID, and the Evaluation Division; one provincial representative and three legal aid plan representatives of the PWG also participated. The facilitated session occurred in October 2015 and focused on the evaluation’s approach to outcomes, efficiency and economy.

After the facilitated session, the evaluation matrix (which links the evaluation questions, indicators and lines of evidence) and the data collection instruments were developed with the input of the Evaluation Advisory Group. The evaluation matrix is included in Appendix B and the data collection instruments are in Appendix C.

Each of the evaluation methods is described more fully below. This section also includes a brief discussion of methodological limitations.

3.1. Document, File, Data and Literature Review

Internal program documents were reviewed (e.g., terms and conditions and contribution agreements for the Program and for the AJAs, business plans, final claims, evaluation/research studies) as well as publicly available information, such as the budget speeches, Speeches from the Throne, departmental performance reports, and reports on plans and priorities. The Evaluation Advisory Group also identified relevant documents to be included in this review.

3.2. Key Informant Interviews

The key informant interviews conducted for this evaluation addressed the majority of evaluation questions, and were an important line of evidence in gathering information on the need for the Program, as well as its effectiveness. A list of potential key informants was prepared, and interview guides tailored to each key informant group were developed in consultation with the Evaluation Advisory Group. The evaluation conducted 25 interviews with 47 participants, which include written submissions from the IRB and some legal aid plans. The specific categories of key informants are included in Table 5.

Table 5: Key informant interviews
Category Number of key informants Number of interviews/ written responses
LAD and PID 8 1
Follow-up interviews with the LAD and PID on individual Program components * 3
Members of the PWG representing provinces and territories and legal aid plans 27 22
Representatives of other federal departments and agencies (IRCC, IRB and Public Prosecution Service of Canada) 8 6
Legal aid plan or provincial representatives managing COCFP/PSAT 4 4
Total 47 36

Three members of the LAD and a representative of the PID participated in these additional interviews on specific Program components in which they are involved.

The following scale has been applied to report on the interview findings:

A few = Less than 15% Some = Approximately 25% Many = Approximately 50% Most = Approximately 75% Almost all = More than 85%
Text equivalent

A few = Less than 15%. Some = Approximately 25%. Many = Approximately 50%. Most = Approximately 75%. Almost all = More than 85%.

3.3. Interviews with justice professionals

To incorporate the experience of those who work in the court room, the evaluation included interviews with 34 criminal justice professionals and a focus group/interviews with six lawyers who handle I&R legal aid cases:

3.4. Interviews with Clients

To ensure that the evaluation included the voices of legal aid clients, the evaluation had a target of interviewing 40 legal aid clients. The choice of location was based on the availability of legal aid clients for interviews, and the mode (in-person or telephone) was based on evaluation travel budget considerations. The interview guide was the same for both in-person and telephone interviews. A total of 38 individuals who had applied for legal aid were interviewed as part of the evaluation.

The clients interviewed included:

Of the 38 clients, 31 of the clients had criminal law matters; four clients had I&R matters; two had family law issues; and one had both family and criminal law issues. Footnote 9

In addition, the Program provided information on interviews conducted with 15 legal aid clients of Aboriginal Legal Services in Toronto.

The interviews focused on obtaining information about experiences with legal aid so most of these were with legal aid clients. Some interviews were conducted with individuals who applied but were not eligible for legal aid in order to gather information on how individuals who do not have legal aid handle their matters. Footnote 10

3.5. Case Studies

Ten case studies were conducted to highlight a variety of legal aid service delivery methods, including those considered to be innovations or promising practices. The case studies were chosen in consultation with the legal aid plans. The case studies included a review of relevant documents, as well as interviews with legal aid plan staff/lawyers involved in the service delivery highlighted, other relevant stakeholders, and where possible, clients. Across the case studies, 68 stakeholders and two clients were interviewed. A brief description of the case studies is provided below. More detailed descriptions are included in Appendix D.

3.6. Limitations

The evaluation faced a few methodological limitations. These are listed below by line of evidence.

Review of documents and data. The Evaluation Division and the LAD explored whether court data could be made available to show the impact of unrepresented accused on the justice system and the system efficiencies provided by having counsel. Although the LAD explored the possibility of accessing court data, the ability to obtain the necessary permissions was not possible during the timelines of the evaluation. In addition, court data may provide a record of whether an accused person had representation at certain points in his criminal case, but not whether the accused person was a legal aid client. The ability of court data to demonstrate outcomes for legal aid clients is a complex issue and access to court data requires sufficient lead time. The Department should consider the use of court data well in advance of the next evaluation in order for it to be available for the evaluation.

Interviews with key informants and case study stakeholders. Interview findings are potentially affected by self-reported response bias and strategic response bias. Self-reported response bias occurs when individuals are reporting on their own activities and so may want to portray themselves in the best light. Strategic response bias occurs when the participants answer questions with the desire to affect outcomes.

Interviews with clients. The evaluation came close to meeting its target of 40 client interviews (38 clients were interviewed), but many of the clients had ongoing legal matters, which limited their ability to respond to evaluation questions. Recruiting legal aid clients for the in-person interviews was very time intensive for the legal aid plans.

Performance data are not being consistently recorded and some outcomes/indicators are not well defined. This remains a limitation from the last evaluation, which noted that the Program needed to improve performance measurement by developing performance measures agreed to by the PWG, and potentially provide federal funding to support ongoing data collection and analysis. As discussed in this report, the PWG is working on determining key performance metrics for which the jurisdictions will collect comparable data. These measures will hopefully minimize this issue for the next evaluation.

Mitigation strategy. Two mitigation strategies were used for the above methodological limitations: