Evaluation of the Legal Aid Program

Appendix B: Evaluation Matrix

Draft Evaluation Matrix for Legal Aid Program Evaluation
Evaluation Questions Potential Indicators Data Sources
Relevance
Issue 1: Continued need for the Program
Does the Program continue to serve the public interest and need?

Trends in demand overall and by province/territory for Program components:

  • Number of applications for criminal legal aid
  • Number of refugee claims
  • Number of applications for I&R legal aid
  • Number of court orders for funded defence counsel in federal prosecutions (COCFP)
  • Number of Public Security and Anti-terrorism (PSAT) cases
  • Number of applications for civil legal aid in the territories

Profile of who is being served by legal aid (base funding [adult and youth], Immigration and Refugee [I&R], PSAT and COCFP) and Access to Justice Services Agreements (AJAs) legal aid component

  • Available demographic information
  • Stakeholder opinion on challenges faced by legal aid clients in those areas covered by federal funding

Crime rates, incarceration rates, and socio-economic indicators by province/territory, and for marginalized or at-risk groups (e.g., individuals with mental health issues, Indigenous peoples)

Opinions of stakeholders on changing demand for criminal legal aid and civil legal aid in the territories (legal aid trends, policy gaps, emerging issues)

Trends in case complexity and factors that contribute to complexity (criminal legal aid, civil legal aid in the territories, I&R, COCFP, PSAT)

Stakeholder opinions on the public interest and need (individual, system) addressed by the availability of legal aid for each Program component, and any regional/jurisdictional factors that affect the level or type of need
  • Key informant interviews
  • Interviews with justice professionals
  • Interviews with clients
  • Case studies
  • Review of documents, files, data, and literature
Issue 2: Alignment with government priorities
To what extent do the Program’s goals and objectives align with the federal priorities and departmental strategic outcomes? Comparison of federal priorities and departmental strategic outcomes with the goals/objectives of the criminal, I&R, COCFP and PSAT legal aid components of the Program and the AJAs legal aid component
  • Key informant interviews (LAD only)
  • Review of documents, files, data and literature
Issue 3: Alignment with federal roles and responsibilities
Is there an appropriate and necessary role for the federal government in the areas of the legal aid components?

Federal/departmental role based on federal jurisdiction.

Federal role based on international agreements.

Opinions of stakeholders regarding federal/departmental role

  • Key informant interviews
  • Document review
Performance (effectiveness, efficiency and economy)
Issue 4: Achievement of expected outcomes — Immediate outcome (enhance capacity by making legal aid available)
To what extent has the Program made progress toward increasing capacity of the provinces and territories and their legal aid plans to provide and deliver legal aid in the areas receiving federal funding?

Trends in number and proportion of applicants receiving criminal legal aid over time (overall, by province/territory, and by type of service).

Information on stays due to lack of defence counsel in cases under the PSAT and COCFP components.

Amount and proportion of provincial/territorial legal aid budgets for criminal and I&R legal aid that comes from the Program versus other sources, and changes over time.

Amount of funding for COCFP and PSAT legal aid reported by province/territory and legal aid plans.

Opinions of stakeholders on the level of the federal contribution and its effect on the availability of criminal legal aid services (and civil legal aid in the territories).

Stakeholders opinions of consequences for provincial and territorial capacity to serve economically disadvantaged individuals, if the component (criminal, civil in the territories, I&R, COCFP, and PSAT) of the Program did not exist.

Evidence of enhancing capacity through work at the national level, as demonstrated by:

  • information sharing and collaboration efforts, i.e. through the PWG
  • research undertaken and distributed
  • policy papers developed and distributed
  • Key informant interviews
  • Document and file review
Issue 4: Achievement of expected outcomes — Intermediate outcome (support effective implementation and delivery of legal aid)
To what extent has federal funding contributed to the effective provision of legal aid to eligible persons in the areas receiving federal funding?

For base funding:

Trends over time showing changes in criminal legal aid services with respect to:

  • eligibility criteria by province/territory
  • types/levels of service provided by provincial and territorial plans (duty counsel, certificates, summary advice, etc.), including trends in number and percentage of individuals receiving each type/level of service

Response of provincial and territorial plans to federal funding level over time, i.e.:

  • changes in eligibility criteria for criminal legal aid
  • changes in criminal legal aid service delivery
  • changes to other legal aid service delivery (i.e. family/civil) Comparison of the financial eligibility guidelines to other economic measures (e.g., low income cut-offs, consumer price index, minimum wage, average hourly earnings) Comparison of court data for legal aid clients with those who applied but were not financially eligible for criminal legal aid (if possible) or comparison of court data for legal aid clients with non-legal aid clients charged with similar offences. The court data might look at:
  • type of resolution
  • length of time between first appearance and resolution
  • number of appearances

Stakeholder opinion on trends in number and percentage of criminal accused who are unrepresented, and reasons for their lack of representation.

Trends in number of Rowbotham orders (provincial/territorial and federal).

Client experience with legal aid and potential consequences if they had not received legal aid.

Stakeholder opinions of consequences to individuals and the justice system of access/lack of access to legal aid.
  • Key informant interviews
  • Interviews with justice professionals
  • Interviews with clients
  • Case studies
  • Review of documents, files, data, and literature
 

For I&R:

Amount and proportion of provincial and territorial I&R legal aid budgets that come from the Program versus other sources, and changes over time.

Trends in number of individuals served by I&R legal aid (overall and by province/territory).

Current coverage of provincial and territorial I&R legal aid services, i.e.:

  • eligibility criteria by province/territory
  • types/levels of service provided by provincial and territorial plans

Opinions of stakeholders on changing nature of the demand for I&R legal aid.

Opinions of stakeholders on effects of the new refugee determination process on the delivery of I&R legal aid.

Opinions of stakeholders on the level of the federal contribution and its effect on the availability of I&R legal aid services.

Client experience with legal aid and potential consequences if they had not received legal aid.

Stakeholder opinions of consequences to individuals and the justice system of access/lack of access to I&R legal aid.
  • Key informant interviews
  • Interviews with justice professionals
  • Review of documents, files, data and literature
 

For COCFP:

Costs expended for COCFP cases and changes over time.

Trends in number of court orders for funded defence counsel in federal prosecutions (COCFP).

Information on stays due to lack of funded defence counsel in COCFP.

Trends in the number of COCFP cases and percentage of total federal prosecutions over time.

Opinions of stakeholders on the administrative requirements for the COCFP.

Stakeholder opinions of consequences to individuals and the justice system of access/lack of access to COCFP.
  • Key informant interviews
  • Interviews with justice professionals
  • Review of documents, files, data
 

For PSAT:

Costs expended for PSAT and changes over time.

Information on stays due to the absence of legal aid in PSAT cases.

Trends in number of PSAT legal aid cases.

Information on stays due to lack of federally funded defence counsel in PSAT.

Trends in number of cases receiving legal aid in PSAT and percentage of total PSAT cases over time.

Stakeholder opinions of consequences to individuals and the justice system of access/lack of access to PSAT legal aid

  • Key informant interviews
  • Interviews with justice professionals
  • Review of documents and files
 

For AJAs:

Trends in the number of individuals receiving legal aid services over time.

Client experience with legal aid and potential consequences if they had not received legal aid.

Stakeholder opinions of how well the AJAs address the unique access to justice needs of individuals living in the territories

  • Key informant interviews • Interviews with justice professionals
  • Interviews with clients
  • Case studies
  • Review of documents and files
 

For PWG:

Evidence of knowledge application by provincial and territorial legal aid plans (e.g., evidence of uptake and application of knowledge produced by PWG research projects and policy papers).

Opinions/perceptions of stakeholders on the extent to which the PWG supports networking and information sharing and policy development

  • Key informant interviews
  • Review of documents and files
Issue 4: Achievement of expected outcomes — Ultimate outcome (benefits)
To what extent has federal funding of legal aid contributed to a Canada-wide justice system that is efficient, fair, relevant and accessible, and that promotes public confidence in the justice system? Evidence from immediate and intermediate outcomes.
  • Key informant interviews
  • Interviews with justice professionals
  • Interviews with clients
  • Case studies
  • Review of documents, files, data and literature
Issue 5: Demonstration of efficiency and economy
Economy
Were Program resources expended as planned? Planned-to-actual resource use/spending (budget versus expenditures) for the Program and explanation of variance
  • Review of documents, files, data
Operational efficiency
What are the costs of delivering legal aid? What are these costs relative to outputs and outcomes?

Overall cost to deliver the components of the Program (including AJA legal aid)

Overall cost to deliver the components of the Program (including AJA legal aid) Administrative costs of the Program over time

  • Review of documents, files, data (Integrated Financial and Material System)
  • Key informant interviews
Allocative efficiency
Are there more efficient ways of achieving the objectives of the components of the Program (criminal, I&R, PSAT, and COCFP, and legal aid component of AJAs)?

Estimated cost savings resulting from delivery of criminal, I&R, and PSAT legal aid through existing legal aid program (based on rates for COCFP).

Estimated cost of private bar legal representation in criminal, I&R, COCFP and PSAT cases.

Trends in cost per application (with inflation factor).

Trends in cost per client served (with inflation factor).

Opinions of stakeholders on the cost drivers of legal aid and their impact on the legal aid system.

Opinions/perceptions of stakeholders regarding potential efficiencies/alternatives.
  • Key informant interviews
  • Document review
  • File review
To what extent were federal resources used reasonably for the outcomes achieved?

Opinions/perceptions of stakeholders on benefits of federal legal aid funding for clients and the justice system.

Potential savings identified through court data analysis (e.g., reduction in number of appearances, trials, length of time between first appearance and resolution).

Opinions/perceptions of stakeholders on potential effects of higher or lower levels of legal aid funding for clients and the justice system

  • Key informant interviews
  • Interviews with justice professionals
  • Interviews with clients
  • Case studies
  • Data review
Best practices/lessons learned
Are there any best practices or lessons learned in the delivery of federally funded legal aid?

Examples from case studies of best practices and lessons learned.

Opinions/perceptions of stakeholders regarding best practices/lessons learned in terms of governance, policies and program delivery.

  • Key informant interviews
  • Case studies