3. Evaluation methodology

The evaluation was guided by an evaluation matrix, which included evaluation questions, indicators, and data collection methods. The matrix was developed through the evaluation scoping and design process. The methodology for this evaluation included multiple lines of evidence described below.       A list of evaluation questions can be found in Appendix A.

3.1 Case Studies

Five case studies were conducted in advance of the evaluation. The subjects of the case studies were selected in collaboration with representatives of the Program, and focused on best practices, lessons learned, the impact of Program funding, and the benefits and challenges of JPIP’s operations. The findings from these case studies have been incorporated into the evaluation report. Data collection for each case study included interviews with project stakeholders and a document and file review. Case studies were summarized in individual case study reports. The projects selected for case studies are as follows:

3.2 Literature Review

A review of information of the relevant literature was completed to identify trends and issues related to access to justice and provide context for the Program. The literature review included a review of recent academic and research literature on access to justice issues.

3.3 Document and File Review

A review of government and publicly available documents was conducted to respond to the evaluation questions. As well, 56 files of projects funded through various Program components were reviewed, including the application and any interim or final reports submitted.

3.4 Key Informant Interviews

A total of 20 interviews were conducted, including: 17 with funding recipients (three via the JPIP General component, three via Independent Legal Advice/Representation and IPV, five via CTA 50 funding, and six via PLEI) and three with Justice Canada representatives. The 20 interviews include two interviews with those familiar with PLEI funding provided through AJAs with the territories.

3.5 Environmental Scan

After identifying 12 possible federal programs for review, information for six was reviewed and three were selected for deeper exploration, including an interview with each of the program representatives and a document review. The programs selected for the environmental scan included:

3.6 Limitations, Challenges and Mitigation Strategies

The evaluation encountered a few methodological limitations or challenges.

Table 2: Summary of Limitations, Challenges and Mitigations Strategies
Line of Evidence Limitation or Challenge Mitigation Strategy
Key informant interviews and case studies Challenges included potential response biases from the sampling approach (selective, non-random), the voluntary nature of participation, and self-reporting (reporting on own activities). The evaluation used multiple lines of evidence and triangulation to confirm results.
Data review Challenges in accessing project data from the Grants and Contributions Information Management System (GCIMS), particularly for CFPs due to the limited extracting capacities of the information management system. A spreadsheet was created where available GCIMS information for each project was captured manually. As well, the evaluation featured a file review to capture information about target audiences, outputs and outcomes from various sources.
All lines of evidence As expected at the planning stage, limited data on the early impacts of projects funded under the CTA 50 component were available, due to the recency of this funding component. Any specific examples of early results achieved through this component, gathered through interviews or interim reports, has been included in the evaluation as anecdotal evidence, where relevant.