Justice Federal Victims Strategy Evaluation
3. Methodology
3.1. Issues and Questions Addressed in the Evaluation
The five broad issues (see Table 2) addressed in the evaluation provide a framework for evaluating the direct, indirect and ultimate outcomes of the Justice FVS.
| Evaluation Issue | Evaluation Question |
|---|---|
| Relevance | |
| Continued Need for Program | To what extent is there a continuing need for the Justice FVS? |
| To what extent do the core activities of the Justice FVS continue to be relevant? | |
| Alignment with Government Priorities | To what extent are the activities of the Justice FVS aligned with 1. Government of Canada priorities and 2. Strategic priorities of the Department of Justice? |
| Alignment with Federal Roles and Responsibilities | To what extent do the activities of the Justice FVS align with federal roles and responsibilities? |
| Performance | |
| Achievement of Expected Outcomes | To what extent has the Justice FVS contributed to strengthened federal leadership, legislation, policy and programs responsive to victim issues? |
| To what extent has the Justice FVS enhanced capacity for the delivery of appropriate, responsive victim services? | |
| To what extent has the Justice FVS contributed to increased awareness and knowledge of victim issues, legislation and available services? | |
| To what extent has the Justice FVS contributed to expanding the scope and reach of victim services? | |
| To what extent has the Justice FVS contributed to reducing hardship for victims of crime? | |
| To what extent has the Justice FVS contributed to a more effective voice for victims in the criminal justice system? | |
| Demonstration of Efficiency and Economy | Are there any other ways to deliver the Justice FVS that would improve its performance, both the achievement of objectives and/or to increase its efficiency? |
| What resources are being used? How did the resource level affect the results achieved? | |
| Is the Victims Fund being administered efficiently? | |
| Is the allocation of resources appropriate to ensure the outcomes are being achieved? | |
3.2 Approach
The evaluation methodology consisted of eight lines of evidence including a document and literature review, key informant interviews, financial assistance surveys, Victims Fund file review, Victims Fund case studies, policy case studies, directed studies and a study of resource utilization.
The use of multiple data sources allowed for the triangulation of findings to more clearly define areas of consensus. The directed studies also allowed for additional data to be collected over a longer period of time and on a broader range of topics to ensure a more complete picture of the Justice FVS activities and their impacts, rather than solely focusing on one period of time of data collection for maximizing confidence in the evaluation findings.
A departmental Evaluation Advisory Committee (EAC) with representation from PCVI, IAID and RSD provided technical assistance throughout the evaluation, and contract resources were used to assist with some of the data collection. The Department’s Evaluation Division was responsible for some of the data collection, analysis of all of the lines of evidence and reporting.
3.2.1. Document and Literature Review
The document and literature review consisted of reviewing secondary documents including research reports, event evaluation surveys, government documents, policy documents, communications materials, announcements, analysis of media analysis and administrative data. The EAC identified relevant documents to be included in this review for the time period ranging from April 2010 to March 2015.
3.2.2. Key informant interviews
Key informant interviews addressed the majority of the evaluation questions and were a principal line of evidence. In-person individual and group interviews as well as telephone interviews were conducted with key informants (n=36) identified by the EAC. The distribution of key informants is shown in Table 3 (below). Interview guides were tailored for each respondent group (see Appendix A). The following table provides a summary of the interviews conducted.
| Key Informant Group | Number Interviewed |
|---|---|
| PCVI | 7 |
| IAID | 8 |
| RSD | 1 |
| Communications Branch | 1 |
| FPTWG on Victims of Crime | 19 |
| Total Interviewed | 36 |
3.2.3. Victims Fund File Review
A sampleFootnote 17 of Victims Fund projects funded between 2010 and 2015 (n=91 or 9%) was reviewed. These files included all funding agreements that were negotiated with the provinces and territories, TLOF projects, MMAW projects, and a sample of NGO projects that were funded between 2010 and 2015. Table 4 provides a summary of the types of files reviewed.
| Victims Fund File Type | Number of Files |
|---|---|
| Provincial and Territorial Projects | 27 |
| TLOF Projects | 10 |
| NGO Projects | 40 |
| MMAW Projects | 14 |
| Total Files | 91 |
The following three documents were reviewed for each file: PCVI Funding Proposal Review Form, Grant and Contribution Approval Document, and the Project Summary Report. A file review template (see Appendix B) was used to ensure consistent information was collected across the different types of files.
3.2.4. Victims Fund Case Studies
In addition to reviewing project files, a sample of NGO projects (n=16) and MMAW projects (n=5) were included as case studies. The case studies involved a more detailed review of the files, as well as a telephone interview with a representative from the projectFootnote 18 (n=23). Table 5 provides a summary of the case studies conducted.
| Case Studies | Number of NGO Projects | Number of MMAW Projects |
|---|---|---|
| National Initiative | 2 | |
| Pan-Territorial Initiative (Yukon, Northwest Territories, Nunavut) | 1 | |
| British Columbia | 2 | 1 |
| Alberta | 1 | |
| Saskatchewan | 1 | |
| Manitoba | 2 | 2 |
| Ontario | 4 | |
| Quebec | 3 | |
| Prince Edward Island | 1 | |
| Yukon | 1 | |
| Total Projects | 16 | 5 |
Two different interview guides were developed for the case study interviews (see Appendices C and D).
3.2.5. Victims Fund Financial Assistance Surveys
Between 2010 and 2015, a total of 655Footnote 19 completed Victims Fund financial assistance surveys were returned to the Evaluation Division (see Table 6 below). These surveys were a key line of evidence for the evaluation, since they are the only opportunity for victims and their support persons to provide feedback directly to the Department on the impact of the financial assistance they received.
| Financial Assistance Survey | Total Received |
|---|---|
| Financial assistance for victims to attend a PBC hearing | 511 |
| Financial assistance for a support person to attend a PBC hearing with a victim | 101 |
| Financial assistance for Canadians victimized abroad | 43 |
| Total | 655 |
The different Financial Assistance Survey questionnaires are included in Appendix E.
3.2.6. Policy Case Studies
To ensure evaluation coverage of the policy work undertaken as part of the Justice FVS between 2010 and 2015, two policy case studies were completed. These included examining the process used in the development of the CVBR through key informant interviews (see Table 7 below) and the 2010 Government of Canada advertising campaign Victims Matter through a document review. The data collection instruments used for the policy case studies can be found in Appendices F and G.
| CVBR Key Informant Groups | Participated in a Group Interview | Participated in an Individual Interview | Number Interviewed |
|---|---|---|---|
| PCVI | 5 | 5 | |
| Criminal Law Policy Section | 3 | 1 | 4 |
| Communications Branch | 1 | 1 | |
| RSD | 1 | 1 | |
| Family Children and Youth Section | 1 | 1 | |
| Human Rights Law Section | 1 | 1 | |
| Public Prosecution Service Canada | 1 | 1 | |
| PSC | 1 | 1 | |
| Total | 9 | 6 | 15 |
3.2.7. Directed Studies
In preparation for the Justice FVS evaluation, a number of directed studies were undertaken between 2012 and 2015 to allow for a more in-depth examination of some of the priority areas of the Justice FVS. The results of these studies were used as primary lines of evidence for the evaluation. Table 8 summarizes the methodologies used for the directed studies.
| Directed Study | Methodology Used |
|---|---|
| Survey of Criminal Justice Professionals for the Evaluation of the FVS |
|
| Public Awareness: Victims Week (2012) |
|
| Impact of Technology on Victims of Crime |
|
| Understanding the Impact of CACs in CanadaFootnote 20 |
|
3.2.8. Resource Utilization
The evaluation included a number of questions in the departmental key informant interview guides to gain a greater understanding of how resources were used and whether there are ways to increase efficiency in how the Justice FVS is delivered. In addition, as per the Treasury Board Secretariat’s (TBS) guidelines in the Directive on the Evaluation Function (2009), an analysis of financial dataFootnote 21 collected from all responsibility centres was undertaken. This data was used for an analysis of resource utilization through three perspectives: economy, operational efficiency and allocative efficiency. The results of this analysis along with the results of the key informant interview questions are reported together to address TBS Directive on the Evaluation Function Core Issue #5, the Demonstration of Efficiency and Economy.
3.3. Methodological Limitations and Mitigation Strategy
The methodological limitations of the evaluation, as well as the strategies used to mitigate them, are outlined below by line of evidence.
Key Informant Interviews
The key informants interviewed during the evaluation and various directed studies were selected based on their involvement and knowledge of the Justice FVS. Since the key informants were identified by representatives from the EAC who have a vested interest in the outcome of the evaluation, there is a potential for selection bias.
There is also the possibility of self-reported response bias, which occurs when individuals are reporting on their own activities and may therefore want to portray themselves in a positive light.
The potential for selection and self-reported response bias was mitigated by the triangulation of multiple lines of evidence to verify findings against other sources and perspectives. This reduced concern that the evaluation findings were based on a single method or source.
Victims Fund File Review
A total of 1059 projects were funded between 2010 and 2015 through the Victims Fund. Although it was not possible to review all files, a sample of 297Footnote 22 (28%) Victim Fund files were included in the directed studies and the evaluation. The files reviewed during the evaluation and the directed studies were selected with input from the EAC, which has the potential for selection bias. To mitigate this limitation, all provincial and territorial projects, TLOF and MMAW files were included in the file review, and samples of Victims Week, CAC and NGO projects were done in a systematic way to ensure representation of the different jurisdictions and types of projects as randomly as possible.
The data available on file for the projects reviewed also varied and in some situations was incomplete. Therefore the use of the file review data is limited to providing descriptive information on the types of projects funded.
Victims Fund Financial Assistance Surveys
The Victims Fund financial assistance surveys were mailed to victims, with completed surveys returned to the Department. The use of a mail-out paper survey tends to have a lower response rate than other methods of administering a survey (e.g., online, in-person or by e-mail). The Victims Fund financial assistance surveys between 2010 and 2015 had a response rate of 29%. Although this is an accepted response rate for a mail-out survey, the low response rate can have a potential for non-response bias, which can produce results that do not accurately reflect a population’s view or feelings.
Even though it is not possible to generalize the results of the Victims Fund financial assistance surveys to the entire population of victims who received funding, the surveys provide important information on the impact of funding on those who responded and are considered within these parameters in the evaluation.
Resource Utilization
Although a full analysis of the Justice FVS resources was undertaken, the operational efficiency and allocative efficiency analyses were limited. The focus of the operational efficiency analysis was based solely on the Victims Fund, the operational component of the Justice FVS. As for the allocative efficiency analysis, it focused on the one aspect of the Justice FVS for which sufficient data existed: whether providing financial assistance to victims to attend PBC hearings is an appropriate use of resources.
Additional data would need to be collected in the future to allow for a more robust analysis of the allocation of resources under the Justice FVS, including a more thorough analysis of the benefits of financial assistance to victims. This could be done through enhancing the surveys currently used to collect information from victims who have received financial assistance.
- Date modified: