Evaluation of the Integrated Market Enforcement Teams Reserve Fund

3. Methodology

This section provides a brief description of the methodology used to evaluate the IMET Reserve Fund.

3.1. Evaluation Approach

This evaluation focuses on the set of funding activities that the Department of Justice Canada undertakes in order to manage the Reserve Fund. The Department’s IMET Reserve Fund mandate is to effectively manage the Fund by providing Provincial Attorneys General with adequate support and resources. The evaluation assessed the extent to which these activities have been successfully carried out, which in turn provides an assessment of the extent to which these activities have contributed to the Fund’s expected outcomes.

In order to successfully address the evaluation issues and questions covered by the evaluation, some existing data and information collected throughout the program’s implementation and ongoing management have been used. Additional evidence was collected through a document review and key informant interviews. The next subsection provides further details on the implementation of these research methods.

Both the data collection and analysis conducted as part of this evaluation align with the overall framework provided by the federal government’s Policy on Evaluation, which expects evaluation to support ongoing accountability, inform government decisions on resource allocation, and support the ongoing management and improvement of the program. Footnote 2

All research activities undertaken as part of this evaluation were administered in accordance with normal practices in the field of program evaluation, including the guidelines provided in the Code of Ethics and the Evaluation Standards of the Canadian Evaluation Society. Footnote 3

3.2. Research Methods

In order to address the questions included in the evaluation matrix (see Appendix A), the evaluation included a document review and key informant interviews.

3.2.1. Document review

A systematic review of relevant information related to the IMET Reserve Fund was conducted. The goal of the document review was to support a thorough understanding of the Fund, and to address a number of the evaluation questions, particularly those pertaining to the Fund’s relevance.

Documents consulted include internal material such as Fund-specific information, performance information, planning documents, and Terms and Conditions. Publicly available documentation was also consulted, including Speeches from the Throne, Departmental Performance Reports, and Reports on Plans and Priorities.

3.2.2. Key informant interviews

Key informant interviews were conducted to provide additional insights on the funding activities undertaken by the IMET Reserve Fund and to address evaluation questions related to the relevance and performance of the program. A total of seven individuals were interviewed, including two Justice Canada representatives, two IMET Working Group members, one subject matter expert, and two Provincial Attorneys General prosecutors from jurisdictions which received funding from the Reserve Fund.

These interviews were conducted in person and by telephone using a structured interview guide (Appendix B), which key informants received in advance of the interview.

Case Studies

As part of the case studies, numerous documentation such as internal emails and correspondence were reviewed, as well as court documents filed by Crowns and the defendants. Discussions with departmental officials and key informants were also important to understand the complexity of these cases.

3.3. Methodological Limitations

3.3.1. Broad outcomes of the IMET Reserve Fund

The main methodological limitation was that the broad nature of the intended outcomes of the Fund made it difficult to measure whether these had been achieved. For example, respondents believed there was insufficient evidence to determine whether the IMET Reserve Fund has contributed to deterrence of offenders, reduced occurrences of serious CMF, or increased Canadian and international investor confidence in the integrity of Canada’s capital markets.

3.3.2. Similar responses from respondents

The interviews with key informants have the possibility of self-reported response bias. This occurs when individuals are reporting on their own activities and answer questions with the desire to affect outcomes. In order to mitigate these limitations, triangulation was used to check findings against other sources and perspectives. The evaluation was able to demonstrate how the Reserve Fund was spent and assessed results in general terms, by using evidence from the document review, file review, case studies and the views expressed by key informants.

3.3.3. Limited amount of IMET-related statistics for the evaluation period

Statistics on the overall cost of IMET-generated cases were limited and covered longer periods than the evaluation period. Documentation referred to the high cost to investigate, lay charges and eventually prosecute IMET cases, but these yearly specific amounts were not available for the evaluation period. Having these statistics would have provided information on the percentage the Reserve Fund represents in relation to the total cost of IMET prosecutions.