Evaluation of the National Anti-Drug Strategy
2. Evaluation Methodology
2.1. Scope and Approach
The scope of the evaluation included activities conducted under the NADS by the 13 participating departments and agencies from April 1, 2011 to March 31, 2016.
The evaluation issues were aligned with the Treasury Board of Canada’s Policy on Evaluation (2009). An outcome-based evaluation approach was implemented to assess the progress made towards the achievement of the expected outcomes. The evaluation matrix detailed the evaluation strategy and provided consistency in the collection of data to support the evaluation. Given the recent mandate extension to include PDA initiatives that are being implemented over several years, as well as the recent implementation of KES, the evaluation questions and methods for these areas focused on the implementation of activities and progress towards anticipated outcomes.
2.2. Data Collection Methodology
Data for the evaluation was collected using various methods, including a document and file review, a literature review, administrative data analysis, key informant interviews, and two case studies (PDA and KES). In alignment with its internal horizontal engagement protocols, the RCMP’s National Program Evaluation Services retained responsibility for collecting and analyzing all internal RCMP information as well as independently conducting the interviews with its members.
2.2.1. Document and File Review
The purpose of the document review was to systematically extract relevant secondary data that provided evidence for a specific evaluation indicator. In addition to contributing to the body of evaluation evidence, these sources informed the development of the data collection instruments for other lines of evidence, and were useful in positioning the findings from the evaluation within the appropriate context. Most documents were categorized into one of three broad categories: strategy-specific documents, government-wide documents, or department-specific documents. Examples of these three categories of documents are summarized in the table below.
Table 1: Documents Reviewed in this Evaluation
Strategy-specific documents
- Justice Departmental Performance Reports, Horizontal Initiatives reporting, which includes financial reporting
- Minutes of meetings/records of decisions
- Annual performance report
Government-wide documents
- Budgets
- Speeches from the Throne
- Policy documents
- Alternative delivery mechanisms/best practices for implementation and operation of similar initiatives/strategies/ programs
Department-specific documents
- Planning documents
- Performance information
- Results of evaluations undertaken by individual partner departments
- Evaluation reports of funded projects
2.2.2. Literature Review
The literature review supports the incorporation of an external perspective in the evaluation, particularly of relevance issues, as well as the extent to which the Strategy has achieved its intended longer-term outcomes. The literature review encompassed the peer-reviewed and grey literature, focusing on reports related to illicit drugs and PDA, non-federal programs dealing with illicit drug and PDA issues, and international drug studies. Sources from other jurisdictions (including provinces and territories within Canada as well as foreign governments and international agencies, including EU, UN, UK, US and Australia) were included.
2.2.3. Key Informant Interviews
Forty-three key informant interviews and three consultations were conducted: 38 interviews and three consultations with senior officials in Strategy partner departments (see breakdown below) and five interviews with stakeholders from external organizations. In alignment with its internal horizontal engagement protocols, the RCMP’s National Program Evaluation Services retained responsibility for collecting and analyzing all internal RCMP information as well as independently conducting the interviews and consultations with its employees.
The interviews were conducted by phone and followed a semi-structured guide developed and tailored for each respondent type that aligned interview questions with the specific evaluation questions. The results are reported at an aggregate level in the report.
| Partner departments | Completed Interviews |
|---|---|
Health Canada |
11 |
Justice Canada |
5 |
Canada Border Services Agency |
2 |
Canada Revenue Agency |
1 |
Correctional Service Canada |
2 |
Financial Transactions and Reports Analysis Centre of Canada |
1 |
Global Affairs Canada |
2 |
Office of the Director of Public Prosecutions |
2 |
Public Safety |
0 |
Public Services and Procurement Canada |
1 |
Canadian Institutes of Health Research |
3 |
Royal Canadian Mounted Police |
8 + 3 consultations |
Total number of interviews with partner departments |
38+ 3 consultations |
2.2.4. Case Studies
Two case studies were conducted on the PDA investments and the KES. The case studies included document reviews, a focus group on KES issues, and information derived from the key informant interviews.
2.3. Limitations, Challenges and Mitigation Strategies
The evaluation encountered some challenges and limitations that are outlined below:
- Ability to aggregate impacts: Given the wide range of activities, intended outcomes and performance indicators associated with the Strategy, it is difficult to quantify and aggregate program outcomes in a meaningful yet concise way. A variety of qualitative research methods have been used to address this issue.
- Reliance on input from departmental representatives. The ability to obtain direct input from the key targets of the Strategy (e.g., at-risk youth, offenders and Aboriginal people) is limited by the size of the evaluation as well as by privacy rights. This issue has been addressed, in large part, through the extensive document review, particularly evaluation and survey reports. The interviews were held largely with departmental representatives, which could have led to potential bias in responses; however, this was mitigated as much as possible by validating findings through other primary and secondary research.
- Attribution: It is difficult to attribute particular outcomes to the Strategy relative to other resources, programs and trends affecting the target groups. In some cases, the funding under the Strategy was used to expand existing capacity rather than to establish new initiatives or services; as such, it is also difficult to attribute particular impacts to the new resources provided under the Strategy versus previously existing resources. This challenge has been addressed by encouraging key informants to provide specific examples of activities or projects when discussing impacts. In addition, the document review further captured impacts attributable to the Strategy.
- Limited data available on program delivery costs: Individual components provided information on program budgets and expenditures. However, given the range of representatives who may be involved in particular activities and the absence of activity-based costing data, it is often not possible to determine the specific resources dedicated to the Strategy. In turn, this made it more difficult to assess program economy and efficiency.
- Inconsistent performance data: In some instances, the annual performance reports contained inconsistent performance indicators across the Strategy components. This made it difficult to aggregate qualitative data and to present the impacts that have occurred during the evaluation period. To mitigate this challenge, references to single-year accomplishments were included, with a focus on more complete activities as well as balancing those figures with a qualitative assessment of trends and major accomplishments over the five years.
Overall, the study limitations were mitigated as much as possible through the use of multiple lines of evidence and triangulation of data to demonstrate reliability and validity of the findings. The analysis of findings was based on the evaluation matrix (see Appendix 2) and conducted by evaluation questions. The remaining sections of the report are structured according to the key evaluation objectives.
- Date modified: