Special Advocates Program Evaluation

3. Methodology

This section provides a brief description of the methodology used to evaluate the Special Advocates Program.

3.1. Evaluation Approach

The strategy that frames the evaluation of the SAP first relies on a clear delineation of what is being evaluated. As already stated, this evaluation focuses on the set of activities that the Department of Justice Canada undertakes in order to manage the SAP. It purposely excludes any assessment of the actual assistance provided by the special advocates who have been assigned to cases. It is also not meant to be an assessment of the legislative framework provided under IRPA, while recognizing that this legislative framework does constitute an important contextual component. Put simply, IRPA mandates the Department of Justice Canada to establish a list of individuals who may act as special advocates and to ensure that these individuals are provided with adequate support and resources. The evaluation assesses the extent to which these activities have been successfully carried out, which in turn informs an assessment of the extent to which these activities have contributed to the Program’s expected outcomes.

In order to successfully address the evaluation issues and questions covered by the evaluation, some existing data and information collected throughout the Program’s implementation and ongoing management have been used. However, additional evidence was required, which was collected through a document review and key informant interviews. The next subsection provides further details on the implementation of these research methods.

Both the data collection and analysis conducted as part of this evaluation align with the overall framework provided by the federal government’s Policy on Evaluation, which expects the evaluation to support ongoing accountability, inform government decisions on resource allocation, and support the ongoing management and improvement of the Program. Footnote 11

All research activities undertaken as part of this evaluation were administered in accordance with normal practices in the field of program evaluation, including the guidelines provided in  the Code of Ethics and the Evaluation Standards of the Canadian Evaluation Society. Footnote 12

3.2. Research Methods

In order to address the questions included in the evaluation matrix (see Appendix A), the evaluation included a document review and key informant interviews.

3.2.1. Document Review

A systematic review of relevant information related to the SAP was conducted. The goal was to support a thorough understanding of the Program, in addition to addressing a number of evaluation questions.

The list of documents consulted includes program-specific information (performance information, terms and conditions, etc.), relevant legislative provisions, court decisions addressing the role of special advocates, and broader contextual information related to the IRPADivision 9 and the National Security Inadmissibility Initiative.

A key limitation in carrying out the document review was the inability for the evaluators to access top secret documents.

3.2.2. Key Informant Interviews

Key informant interviews were conducted to provide additional insights on activities undertaken by special advocates, and address evaluation questions related to the relevance and performance of the Program. A total of 11 individuals were interviewed, including special advocates, Minister’s counsel, public counsel, and program representatives.

These interviews were conducted using a structured interview guide (included in Appendix B of this report), which key informants received in advance of the interview. With the permission of key informants, interviews were digitally recorded to assist in accurate note taking. Key informants were provided with the opportunity to review these notes.