Justice Leaders of Tomorrow Program
Formative Evaluation of the Pilot

3. METHODOLOGY


3. METHODOLOGY

Three main research methods were used to develop this evaluation: key informant interviews, a survey of the target population and document review. Some evaluation questions were asked in both the interviews and the survey.

3.1. Key informant interviews

The evaluation included interviews with members of all key stakeholder groups:

A total of 32 people were interviewed in November and December 2007.

3.2. Survey of the target population

A Web-based survey was carried out for this evaluation. A questionnaire was sent in November 2007 to a sample of 935 of the 2,288 departmental employees who were eligible for the Program at the time of program recruitment, i.e. from June to September 2006.

The sample comprised:

The Research and Statistics Division administered this on-line survey, to which 202 people (22% of the sample) responded. This response rate is about average for on-line surveys. Table 1 presents the characteristics of the target population, the sample and survey respondents. As can be seen, the survey respondents are representative of the target population in terms of the breakdown by professional group; however, there are higher proportions of employees from regional offices and from designated EE designated groups among survey respondents than among the target population.

Table 1: Characteristics of Survey Sample and Respondents

  Population Sample[1] Respondents
Professional Groups Number Percentage Number Percentage Number Percentage
LA 2,058 90 850 91 182 90
Other Professional Groups 230 10 85 9 20 10
Total 2,288 100 935 100 202 100

  Population Sample[1] Respondents
Member of Designated Groups (self-identified) Number Percentage Number Percentage Number Percentage
Yes 174 8 174 19 46 29
No 2,114 92 761 81 114 71
Total 2,288 100 935 100 160 100

  Population Sample[1] Respondents
Location Number Percentage Number Percentage Number Percentage
NCR 1,336 58 554 59 88 48
Regional Office 952 42 381 41 95 52
Total 2,288 100 935 100 183 100

3.3. Document review

The document review was limited in scope as the JLTP is a new and relatively small program. Nevertheless, the following key documents provide some valuable background for and insights into the Program.

3.4. Limitations

It should be noted that this is a formative evaluation and, as such, it focuses on the implementation of the Program (rationale, activities and outputs) rather than on its outcomes or impact. This is largely due to the fact that the Pilot Program is still being implemented, a process that spans more than two years and that the longer-term outcomes of this Program may take another few years to fully materialize.

It should also be noted that, for the survey, more emphasis was placed on obtaining a high response rate and rich information, especially from certain segments of the targeted population than on the representativeness of the sample. For example, we over-sampled members of EE designated groups and current JLTP participants to make sure that sufficient information important to the evaluation could be collected. As a result, the survey provided important insights on many important aspects of program implementation.

[1]   For confidentiality reasons, the names and characteristics of members of designated groups were not provided to us. The figures in this column are estimates based on the assumption that the proportions of people belonging to the LA group or working in the NCR are the same for members of designated groups and for people who are not members of designated groups.