Legal Excellence Program Evaluation, Final Report
3. METHODOLOGY
Three research methods were used to conduct this evaluation: key informant interviews, surveys of the target population, and stakeholder groups and document review.
3.1 Key Informant Interviews
Table 2 shows the distribution of interviews included in the evaluation by respondent category. A total of 62 interviews were conducted involving 58 people[5]. Most of the interviews were conducted in-person between November and December of 2008.
| Respondent Category | Number of interviews |
|---|---|
| Senior managers and HR managers responsible for the LEP and managers who hired LEP participants | 20 |
| LEP coordinators and HR advisors | 15 |
| Immediate supervisors | 12 |
| Lawyers who worked in private firms before coming to the DOJ | 5 |
| University Directors of Career Services | 10 |
| Total | 62 |
3.2 Surveys of the Stakeholder Groups
Three Web-based surveys were carried out for this evaluation during the months of January and February of 2009. The Research and Statistics Division administered these surveys and provided a summary report for each.
The survey populations were third-year law students across the country; current and former DOJ articling students working in the Department; and DOJ managers who either have employed or may employ articling students. In each case, the whole population was surveyed. In the case of the third-year students, this was done in anticipation of a low response rate. In the case of the two internal stakeholder groups, the population is relatively small; therefore, sampling was not necessary.
For the third-year law student survey, all 20 Canadian law schools were approached and 18 collaborated. The survey link was sent to a total 3,342 students. While the response rate (7 percent) was lower than anticipated, it is close to the norm (10 percent) for Web-based surveys. Those students who did respond were conscientious in providing comments or suggestions.
In addition, two other electronic surveys were sent to current and former DOJ articling students and to managers. The survey of managers did not include the 20 managers who were interviewed for this evaluation.
The table below provides a summary of the survey populations and response rates.
| Survey Group | Survey Population | Respondents | Respondent Gender | Respondent Language | Response Rate |
|---|---|---|---|---|---|
| Third-year law students | 3,342 | 219 | F 63% M 37% |
E 61% F 39% |
7% |
| Current and former DOJ articling students | 275 | 139 | F 62% M 38% |
E 73% F 27% |
51% |
| DOJ managers | 88[6] | 27 | n/a | n/a | 31% |
3.2.1 Limitations of the Surveys
Due to certain program realities and constraints beyond our control, several limitations inherent to our approach regarding the surveys have been identified as follows:
- There could be an uneven representation across the regions in the two internal surveys (articling students and managers involved). Since there was no central database for the Program, the names of the survey respondents had to be collected from the LEP personnel across the regions. Due to staff turnover, some regions had difficulty in generating their list, and it is uncertain the extent to which these lists were complete for all regions.
- The uneven participation could be an issue with the survey of third-year law students as well. Participation in the survey was hampered by certain university policies that preclude third parties from approaching students directly. At two universities, students could only find out about the survey if they happened to open a particular newsletter where the survey link was attached. As a result, the response rate from one of these universities was particularly low.
- The results of the student survey may overstate the level of awareness and knowledge of the LEP and the DOJ, as students who responded to the survey may already have been interested in a legal career in the federal public service and were more informed about the Department than the general student population.
- The results from the survey of current and former articling students may present a more favourable view of the LEP and the DOJ, particularly among those who were already retained by the Department.
3.3 Document Review
Although only a very limited number of historical documents and statistics on the Program were available, those documents that were examined provided some contextual information for this evaluation. Included in the review was information on the LEP Website, memos, presentations and the Ontario Regional Office (ORO) survey of articling students. In addition, the evaluation examined a number of key documents on the recently launched DOJ HR initiatives, such as Law Practice Model Support, February 2009, the DOJ Human Resource Management Plan (2007-2010) and the DOJ Employment Equity Progress Report, 2008-2009. The review extended to several published and unpublished studies on the subject of branding the Public Service of Canada, post-secondary recruitment and career choices of the legal profession.
- Date modified: