Legal Risk Management in the Department of Justice

3.0. METHODOLOGY


3. METHODOLOGY

The evaluation was comprised of six lines of evidence and was guided by an evaluation framework. Data collection instruments were used for the evaluation. The methodological approach and the instruments were developed in consultation with the Evaluation Advisory Committee (the Committee), which included representatives from headquarters, regions, and LSUs and oversaw the evaluation from design through to final reporting. This section of the report describes each of the lines of evidence.

3.1. Document review

The document review included several types of documents and data:

A review of program documents

Primary sources of information, provided by the Law Practice Management Directorate, were reviewed. The list of these documents included presentation decks on LRM; the Business Case for LRM; materials on LRM from the Department intranet; the LRM AFGS; internal Department (portfolio/sector/region/legal services unit) documents on LRM practices; and materials from the 2007 Vancouver workshop.

An analysis of iCase data reports

The evaluation used information from iCase, a national, Web-based case management and timekeeping system developed by the Department. iCase includes a risk evaluation page that captures information on legal risk provided by counsel. The evaluation included two forms of iCase information:

  • The results of iCase audits. These audits are conducted by the National Business Application Analyst Team and consider compliance with standards established in each Portfolio's iCase Business Standards Manual. The evaluation considered the results of audits conducted during fiscal years 2005-2006 and 2006-2007 (quarterly) and 2007-2008 (three cycles of which two were completed). The results include litigation and advisory files only, and review whether iCase data is inputted for mandatory LRM fields.[10] The audits do not consider the reliability of the information.

  • A review of closed files from 2005-2006 and 2006-2007. This review focused on whether counsel had assigned numeric risk assessment levels to the files.

Results for relevant questions from the Justice Services Satisfaction Survey

The Client Feedback Survey was developed by the Department's Office of Strategic Planning and Performance Management in partnership with Statistics Canada. Relevant results from the survey were provided for this evaluation. This survey was administered between July 2006 and February 2008 on a portfolio by portfolio basis (except for Tax Law). Employees at EX minus 1 and above were invited to participate in the survey. The response rate for the survey was 31%.

A focused literature review

There is no publicly available literature on LRM in the public sector outside of that found on the Department's and Treasury Board's websites. Therefore, the evaluation focused its review of literature on best practices with respect to the principles and implementation of public sector risk management more generally. Even then, the literature on best practices is scant.

3.2. Key informant interviews

During the months of February to March 2008, a total of 48 representatives from Department headquarters (n=28), regional offices (n=8), and LSUs (n=12) were interviewed.[11] The interviews were conducted either in-person or over the phone in the preferred official language of the key informant. All key informants received an interview guide prior to the interview itself. A representative of the LPMD or the Evaluation Division attended almost all of the interviews.

3.3. Legal counsel survey

The Department hosted a legal counsel survey on its intranet, and all 2,182 Justice counsel were invited to participate in the survey. The survey was online from March 5-March 20. To encourage responses, two reminder emails were sent to counsel. A total of 636 counsel completed the survey for a response rate of 29%. Survey data were analyzed using the Statistical Package for the Social Sciences (SPSS) program. Table 6 provides a profile of survey respondents and shows that generally respondents were representative of the population of DOJ counsel, although more counsel with less than one year experience responded to the survey.

Table 6: Profile of survey respondents

What is the classification level of the position you currently occupy?
Characteristics Population (%) Survey respondents (%)
LA - 01 15% 15%
LA - 2A 55% 54%
LA - 2B 19% 20%
LA - 3A 8% 8%
LA - 3B 3% 2%
LA - 3C <1% <1%
Senior Management (above LA-3C level)* <1% --
No response -- <1%
Total 101% 101%
When did you first join the Department?
Characteristics Population (%) Survey respondents (%)
Less than one year ago 2% 8%
Between 1 and 5 years ago 27% 22%
Between 6 and 10 years ago 35% 33%
More than 10 years ago 36% 38%
Total 100% 101%
Where do you currently work?
Characteristics Population (%) Survey respondents (%)
Headquarters 26% 25%
Regional office 40% 40%
LSU/Satellite office 34% 35%
Total 100% 100%
Are you in a management position?
Characteristics Population (%) Survey respondents (%)
Yes Not available 15%
No Not available 85%
Don't know/No response Not available <1%
Total Not available >100%

Note: Totals do not sum to 100% due to rounding.

* Option not offered on survey.

Counsel answered the survey questions relevant to the type of legal work they do most often that involves the management of legal risk. The type of legal work was divided into the following categories (largely based on the main legal activities of the Department described on page 13): litigation; litigation support (assisting the litigator and the client with a case that is in litigation, including the provision of legal advice); legal advice (providing legal opinion or advice on a non-litigation matter when requested by a client department or agency); policy and program development; and legislative and regulatory services. Where appropriate, results are provided by these categories.

3.4. Case studies

The evaluation included four case studies to provide useful detail on how LRM has been integrated into operations at different levels of the Department. Because the file review (see Section 3.5) encompasses a more detailed study of litigation and advisory/legislative files, the case studies have a broader operational focus and examine how LRM is integrated into different regional/portfolio/LSU settings. The case studies were chosen in consultation with the Committee.

Each case study included interviews and a review of relevant documents, such as operational documents to support LRM including guidelines, procedures, or templates. The research team conducted interviews primarily on-site over one or two days. A few follow-up interviews were conducted by telephone. Interviews were a mixture of individual and small group interviews. Interview participants for each case study were determined in consultation with a key contact person who was typically the person responsible for LRM in the LSU or regional office. Those interviewed were contacted in advance and provided with the interview guide. The number of individuals interviewed for each case study ranged from 5 to 20 and a total of 42 individuals were interviewed.

The evaluation could not conduct a cost comparison of the LRM activities for the case studies because these costs are not tracked. Each counsel working on high risk files spends a portion of his or her time on LRM and various counsel are involved in LRM as part of committee work. The evaluation is able to provide some estimates of time spent by counsel and staff who do a considerable amount of LRM work. These individuals may not be exclusively dedicated to LRM, but some of their responsibilities include ongoing support of the LRM structures in their offices. Table 7 provides the estimates of time spent on LRM activities that were provided in case study interviews. The table does not include the time devoted to LRM by other litigation and/or risk-related committee members or counsel working on files.

Table 7: Staff resources devoted to LRM - Case studies

BCRO
Regional Offices Estimated percentage of time on LRM
LRM and Policy Coordinator 50-100% depending on current file load
Consultation and Accommodation Coordinator 50-100% depending on current file load
Chair of the Regional Law and Litigation Committee 25%
Communications Advisor Currently on leave
Paralegal 60-75%
Legal assistant 60-75%
ORO
Regional Offices Estimated percentage of time on LRM
Special Advisor to the Senior Regional Director 20-40%
Chair of the Litigation Committee 10-30%
Paralegal 80-90%
LSUs - DFO
Regional Offices Estimated percentage of time on LRM
Senior General Counsel 10-20%
LRM Coordinator/Senior Counsel Legal Risk Management <50% depending on current file load
Paralegal 90%
Legal assistant 50%
LSUs - PCH
Regional Offices Estimated percentage of time on LRM
Director and General Counsel 5%
Paralegal 10%

3.5. Review of closed files

The file review provides a detailed qualitative analysis of the LRM process through a study of a selection of closed litigation, advisory, legislative services, and policy files.

In all, 69 files closed between 2003 and 2008 were reviewed: 51 litigation files; 7 legislative drafting files; 6 advisory files; and 5 policy files.[12] Of the litigation files, 15 files were appeals or other proceedings that relate to 6 separate cases. The litigation files were identified through iCase and were mainly high risk files (risk level 7-9), as the evaluation focused on the use of processes and tools in high risk cases. A smaller selection of risk level 6 files was also included. The non-litigation files were primarily identified with the assistance of counsel. The files were chosen to represent different regions, portfolios, and level of court (if applicable).[13] Consideration was also given to other factors such as complexity of the file, length of time it was open, type of proceeding, and final outcome to the Crown.

Counsel with the Department conducted the file review using a template developed for the evaluation. The data collected was entered into the Computer Assisted Telephone Interviewing (CATI) program, before transferring it to SPSS for analysis.

Table 8 shows the distribution of the closed files by organizational units, portfolios and sectors.

Table 8: File review distribution (n=69)
Organizational Unit Portfolios Sector Total
Aboriginal BRL CIPS
British Columbia 3 Litigation
1 Advisory		 4 Litigation
1 Advisory		 8 Litigation
1 Advisory		   15 Litigation
3 Advisory
Prairies 2 Litigation
1 Advisory		 3 Litigation 1 Litigation   6 Litigation
1 Advisory
Ontario   1 Litigation 5 Litigation   6 Litigation
Quebec 3 Litigation 7 Litigation 7 Litigation
1 Advisory		   17 Litigation
1 Advisory
Atlantic 1 Litigation 1 Advisory     1 Litigation
1 Advisory
National Capital Region - Civil litigation 2 Litigation 4 Litigation     6 Litigation
Policy Sector       5 Policy 5 Policy
Legislative Services Branch       7 Legislative Services 7 Legislative Services
Total         51 Litigation
7 Legislative Services
6 Advisory
5 Policy

3.6. Focus groups

After completion of the above data collection methods and a preliminary analysis of findings, three focus groups with client departments and agencies were conducted. The focus groups explored issues raised in the evaluation findings and served as a process for validating some of the findings. To ensure that participants would have ample time to discuss the issues, a selection of departments and agencies was invited to participate in the groups. These departments and agencies were chosen so that a range of experiences in terms of litigation volume, legal issues, and legal risk management activities would be included.

LSU heads assisted by providing an appropriate client contact, who in turn assisted the evaluation by identifying an individual within their department or agency to participate in the focus groups. To provide some homogeneity among the participants, the groups were formed based on how active the client department or agency is in legal risk management.

The groups were held on May 28-29, 2008 and each focus group lasted approximately 1½ hours. A total of 13 client departments and agencies participated. The participants were roughly evenly divided between managers and working level staff.