Evaluation of Litigation Services

• Previous Page
• Table of Contents

Appendix C: Methodology

The methodology for the evaluation consisted of multiple lines of evidence, including a document review, key informant interviews, an administrative data review, and a survey of litigation staff. Each of these methods is described below.

Document review

The document review provided descriptive information on litigation-related activities as well as information responding to most evaluation questions. This review was ongoing throughout the life of the project, and included the following types of documents:

• Program documents. A wide variety of administrative and internal documents related to the function of the NLS and the Department’s provision of litigation services were available for the evaluation.
• Publicly available departmental and other government documents. The document review included review of Departmental Performance/Results Reports, Reports on Plans and Priorities, Litigation Year in Review reports, and other publicly available documents containing relevant information on federal priorities related to litigation.
• Results for relevant questions from the Department of Justice Canada Client Feedback Survey. This survey is administered by the Corporate Planning, Reporting, and Risk Division as part of its overall performance management agenda. The purpose of the survey is to obtain feedback on the degree to which Justice Canada legal services respond to the needs of federal departments and agencies. The questions are designed to measure the extent to which Justice Canada is achieving its service standards outlined in the MOUs with client departments/agencies. The evaluation included available results for Cycle III (2016-2019) of the Survey.
• Results for relevant questions from the 2017 Public Service Employee Survey (PSES). The office of the Chief Human Resources officer, Treasury Board of Canada conducts the PSES to gather federal government employees' opinions about various dimensions of their work experience. Results from the 2017 PSES (specific to the NLS overall and for NLS Regional offices) were available for the evaluation.

Key informant interviews

Key informant interviews were conducted to obtain the opinions, perceptions, and experiences of key stakeholders with knowledge of the Department’s litigation services and the NLS. Interviews involved various groups within Justice Canada, including NLS staff, as well as representatives from other areas of Justice Canada which are involved in the Department’s litigation activities either directly (through direct provision of litigation services) or indirectly (through provision of support or advice to departmental litigators).

Interviews were conducted using structured interview guides tailored to the specific groups, with questions designed to address the evaluation issues. To maximize the number of groups and individuals involved, interviews were primarily held in small group (as opposed to individual) interviews. Senior managers were selected from the areas of Justice involved in the delivery of litigation services or litigation support and asked to participate in these small group interviews with other senior managers in their sectors. A total of 49 interviews were conducted with 122 individuals. Interviews were conducted either by telephone or in person in the key informants’ official language of choice, and were generally between 75 and 90 minutes in length.

Table 4 below provides a more detailed breakdown of the key informants and groups that participated in interviews.

The following descriptive scale was used to report on some aspects of key informant interviews, with “a few” being approximately 10% to 15% or less of respondents, “some” being more than 15% to approximately 40%, “many” being more than 40% to approximately 60%, “most” being more than 60% to approximately 80%, and “almost all” being over 80%.

Administrative data review

Administrative data was obtained from Justice Canada’s Departmental Business Analytics System (i.e., Explore). Data was extracted from Explore’s Data Warehouse via Tableau, which includes data from iCase, IFMS, and HRMS. The data were extracted between March and June of 2019.

The data review focussed on litigation files led by the NLS or by one of the other litigation units between FYs 2014-2015 and 2017-2018. The administrative data for the other litigation units focussed on the number of actively managed litigation files opened and closed. TC, due to the small number of litigation files handled, and NSLAG, due to the sensitive nature of their litigation files, were not included in the administrative data review. The data review for the NLS was more extensive and also considered volume by type of client, outcome data, legal risk and complexity ratings, and level of effort (hours). The data review for the other litigation units was more limited, and included the number of litigation files opened, closed, and actively managed.

To the extent possible, the administrative data review considered trends over time, pre/post-NLS comparisons, and regional trends as applicable and relevant. However, as data was provided primarily in aggregate form (e.g., summarized at the level of NLS units), statistical analysis was not typically feasible due to the reduced degrees of freedom and statistical power associated with summary data. Trends were therefore examined graphically.

Survey of litigation staff

To gather the input of litigation staff, the evaluation included an anonymous and confidential bilingual web-based survey. The survey population included all litigation staff (including counsel, paralegals, and legal assistants) within the NLS, as well as staff outside the NLS that provide litigation services.^{Footnote 49}

Prior to administering the survey, a bilingual survey questionnaire was developed and pretested by PRA and the Evaluation Branch to ensure that all skip logic was working correctly and that there were no technical issues. Skips were used to ensure that survey respondents were only asked questions that were relevant to them.

The survey was online for just over three weeks, from March 1 to March 20, 2019. During this period, three reminders were sent to potential participants to increase the response rate. Survey invitations were sent to 1,615 potential respondents. Ninety-nine respondents were screened out of the survey (for not working in a targeted area of Justice Canada, or because they were not primarily involved in litigation work). In total, 530 respondents completed the survey for a response rate of 35%.^{Footnote 50} Once the survey was finished, open-ended questions were coded and the survey data was analyzed using SPSS, a statistical software package.

Table 5 provides a profile of survey respondents. The vast majority (93%) of respondents were from the NLS as opposed to LSUs or other areas involved directly in litigation (7%).

Limitations, challenges, and mitigation strategies

Interviews and the survey. One important limitation was the possibility of introducing bias as a result of the approach to sampling for the survey and key informant interviews, as well as the voluntary nature of participation in these data collection methods. For the survey of litigation staff (counsel and paralegals), this risk was mitigated somewhat by taking a census approach (i.e., inviting all litigation staff to participate). While this approach does not eliminate all forms of bias (most notably, self-selection bias among those who choose to complete the survey), it does remove the possibility of introducing bias at the sample development stage. The interviews with key informants and the survey of litigation staff also have the possibility of introducing self-reported response bias and strategic response bias. Self-reported response bias occurs when individuals are reporting on their own activities and so may want to portray themselves in the best light. Strategic response bias occurs when the participants answer questions with the desire to affect outcomes.

The risk of bias was mitigated by the use of multiple lines of evidence, including objective sources of data, such as documents and administrative data (to the extent that these are available), to arrive at the overall evaluation conclusions.

The survey had the added challenge of including several categories of staff (counsel, paralegals, legal assistants) and areas of Justice Canada (regional offices, CLS, NeDLSS, ADAGO, other litigation units outside of NLS). The development of the survey questionnaire attempted to ensure that respondents received questions that would be relevant to them. The feedback of the evaluation working group and the use of complicated skip logic were the methods used to direct respondents to appropriate questions.

Administrative data review. An objective of the evaluation was to consider the early impact of the NLS on the effectiveness and efficiency of litigation services. The evaluation period covered two years before the creation of the NLS (FYs 2014-2015 and 2015-2016) and two years after its launch (FYs 2016-2017 and 2017-2018). The short time period since the creation of the NLS meant that there was limited ability to conduct a pre/post analysis using administrative data. As a result, this type of pre/post comparison relied largely on qualitative data gathered through interviews. In addition, a number of data files had changed during the time period, which sometimes meant that they were not available pre-NLS or even for both years post-NLS. The evaluation relied on the NLS to assist with identifying when this had occurred and when mapping of previous data to new fields was possible and yielded reliable results.

The financial data for Justice Canada used to be maintained based on organizational units within the Department, but starting in FY 2018-2019, financial data by business line is being kept. Because this evaluation is of the litigation services business line, the evaluation team worked with Financial Management Services to construct financial data for litigation services. Doing so was also complicated by the fact that the NLS did not exist for the first two years of the evaluation, not all NLS units were included in the evaluation, and NLS units are involved in both litigation and advisory work.

The cost of legal services relative to the file outcomes was an indicator for efficiency of litigation services. Because of the difficulty in obtaining accurate cost over time, the evaluation relied on the level of effort (hours recorded to the file) as a stand-in for cost.

• Previous Page
• Table of Contents

Date modified:

2020-05-08