Business and Regulatory Law Portfolio Evaluation

3. Methodology

The evaluation draws on four lines of evidence: a document and data review; key informant interviews with departmental officials and other stakeholders; a Legal Counsel Survey; and case studies. The methodology was developed in consultation with the BRLP Evaluation Advisory Committee, which also reviewed all data collection instruments.

The evaluation matrix, which identifies the evaluation questions, indicators and lines of evidence and which was used to guide the evaluation, is included in Appendix A. The data collection instruments developed to respond to the evaluation matrix are included in Appendix C.

3.1. Document and Data Review

The document and data review was conducted both to inform the development of data collection instruments and to address the evaluation questions. Internal Portfolio documents were reviewed (e.g. business plans, newsletters), as well as publicly available information, such as Budget speeches, Speeches from the Throne, Departmental Performance Reports and Reports on Plans and Priorities.

In addition to documents, the evaluation also included a review of iCase data from fiscal years 2009-10 to 2013-14. iCase is the Department’s integrated case management, timekeeping and billing, document management and reporting system.

3.2. Key Informant Interviews

Key informant interviews were a key line of evidence in gathering information on the effectiveness of the Portfolio’s activities. A list of potential key informants was prepared, and interview guides tailored to each key informant group were developed in consultation with the Evaluation Advisory Committee. Interviews were conducted with a total of 90 key informants, most of whom were in managerial-level positions. Table 4 below provides a breakdown of the number of key informants interviewed by respondent group and includes those interviewed as part of the case studies.

Table 4: Key informants
Category Number
BRLP
ADMO 5
DLSUs 34
Regional Offices 10
CLS 2
Sub total 51
Justice – other areas 9
Sub total 9
Other federal departments and agencies
Client departments and agencies 27
Public Prosecution Service of Canada 3
Sub total 30
TOTAL 90

Potential interviewees received an invitation to participate in an interview. Those who agreed to participate were provided with a copy of the interview guide (in the official language of their choice) prior to the interview and each interview was conducted in the respondents’ preferred official language. The evaluation included a mix of telephone and in-person interviews. Interview notes were sent to each respondent for their review upon completion of the interview. This helped ensure that notes taken during the interview were accurate and complete and served to enhance the overall validity of the interview data. Additionally, multiple evaluators were tasked with conducting the interviews. Using several interviewers helped reduce the potential bias that could come from a single person doing all the data collection and provided another means of more directly assessing the consistency of the data obtained.

3.3. Survey of Business and Regulatory Law Portfolio Counsel

To gather the input of counsel more generally, the evaluation included a web-based survey. The survey was online for a total of three weeks – from May 8 to May 29, 2015. Invitations were sent to all counsel in the DLSUs and in the CLS and to all counsel in the regions who work on BRLP files (n=622). To increase the response rate, a message was sent to counsel from the ADM and Deputy ADM, BRLP encouraging their participation. Additionally, the Evaluation Division sent a reminder one week prior to closing the survey. In total, 262 counsel completed the survey for a response rate of 44%Footnote 5. Overall, survey respondents were well distributed among the various Portfolio categories (e.g. work unit, classification) including within DLSUs and regional offices and were largely reflective of the general population.

3.4. Case Studies

Fifteen case studies were conducted as part of the evaluation: one for each of the fourteen DLSUs in the Portfolio and one for the CLS. Four of the case studies included a litigation component which involved regional counsel. The purpose of the case studies was two-fold. First, it allowed for a more in-depth understanding of the life of a file and how the Portfolio conducts its work. Secondly, it allowed the evaluation to explore whether the information obtained from key informants on how the Portfolio conducts its work was supported by a review of selected case files. The case study files were selected in consultation with each of the DLSU heads and the Director, CLS. They included not only day-to-day files typically handled by the Portfolio, but also more complex files involving multiple departments and requiring a Portfolio-wide approach and coordination. The case study files were largely reflective of the proportion of each file type (e.g. advisory, litigation, legislative) managed by the Portfolio. The Evaluation Division provided BRLP officials with guidelines for selecting the case study files. For example, the file had to have been opened and completed during the evaluation period and, if not, discrete work had to have been conducted and completed during that time. The availability and willingness of both the counsel assigned to the file and the client to participate in an interview, as well as the completeness of the information on the file were also considered. The case studies included a review of the file and an examination of the iCase data on the file, followed by key informant interviews with the legal counsel (from DLSUs and regional offices as applicable) and clients involved in the file.

To ensure that comparable information was collected from each file, the evaluators completed a standard file review template that was developed for the evaluation (see Appendix C). The template collected information to respond to the evaluation matrix and focused on factual information available in the files and in iCase.

3.5. Methodological Limitations and Mitigation Strategy

Interviews and Case Studies

The interviews with key informants have the possibility of self-reported response bias, which occurs when individuals are reporting on their own activities and may therefore want to portray themselves in a positive light; and strategic outcomes response bias, whereby the participants answer questions with the desire to affect outcomes. Additionally, the sample of files for the case studies was not chosen by random selection and the sample is not large, nor representative. However, the files chosen were considered to provide a good representation of the diversity of the Portfolio’s work and were intended only to be illustrative of the Portfolio’s approach to its different types of work. Both the interviews and case studies also have the possibility of selection bias in that client key informants and files were identified by BRLP counsel.

In order to mitigate these limitations, triangulation was used to check findings against other sources and perspectives. The evaluators compared and cross-checked the consistency of information derived by different means by comparing the perspectives of people from different points of view. By seeking information from the full circle of Portfolio stakeholders, the evaluators were able to triangulate the views of Portfolio staff, clients and partners and check interview findings against written documentation on file, internal documents and reports, and secondary data (e.g. results from the Public Service Employee Survey [PSES] and the Client Feedback Survey) to corroborate what interview respondents reported. The evaluators found consistency in overall patterns of data from different sources.

iCase data

Although advisory files represented the majority (71%) of files managed by the Portfolio and half of the hours recorded by BRLP counsel during the time period covered by the evaluation, an average of 78% of actively managed advisory files did not have a risk rating recorded in iCase as it was not mandatory to enter risk rating information for advisory files until September 2013. The quality of the iCase data on risk and complexity ratings for litigation files was not reliable either, as an average of 33% of litigation files did not have a risk ratingFootnote 6 and an average of 50% of litigation files did not have a complexity rating. Due to the lack of risk information, it was not possible to undertake analyses to determine whether appropriate counsel were assigned to legal files based on their classification and the risk level and complexity of the file (e.g. whether more senior counsel worked on higher risk and complexity files and conversely, whether junior counsel worked on lower risk and complexity files). Instead, the evaluation examined the level of effort (hours) by classification more generally, without accounting for risk level or file complexity and relied on the results of the Legal Counsel Survey to assess whether files were assigned to the appropriate level and number of counsel given their legal risk and complexity level. It is expected that with the implementation of the National Timekeeping Protocol in 2015, that sufficient risk and complexity data will be available for the next evaluation.

2009 Client Feedback Survey

The evaluation report relied in part on the findings from the most recent BRLP Client Feedback Survey conducted in 2009. The survey provided valuable insight into the Portfolio’s performance against the department’s service standards due to the large number of clients (n=3,032) who provided feedback. Since the survey results were somewhat dated, the evaluation also relied on other sources of data regarding client perceptions of the quality of BRLP’s legal services through both case study and general client interviews. In total, 27 clients were interviewed as part of the evaluation. As part of the analysis, the evaluators compared the results of the Client Feedback Survey with the results of the evaluation interviews, which were found to be consistent.