Business and Regulatory Law Portfolio Evaluation

Appendix C: Data Collection Tools

Key Informant Interview Guide for Business and Regulatory Law Portfolio Senior Management

The Department of Justice Evaluation Division is conducting an evaluation of the Business and Regulatory Law Portfolio (BRLP).This evaluation comes in response to the Treasury Board Policy on Evaluation, which requires government departments to evaluate all direct expenditures (including legal services) every five years. The purpose of the evaluation is to assess the extent to which the BRLP provides effective and responsive legal services to its client departments and agencies. In addition to these interviews, the evaluation will also rely on other lines of evidence, including a review of documents and data, key informant interviews with representatives of the Department of Justice and clients, a file review, case studies, a survey of legal counsel, and focus groups.

The information gathered through this interview will be summarized in aggregate form and individual respondents will not be identified in the evaluation report. You will have an opportunity to review the written summary of the interview and make any corrections or additions.

Some questions may not be applicable to the work you do. Please let us know and we will skip those questions.

The evaluation focuses on the five year period from 2009-10 to 2013-14, so please consider your experiences during those years in your responses. Unless otherwise stated, any reference in this interview guide to the work of the BRLP is meant to include the work of the Office of the Assistant Deputy Minister (ADMO), all Departmental Legal Services Units (DLSUs) within the Portfolio, the Commercial Law Section, and BRLP sections in the regional offices.

Relevance

1. How have the demands placed on the BRLP evolved over the past five years? What have been the most significant changes, if any? How has the BRLP responded to these changes?

Performance — Effectiveness

2. Considering the current structure and composition of the Portfolio, how would you describe its key strengths? What are its key challenges? What changes, if any, could be considered to enhance the efficiency and/or the effectiveness of the Portfolio’s structure and composition?

3. What structures and mechanisms are in place within the BRLP to coordinate and manage significant legal issues (for e.g. high impact files, files involving multiple departments or agencies)? How well do these structures work?

4. How would you describe the nature, level and scope of integration and cooperation within the BRLP (e.g. between DLSUs and regional offices and amongst DLSUs)?

5. What practices and/or mechanisms are in place to ensure that the advice provided by the BRLP is consistent with other advice provided or positions taken by Justice on the same legal question? What, if anything, could be improved, and how might the BRLP support any improvements?

6. What have been some of the greatest accomplishments of the BRLP over the last five years?

7. What have been the greatest challenges faced by the BRLP over the last five years?

Performance — Efficiency and Economy

8. Is the BRLP facing any challenges in terms of available funding, staffing, or internal support (e.g. training, technological, research) that affect its ability to deliver timely, high-quality legal services? How does it manage these challenges?

9. Please describe the types of professional development activities available to BRLP counsel to support their work. Are there any gaps? Should changes to the type and range of issues addressed by professional development be considered?

10. What measures are in place to ensure that the Portfolio’s activities are carried out efficiently? Cost-effectively? Are there improvements that could be made?

11. What mechanisms and structures are in place within the BRLP to facilitate information exchange/communication between DLSUs, regional offices and ADMO? Are there improvements that could be made? To what extent are best practices, lessons learned, and work products (e.g. legal opinions) shared within the Portfolio?

12. Are there any aspects of legal management practices or models in other portfolios of Justice Canada or in other jurisdictions that could be considered for implementation within the BRLP to enhance efficiency and/or effectiveness? If so, please describe them.

Conclusion

13. Do you have any other comments?

Thank you. We greatly appreciate your participation.

Key Informant Interview Guide for Business and Regulatory Law Portfolio Management

The Department of Justice Evaluation Division is conducting an evaluation of the Business and Regulatory Law Portfolio (BRLP). This evaluation comes in response to the Treasury Board Policy on Evaluation, which requires government departments to evaluate all direct expenditures (including legal services) every five years. The purpose of the evaluation is to assess the extent to which the BRLP provides effective and responsive legal services to its client departments and agencies. In addition to these interviews, the evaluation will also rely on other lines of evidence, including a review of documents and data, key informant interviews with representatives of the Department of Justice and clients, a file review, case studies, a survey of legal counsel, and focus groups.

The information gathered through this interview will be summarized in aggregate form and individual respondents will not be identified in the evaluation report. You will have an opportunity to review the written summary of the interview and make any corrections or additions.

Some questions may not be applicable to the work you do. Please let us know and we will skip those questions.

The evaluation focuses on the five year period from 2009-10 to 2013-14, so please consider your experiences during those years in your responses. Unless otherwise stated, any reference in this interview guide to the work of the BRLP is meant to include the work of the Office of the Assistant Deputy Minister (ADMO), all Departmental Legal Services Units (DLSUs) within the Portfolio, the Commercial Law Section, and BRLP sections in the regional offices.

Introduction

1. Please briefly describe your current roles and responsibilities within the BRLP.

Relevance

2. How have the demands placed on your DLSU evolved over the past five years? What have been the most significant changes, if any? How has your DLSU responded to these changes?

3. In the context of your work in the BRLP, how do the Portfolio’s services support Government of Canada priorities?

Performance — Effectiveness

4. Considering the current structure and composition of the Portfolio, how would you describe its key strengths? What are its key challenges? What changes, if any, could be considered to enhance the efficiency and/or the effectiveness of the Portfolio’s structure and composition?

5. Based on your experience, is your DLSU generally consulted by client departments when needed? In your response, please consider whether client requests for assistance are timely and appropriate.

6. What factors affect clients’ willingness or ability to consult with the DLSU (e.g., cost, awareness/understanding of when it would be advisable to consult, perception of usefulness/desirability of obtaining advice)? What are the potential risks to client departments, the Department of Justice, and the Government of Canada when the DLSU is not consulted when needed?

7. Please describe the level and the nature of collaboration between your DLSU and a) the specialized sections within Justice Canada (e.g. Public Law Sector, Litigation Branch and Legislative Services Branch) and b) other portfolios? How effective has this collaboration been? What changes, if any, could improve this collaboration?

8. What structures and mechanisms are in place within the BRLP to coordinate and manage significant legal issues (for e.g. high impact files, files involving multiple departments or agencies)? How well do these structures work?

9. How would you describe the nature, level and scope of integration and cooperation within the BRLP (e.g. between DLSUs and regional offices and amongst DLSUs)?

10. What practices and/or mechanisms are in place to ensure that the advice provided by the BRLP is consistent with other advice provided or positions taken by Justice on the same legal question? What, if anything, could be improved, and how might the BRLP support any improvements?

11. One of the outcomes identified for the BRLP is that client departments/agencies have access to timely, responsive, and useful advisory and litigation services. To what extent is your DLSU able to achieve this outcome? What, if any factors, affect your DLSU’s ability to provide timely, responsive, and useful legal services?

12. Please describe how your DLSU works with its clients to identify and assess legal risks and provide legal options for mitigating them. In particular, what are the key processes, tools, committees or structures currently used to assist in identifying and mitigating legal risks? In your opinion, how efficient and/or effective have they been?

13. In your opinion, has your client’s level of awareness and understanding of legal risks improved over the last five years? If so, what do you consider to be the key contribution of your DLSU towards this outcome? If not, please explain. What more needs to be done? Has any formal training of the client been undertaken by your DLSU with respect to legal risk management or other legal issues?

14. To what extent does your DLSU’s advice appear to inform decisions made by Ministers and client departments/agencies? What is the basis for your opinion?

15. How do your DLSU’s advisory services support the implementation of government decisions?

Performance — Efficiency and Economy

16. Is your DLSU facing any challenges in terms of available funding, staffing, or internal support (e.g. training, technological, research) that affect its ability to deliver timely, high-quality legal services ? How does it manage these challenges?

17. Please describe the types of professional development activities available to BRLP counsel to support their work. Are there any gaps? Should changes to the type and range of issues addressed by professional development be considered?

18. What measures are in place to ensure that the Portfolio’s activities are carried out efficiently? Cost-effectively? Are there improvements that could be made?

19. In your opinion, what role do client departments/agencies play in improving the effectiveness, efficiency, and economy of BRLP legal services? To what extent are they fulfilling this role?

20. What mechanisms and structures are in place within the BRLP to facilitate information exchange/communication between DLSUs, regional offices and ADMO? Are there improvements that could be made? To what extent are best practices, lessons learned, and work products (e.g. legal opinions) shared within the Portfolio?

21. Are there any aspects of legal management practices or models in other portfolios of Justice Canada or in other jurisdictions that could be considered for implementation within the BRLP to enhance efficiency and/or effectiveness? If so, please describe them.

Conclusion

22. Do you have any other comments?

Thank you. We greatly appreciate your participation.

Key Informant Interview Guide for Managers in Regional Offices

The Department of Justice Evaluation Division is conducting an evaluation of the Business and Regulatory Law Portfolio. This evaluation comes in response to the Treasury Board Policy on Evaluation, which requires government departments to evaluate all direct expenditures (including legal services) every five years. The purpose of the evaluation is to assess the extent to which the BRLP provides effective and responsive legal services to its client departments and agencies. In addition to these interviews, the evaluation will also rely on other lines of evidence, including a review of documents and data, key informant interviews with representatives of the Department of Justice and clients, a file review, case studies, a survey of legal counsel and focus groups.

The information gathered through this interview will be summarized in aggregate form and individual respondents will not be identified in the evaluation report. You will have an opportunity to review the written summary of the interview and make any corrections or additions.

Some questions may not be applicable to the work you do. Please let us know and we will skip those questions.

The evaluation focuses on the five year period from 2009-10 to 2013-14, so please consider your experiences during those years in your responses. Unless otherwise stated, any reference in this interview guide to the work of the BRLP is meant to include the work of the Office of the Assistant Deputy Minister (ADMO), all Departmental Legal Services Units (DLSUs) within the Portfolio, the Commercial Law Section, and BRLP sections in the regional offices.

Introduction

1. Please describe your current roles and responsibilities as they relate to the BRLP specifically. With which group(s) do you interact most often in the Portfolio?

Relevance

2. Considering the range of files involving departments and agencies included in the BRLP, please describe how the demands for legal services from your regional office have evolved over the past five years. What have been the most significant changes, if any?

Performance - Effectiveness

3. Considering the current structure and composition of the Portfolio, how would you describe its key strengths? What changes, if any, could be considered to enhance the efficiency and/or effectiveness of the Portfolio’s structure and composition?

4. Please describe the level and nature of the collaboration between your regional office and BRLP DLSUs? How effective has this collaboration been? What changes, if any, could improve this collaboration?

5. Please describe the level and nature of the collaboration between your regional office and Justice Canada’s specialized sections (e.g. Public Law Sector, Litigation Branch, Legislative Services Branch) on BRLP files? How effective has this collaboration been? What changes, if any, could improve this collaboration?

6. What structures and mechanisms are in place within the BRLP to coordinate significant legal issues (e.g. files involving multiple departments or agencies)? How well do these structures work?

7. What practices and/or mechanisms are in place to ensure that the advice provided by the BRLP is consistent with other advice provided or positions taken by Justice on the same type of legal question? What, if anything, could be improved, and how might the BRLP support any improvements?

8. Please describe how the BRLP works with its clients to identify legal risks and legal options for mitigating them. In particular, what key tools, strategies, committees or structures are currently utilized to mitigate and manage legal risks? Do those tools differ from the ones developed by your regional office? In your opinion, how effective and/or efficient have they been?

9. Please describe what you consider to be the key impacts of professional development activities undertaken to date in support of the work of regional legal counsel who work on BRLP files (practice groups, training sessions, presentations, etc.). Are there any gaps? Should changes to the type and range of issues addressed by professional development be considered?

10. In your opinion, has the level of awareness and understanding of legal risks among client departments and agencies improved over the last five years? If so, what do you consider to be the Portfolio’s contribution towards this outcome? Has your region offered any client training in this area?

11. One of the outcomes identified for the BRLP is that client departments/agencies have access to timely, responsive, and useful advisory and litigation services. To what extent is the BRLP able to achieve this outcome? What, if any factors, affect the BRLP’s ability to provide timely, responsive, and useful advice?

Performance – Efficiency and Economy

12. What mechanisms and structures are in place within the BRLP to facilitate information exchange/communication between DLSUs, regional offices and ADMO? Are there improvements that could be made? To what extent are best practices, lessons learned, and work products (e.g. legal opinions) shared within the Portfolio?

13. Is there anything that the BRLP could do to make your work more effective and efficient?

14. What specific steps has your regional office taken to reduce legal costs related to BRLP files?

15. In general, have appropriate DLSU counsel been assigned to work with your office (considering the level of classification, expertise, and experience of the counsel and the nature of the file)?

16. Are there any aspects of legal management practices or models in other portfolios of Justice Canada or in other jurisdictions that could be considered for implementation within the BRLP to enhance efficiency and/or effectiveness? If so, please describe them.

Conclusion

17. Do you have any further comments relating to the work of the BRLP?

Thank you. We greatly appreciate your participation.

Key Informant Interview Guide for Special Advisors

The Department of Justice Evaluation Division is conducting an evaluation of the Business and Regulatory Law Portfolio (BRLP). This evaluation comes in response to the Treasury Board Policy on Evaluation, which requires government departments to evaluate all direct expenditures (including legal services) every five years. The purpose of the evaluation is to assess the extent to which the BRLP provides effective and responsive legal services to its client departments and agencies. In addition to these interviews, the evaluation will also rely on other lines of evidence, including a review of documents and data, key informant interviews with representatives of the Department of Justice and clients, a file review, case studies, a survey of legal counsel, and focus groups.

The information gathered through this interview will be summarized in aggregate form and individual respondents will not be identified in the evaluation report. You will have an opportunity to review the written summary of the interview and make any corrections or additions.

Some questions may not be applicable to the work you do. Please let us know and we will skip those questions.

The evaluation focuses on the five year period from 2009-10 to 2013-14, so please consider your experiences during those years in your responses. Unless otherwise stated, any reference in this interview guide to the work of the BRLP is meant to include the work of the Office of the Assistant Deputy Minister (ADMO), all Departmental Legal Services Units (DLSUs) within the Portfolio, the Commercial Law Section, and BRLP sections in the regional offices.

Introduction

1. Please briefly describe your current roles and responsibilities within the BRLP.

Relevance

2. How have the demands placed on the Portfolio evolved over the past five years? What have been the most significant changes, if any? How has the Portfolio responded to these changes?

3. In the context of your work in the BRLP, how do the Portfolio’s services support Government of Canada priorities? Which files should be highlighted in the evaluation report?

Performance — Effectiveness

4. Considering the current structure and composition of the Portfolio, how would you describe its key strengths? What are its key challenges? What changes, if any, could be considered to enhance the efficiency and/or the effectiveness of the Portfolio’s structure and composition?

5. Please describe the level and the nature of collaboration between your office and a) the BRLP DLSUs b) the Regional Offices and c) the specialized sections within Justice Canada (e.g. Public Law Sector, Litigation Branch and Legislative Services Branch)? How effective has this collaboration been? What changes, if any, could improve this collaboration?

6. What structures and mechanisms are in place within the BRLP to coordinate and manage significant legal issues (for e.g. high impact files, files involving multiple departments or agencies)? How well do these structures work?

7. How would you describe the nature, level and scope of integration and cooperation within the BRLP?

8. What practices and/or mechanisms are in place to ensure that the advice provided by the BRLP is consistent with other advice provided or positions taken by Justice on the same legal question? What, if anything, could be improved, and how might the BRLP support any improvements?

9. To what extent does BRLP’s advice appear to inform decisions made by Ministers and client departments/agencies? What is the basis for your opinion?

10. What have been some of the greatest accomplishments of the BRLP over the last five years?

11. What have been the greatest challenges faced by the BRLP over the last five years?

Performance — Efficiency and Economy

12. Please describe the types of professional development activities available to BRLP counsel to support their work. Are there any gaps? Should changes to the type and range of issues addressed by professional development be considered?

13. What measures are in place to ensure that the Portfolio’s activities are carried out efficiently? Cost-effectively? Are there improvements that could be made?

14. What mechanisms and structures are in place within the BRLP to facilitate information exchange/communication between DLSUs, regional offices and ADMO? Are there improvements that could be made? To what extent are best practices, lessons learned, and work products (e.g. legal opinions) shared within the Portfolio?

15. Are there any aspects of legal management practices or models in other portfolios of Justice Canada or in other jurisdictions that could be considered for implementation within the BRLP to enhance efficiency and/or effectiveness? If so, please describe them.

Conclusion

16. Do you have any other comments?

Thank you. We greatly appreciate your participation.

Key Informant Interview Guide for Client Departments

The Department of Justice Evaluation Division is conducting an evaluation of the Business and Regulatory Law Portfolio (BRLP). This Portfolio is responsible for providing legal services and advice to a number of regulatory departments and agencies, including your own. This evaluation comes in response to the Treasury Board Policy on Evaluation, which requires government departments to evaluate all direct expenditures (including legal services) every five years. The purpose of the evaluation is to assess the extent to which the BRLP provides effective and responsive legal services to its client departments and agencies.

The information gathered through this interview will be summarized in aggregate form and individual respondents will not be identified in the evaluation report. You will have an opportunity to review the written summary of the interview and make any corrections or additions.

Some questions may not be applicable to the work you do. Please let us know and we will skip those questions.

The evaluation focuses on the five year period from 2009-10 to 2013-14, so please consider your experiences during those years in your responses. Unless otherwise stated, any reference in this interview guide to the work of the BRLP is meant to include the work of the Office of the Assistant Deputy Minister (ADMO, BRLP), all Departmental Legal Services Units (DLSUs) within the Portfolio, the Commercial Law Section, and BRLP sections in the regional offices.

Relevance

1. Over the last five years, have you observed any changes in your department or agency’s demand for and use of legal services? In your response, please consider the volume, type of legal issues, complexity, legal risk level, the urgency of the requests being made and any other characteristics.

Performance — Effectiveness

2. Bearing in mind your department’s priorities over the review period, please comment on the following dimensions of the quality of the overall services provided by your DLSU, by regional offices or other BRLP offices:

  1. The extent to which counsel respond to your requests for legal services in a timely manner.
  2. The extent to which counsel negotiate and meet mutually acceptable deadlines.
  3. The extent to which you are provided with regular and informative progress reports or ongoing feedback in respect of requests for services.
  4. The extent to which your department/agency is provided with clear and practical guidance for resolving legal issues.
  5. The extent to which counsel identifies means to prevent and resolve legal disputes at the earliest opportunity.
  6. The extent to which your department/agency is provided with advice/options appropriate to your policy and program objectives.
  7. The extent to which legal counsel work with you to identify legal risks.
  8. The extent to which legal counsel involve you in the review/development of legal options to mitigate identified legal risks.
  9. The extent to which your department or agency receives consistent legal advice.

3. Bearing in mind your department’s/ agency’s priorities over the review period, overall, how satisfied were you with the quality of the legal services provided by your DLSU, by the regional offices or other BRLP offices?

4. Based on your experience, is your DLSU consulted when it should be? What situations, if any, have limited the use of your DLSU’s services (e.g., cost, lack of awareness/understanding of when it would be advisable to consult the DLSU, perception of usefulness/desirability of obtaining DLSU advice)?

5. Have you participated in any legal training activities or information sessions on legal issues and legal risks offered by your DLSU, by the regional offices or by another part of the BRLP within the past five years? If yes, what type of training did you receive? How effective was this training in improving your understanding of the relevant subject-matter?

6. In your experience, to what extent is the advice provided by the BRLP considered in the legal strategies pursued and decisions made by your department or agency? What is the basis for your opinion?

Performance — Efficiency and Economy

7. To your knowledge, are there any factors that influence the ability of your DLSU or the regional offices to deliver their services effectively? If so, please describe these factors. What changes, if any, are needed to increase the efficiency and/or effectiveness of the services offered?

8. Has your department or agency tried to reduce or manage the level of demand for BRLP legal services? If so, how?

Conclusion

9. Do you have any other comments?

Thank you. We greatly appreciate your participation.

Key Informant Interview Guide for Commercial Law Section Clients

The Department of Justice Evaluation Division is conducting an evaluation of the Business and Regulatory Law Portfolio (BRLP). This evaluation comes in response to the Treasury Board Policy on Evaluation, which requires government departments to evaluate all direct expenditures (including legal services) every five years. The purpose of the evaluation is to assess the extent to which the BRLP provides effective and responsive legal services to its client departments and agencies.

The information gathered through this interview will be summarized in aggregate form and individual respondents will not be identified in the evaluation report. You will have an opportunity to review the written summary of the interview and make any corrections or additions.

Some questions may not be applicable to the work you do. Please let us know and we will skip those questions.

The evaluation focuses on the five year period from 2009-10 to 2013-14, so please consider your experiences during those years in your responses. Unless otherwise stated, any reference in this interview guide to the work of the BRLP is meant to include the work of the Office of the Assistant Deputy Minister (ADMO, BRLP), all Departmental Legal Services Units (DLSUs) within the Portfolio, the Commercial Law Section, and BRLP sections in the regional offices.

Relevance

1. Over the last five years, have you observed any changes in your DLSU’s demand for and use of the legal services provided by the Commercial Law Section? In your response, please consider the volume, type of legal issues, complexity, legal risk level, the urgency of the requests being made and any other characteristics.

Performance — Effectiveness

2. Please comment on the following dimensions of the quality of the overall services provided by the Commercial Law Section:

  1. The extent to which counsel respond to your requests for legal services in a timely manner.
  2. The extent to which counsel negotiate and meet mutually acceptable deadlines.
  3. The extent to which you are provided with regular and informative progress reports or ongoing feedback in respect of requests for services.
  4. The extent to which counsel provide clear and practical guidance for resolving legal issues.
  5. The extent to which counsel identify means to prevent and resolve legal disputes at the earliest opportunity.
  6. The extent to which the advice/options provided is appropriate to your client department’s policy and program objectives.
  7. The extent to which legal counsel work with you to identify legal risks.
  8. The extent to which legal counsel involve you in the review/development of legal options to mitigate identified legal risks.
  9. The extent to which your department or agency receives consistent legal advice.

3. Overall, how satisfied were you with the quality of the legal services provided by the Commercial Law Section?

4. Have you participated in any legal training activities or information sessions on legal issues and legal risks offered by the Commercial Law Section within the past five years? If yes, what type of training did you receive? How effective was this training in improving your understanding of the relevant subject-matter?

5. In your experience, to what extent is the advice provided by the Commercial Law Section considered in the legal strategies pursued and decisions made by the DLSU and/or the client department or agency? What is the basis for your opinion?

Performance — Efficiency and Economy

6. To your knowledge, are there any factors that influence the ability of the Commercial Law Section to deliver their services effectively? If so, please describe these factors. What changes, if any, are needed to increase the efficiency and/or effectiveness of the services offered?

Conclusion

7. Do you have any other comments?

Thank you. We greatly appreciate your participation.

Key Informant Interview Guide for Other Justice Sections

The Department of Justice Evaluation Division is conducting an evaluation of the Business and Regulatory Law Portfolio (BRLP). This evaluation comes in response to the Treasury Board Policy on Evaluation, which requires government departments to evaluate all direct expenditures (including legal services) every five years. The purpose of the evaluation is to assess the extent to which the BRLP provides effective and responsive legal services to its client departments and agencies. In addition to these interviews, the evaluation will also rely on other lines of evidence, including a review of documents and data, key informant interviews with representatives of the Department of Justice and clients, a file review, case studies, a survey of legal counsel, and focus groups.

The information gathered through this interview will be summarized in aggregate form and individual respondents will not be identified in the evaluation report. You will have an opportunity to review the written summary of the interview and make any corrections or additions.

Some questions may not be applicable to the work you do. Please let us know and we will skip those questions.

The evaluation focuses on the five year period from 2009-10 to 2013-14, so please consider your experiences during those years in your responses. Unless otherwise stated, any reference in this interview guide to the work of the BRLP is meant to include the work of the Office of the Assistant Deputy Minister (ADMO), all Departmental Legal Services Units (DLSUs) within the Portfolio, the Commercial Law Section, and BRLP sections in the regional offices.

Introduction

1. Please describe the level of your involvement and the nature of your work with the BRLP.

2. What types of legal services (e.g., what type of advice, litigation support) have you provided to or sought from the BRLP over the past five years?

Relevance

3. Please describe how your work with the BRLP has evolved over the past five years. For example, have there been changes in the volume or nature of legal issues, the complexity and/or urgency of the requests being made, file complexity or any other characteristics?

Performance — Effectiveness

4. Please describe how your work with the BRLP is currently structured. We are particularly interested in any processes you may have in place (formal or informal) to deal with the Portfolio’s files.

5. Please describe your level of satisfaction with the following dimensions of your work with the BRLP:

  1. The extent to which your group is consulted in a timely manner.
  2. The extent to which the timeframes for completing requests are appropriate.
  3. The extent to which you are consulted on the appropriate issues.

6. Please describe the type and frequency of consultations that occur between your group and BRLP. Are they effective? What works particularly well? What, if anything, could be improved, and how might BRLP support any improvements?

7. How effective are consultations among the BRLP, other Justice areas, and other federal departments and agencies in ensuring a consistent legal position across government and developing a whole-of-government approach? How effectively are differences in legal opinions resolved and through what process? What, if anything, could be improved, and how might the BRLP support any improvements?

8. Please describe how/if BRLP legal counsel and counsel in your area work together to identify and assess legal risks, and how/if they work together to develop legal and/or policy options to manage or mitigate those risks. In your opinion, how effective is this collaboration?

9. In your opinion, to what extent does the BRLP meet the needs of the Government of Canada in terms of the provision of timely, high-quality legal services? Is there anything you are aware of that would require the BRLP to change the way it provides legal services?

Performance — Efficiency and Economy

10. In what ways, if any, could the BRLP work more effectively and efficiently with your group?

11. In general, have appropriate BRLP counsel worked with your group (considering the level of classification, expertise, and experience of the counsel and the nature of the file)?

Conclusion

12. Do you have anything you would like to add about your group’s relationship with the BRLP?

Thank you. We greatly appreciate your participation.

Key Informant Interview Guide for Public Prosecution Service of Canada

The Department of Justice Evaluation Division is conducting an evaluation of the Business and Regulatory Law Portfolio (BRLP). This evaluation comes in response to the Treasury Board Policy on Evaluation, which requires government departments to evaluate all direct expenditures (including legal services) every five years. The purpose of the evaluation is to assess the extent to which the BRLP provides effective and responsive legal services to its client departments and agencies. In addition to these interviews, the evaluation team will also rely on other lines of evidence, including a review of documents and data, key informant interviews with representatives of the Department of Justice and clients, a file review, case studies, a survey of legal counsel and focus groups.

The information gathered through this interview will be summarized in aggregate form and individual respondents will not be identified in the evaluation report. You will have an opportunity to review the written summary of the interview and make any corrections or additions.

It is our understanding that the Public Prosecution Service of Canada (PPSC) is not a typical recipient of legal services from BRLP, but instead, collaborates with the BRLP in your role in providing prosecution services to BRLP regulatory clients. Some questions may not be applicable to the work you do. Please let us know and we will skip those questions.

The evaluation focuses on the five year period from 2009-10 to 2013-14, so please consider your experiences during those years in your responses. Unless otherwise stated, any reference in this interview guide to the work of the BRLP is meant to include the work of the Office of the Assistant Deputy Minister, all Departmental Legal Services Units within the Portfolio, the Commercial Law Section, and BRLP sections in the regional offices.

Introduction

1. Please describe the level of your involvement and nature of your work with the BRLP.

2. What type of legal services (e.g. what type of advice, prosecutorial support, consultation) have you provided to or sought from the BRLP over the past five years?

Relevance

3. Please describe how your work with the BRLP has evolved over the past five years. For example, have there been changes in the volume or nature of legal issues, the complexity and/or urgency of the requests being made, file complexity or any other characteristics?

Performance — Effectiveness

4. Please describe how your work with the BRLP is currently structured. We are particularly interested in any processes (formal or informal) you may have in place to deal with the Portfolio’s files.

5. Please describe the type and frequency of consultations that occur between the PPSC and BRLP. Are they effective? What works particularly well? What, if anything, could be improved, and how might BRLP support any improvements?

6. Are there any barriers to consulting with the BRLP? What are the potential risks to the Government of Canada if the BRLP is not appropriately consulted?

7. Please describe how/if BRLP legal counsel and PPSC work together to identify and assess legal risks, and how/if they work together to develop legal and/or policy options to manage or mitigate those risks. In your opinion, how effective is this collaboration?

8. Have BRLP counsel provided your group with any formal training with respect to legal issues? If so, what was the nature of the training? How satisfied were you with the training provided?

Performance — Efficiency and Economy

9. In what ways, if any, could the BRLP work more effectively and efficiently with the PPSC?

10. In your opinion, are there processes/practices that could improve communications between BRLP and PPSC?

11. In general, have appropriate BRLP counsel been assigned to work with the PPSC (considering the level of classification, expertise and experience of the counsel, and the nature of the file)?

Conclusion

12. Do you have anything you would like to add about PPSC’s relationship with the BRLP?

Thank you. We greatly appreciate your participation.

Case Study Guide for Client Representatives

The Department of Justice Evaluation Division is conducting an evaluation of the Business and Regulatory Law Portfolio (BRLP). This Portfolio is responsible for providing legal services and advice to a number of regulatory departments and agencies, including your own. This evaluation comes in response to the Treasury Board Policy on Evaluation, which requires government departments to evaluate all direct expenditures (including legal services) every five years. The purpose of the evaluation is to assess the extent to which the BRLP provides effective and responsive legal services to its client departments and agencies. The evaluation relies on multiple lines of evidence, including case studies that focus on specific files involving counsel from the BRLP. These case studies provide a unique opportunity to better understand the work of the Portfolio at an operational level.

The information gathered through this interview will be summarized in aggregate form and individual respondents will not be identified in the evaluation report. You will have an opportunity to review the written summary of the interview and make any corrections or additions.

1. How would you describe the working relationship between your group and counsel on this file? What factors contributed to, or detracted from, a positive working relationship between your group and counsel? In your opinion, are there processes/practices that could improve how counsel and clients work together?

2. To your knowledge, was more than one group within Justice Canada involved in this file? If so, how efficiently was the work coordinated between these different groups?

3. Did the Department of Justice Canada work with you to identify and assess the legal risks associated with this file? If so, how satisfied are you with the role you played?

4. How well did your department/agency understand the legal issues and risks involved?

5. Based on your experience, please comment on the following dimensions of the quality of the overall services provided by your legal services unit in relation to this file:

  1. its capacity to respond to your requests in a timely manner
  2. its capacity to meet mutually agreed-upon deadlines
  3. its understanding of the nature of the issue for which assistance was sought
  4. the extent to which you were provided with regular and informative progress reports or ongoing feedback in respect of your request for service
  5. the extent to which counsel involved you in the development of legal strategy and positions
  6. the extent to which you were provided with clear and practical guidance on resolving legal issues

6. (For litigation files only)Were early resolution strategies or alternative dispute resolution strategies considered or used in this file? Why or why not? If they were used, what caused them to be successful or unsuccessful in resolving the file?

7. To what extent did the advice provided by Justice Canada on this file inform the decisions made by your department/agency? In your opinion, what factors affected whether the advice was considered?

8. Overall, how satisfied were you with the quality of the legal services you received on this file?

9. Were adequate resources assigned to undertake the work required (appropriate counsel and number of counsel)? Please explain any capacity issues you encountered.

10. In your opinion, was this file handled in a cost-effective manner? What, if anything, could have been done differently by Justice Canada and/or your department/agency to improve timeliness or to reduce costs?

11. In your view, what, if anything, could Justice Canada have done differently to better serve your department’s/agency’s needs?

12. Do you have any further comments relating to this file?

Thank you for your participation.

Case Study Guide for Commercial Law Section Clients

The Department of Justice Evaluation Division is conducting an evaluation of the Business and Regulatory Law Portfolio (BRLP). This Portfolio is responsible for providing legal services and advice to a number of regulatory departments and agencies, including your own. This evaluation comes in response to the Treasury Board Policy on Evaluation, which requires government departments to evaluate all direct expenditures (including legal services) every five years. The purpose of the evaluation is to assess the extent to which the BRLP provides effective and responsive legal services to its client departments and agencies. The evaluation relies on multiple lines of evidence, including case studies that focus on specific files involving counsel from the BRLP. These case studies provide a unique opportunity to better understand the work of the Portfolio at an operational level.

The information gathered through this interview will be summarized in aggregate form and individual respondents will not be identified in the evaluation report. You will have an opportunity to review the written summary of the interview and make any corrections or additions.

1. How would you describe the working relationship between your DLSU and the Commercial Law Section on this file?

2. Based on your experience, please comment on the following dimensions of the quality of the overall services provided by the Commercial Law Section in relation to this file:

  1. its capacity to respond to your requests in a timely manner
  2. its capacity to meet mutually agreed-upon deadlines
  3. its understanding of the nature of the issue for which assistance was sought
  4. the extent to which you were provided with regular and informative progress reports or ongoing feedback in respect of your request for service
  5. the extent to which counsel involved you in the development of legal strategy and positions
  6. the extent to which you were provided with clear and practical guidance on resolving legal issues

3. To what extent did the advice provided by the Commercial Law Section on this file inform the decisions made by your DLSU and/or the client department? In your opinion, what factors affected whether the advice was considered?

4. Overall, how satisfied were you with the quality of the legal services you received from the Commercial Law Section on this file?

5. Were adequate resources assigned within the Commercial Law Section to undertake the work required (appropriate counsel and number of counsel)? Please explain any capacity issues you encountered.

6. In your opinion, was this file handled in a cost-effective manner? What, if anything, could have been done differently by either your DLSU or the Commercial Law Section to improve timeliness or to reduce costs?

7. In your view, what, if anything, could the Commercial Law Section have done differently to better serve your needs?

8. Do you have any further comments relating to this file?

Thank you for your participation.

Case Study Guide for Other Justice Counsel

The Department of Justice Evaluation Division is conducting an evaluation of the Business and Regulatory Law Portfolio (BRLP). This evaluation comes in response to the Treasury Board Policy on Evaluation, which requires government departments to evaluate all direct expenditures (including legal services) every five years. The purpose of the evaluation is to assess the extent to which the BRLP provides effective and responsive legal services to its client departments and agencies. The evaluation relies on multiple lines of evidence, including case studies that focus on specific files involving counsel from the BRLP. These case studies provide a unique opportunity to better understand the work of the Portfolio at an operational level.

The information gathered through this interview will be summarized in aggregate form and individual respondents will not be identified in the evaluation report. You will have an opportunity to review the written summary of the interview and make any corrections or additions.

Processes and tools

1. If you are aware of it, please describe the process used to engage your group’s services on this file. In your opinion, was your group engaged in a timely manner? Why or why not?

2. How were the roles and responsibilities among the various legal counsel involved in this file distributed? What role did you specifically play? In your opinion, were the roles and responsibilities adequately assigned?

3. As applicable, which processes were used to coordinate the work of legal counsel on this file? Based on your experience, what worked well with these processes and what challenges, if any, did you encounter?

Legal risk assessment

4. Were you involved in the communication of legal risk to the client department or agency? If so, please describe your involvement. Was your role appropriate in this regard?

Deployment of resources

5. In your opinion, were appropriate counsel assigned to this file, specifically with respect to a) the level and expertise of counsel assigned to the file? b) the number of counsel assigned to the file c) the existing workload of counsel assigned to the file? d) if applicable, the timeline for completion associated with the file?

6. Did you observe any duplication of work or redundancies associated with this file? If yes, can you suggest ways that this work could have been completed more efficiently?

7. In your opinion, was this file handled in a cost-effective manner? What, if anything could have been done differently by the Legal Services Unit to reduce costs?

Outcomes

8. Were you in a position to deliver your services in a timely manner? What challenges, if any, did you face in attempting to meet the time frame required?

9. If you are aware of it, please describe the impact of the legal services you provided on the decisions made by either the other counsel assigned to this file or by the client, as the case may be?

10. Please describe any best practices or lessons learned resulting from this file that could be applied to other similar files, or to any files in general, that would improve the provision of legal services in the future.

Conclusion

11. Do you have any further comments relating to this file?

Thank you for your participation.

Case Study Guide for BRLP Representatives

The Department of Justice Evaluation Division is conducting an evaluation of the Business and Regulatory Law Portfolio (BRLP). This evaluation comes in response to the Treasury Board Policy on Evaluation, which requires government departments to evaluate all direct expenditures (including legal services) every five years. The purpose of the evaluation is to assess the extent to which the BRLP provides effective and responsive legal services to its client departments and agencies. The evaluation relies on multiple lines of evidence, including case studies that focus on specific files involving counsel from the BRLP. These case studies provide a unique opportunity to better understand the work of the Portfolio at an operational level.

The information gathered through this interview will be summarized in aggregate form and individual respondents will not be identified in the evaluation report. You will have an opportunity to review the written summary of the interview and make any corrections or additions.

Processes and tools

1. How were the roles and responsibilities among the various legal counsel involved in this file distributed? What role did you specifically play? In your opinion, were the roles and responsibilities adequately assigned?

2. As applicable, which processes were used to coordinate the work of legal counsel on this file? Based on your experience, what worked well with these processes and what challenges, if any, did you encounter?

3. Were other groups within Justice Canada (scratch committees, regional or national litigation committees, specialized units from the Public Law Sector, etc.) involved in this file? If so, please describe their involvement and their impact on the file.

4. Was the Business and Regulatory Law Portfolio office (ADMO) involved in this file? If so, please describe its involvement and its impact on the file.

Legal risk assessment

5. Did you work with the client to identify and assess the legal risks on this file? What processes were used to identify and assess legal risk?

6. How was the legal risk communicated to the client department or agency? How well did the client department or agency understand the legal risks involved in the file? How did the client use the assessment?

7. What challenges, if any, did you face in assessing or communicating the legal risk associated with this file?

Deployment of resources

8. In your opinion, were appropriate counsel assigned to this file, specifically with respect to a) the level and expertise of counsel assigned to the file? b) the number of counsel assigned to the file c) the existing workload of counsel assigned to the file? d) if applicable, the timeline for completion associated with the file?

9. Did you observe any duplication of work or redundancies associated with this file? If yes, can you suggest ways that this work could have been completed more efficiently?

10. In your opinion, was this file handled in a cost-effective manner? What, if anything could have been done differently by the Legal Services Unit to reduce costs?

Outcomes

11. Were you in a position to deliver your services in a timely manner? What challenges, if any, did you face in attempting to meet the time frame required by the client department or agency?

12. How would you describe the impact of the legal services you provided on the decisions made by the client department or agency?

13. Please describe any best practices or lessons learned resulting from this file that could be applied to other similar files, or to any files in general, that would improve the provision of legal services in the future.

Conclusion

14. Do you have any further comments relating to this file?

Thank you for your participation.

File Review Guide

1. File code number:

2. Date file opened: (mm/dd/yy)

3. Date file closed: (mm/dd/yy)

4. Lead counsel (specify):

4. Case type:

5. Total number of counsel who worked on the file:

6. Total number of paralegals who worked on the file:

7. Client:

8. Notes

Questions 9 – 12 are for litigation files only.

9. BRLP is:

10. Level of court:

11. What was the outcome of the case?

12. Has the case been appealed?

If yes, remember to complete Appeal section.

Question 13 is for advisory files only.

13. What was the outcome of the file? (Check all that apply)

iCase Information – All File Types

14. Potential client impact:

15a. Risk level (1-9 or Low-Medium-High):

15b. Earlier risk level (if available through iCase)

16. Complexity

17. What was the group/level of lead counsel:

18. How many hours did lead counsel spend on the file? (hours)

19. Indicate the number of additional counsel on the file by seniority level and indicate the number of hours spent on the file:

20. Indicate the number of Justice paralegals on the file and indicate the number of hours spent on the file by each paralegal:

Information from file

21. Did legal counsel miss any client-imposed deadlines?

22. (If yes to Q21) How many times, and for what reason(s) (if an explanation to the client is available on file)?

23. Is there documented evidence that timelines were negotiated?

24. (If yes to Q23) What evidence is there in the files?

Questions 25 – 27 are for litigation files only.

25. Did legal counsel miss any court deadlines?

26. (If yes to Q25) How many times?

27. (If yes to Q25) Were additional court procedures required (e.g., motions)?

28. How did BRLP legal counsel consult with client department(s)? Consultations can include oral/written updates or oral/written discussions of possible strategies, options, approaches to the file.

29. Is there documentation in the file that shows what the client department(s) was consulted about? (Check all that apply.) (If none apply, go to Q31)

30. (If identify any categories listed in Q29) What evidence is there in the files?

31. Have client department(s) expressed any concerns relating to a lack of consultation?

32. How often did BRLP legal counsel respond to documented client requests?

33. On average, how soon after documented client requests did BRLP legal counsel respond?

34. Was the file brought to the attention of any BRLP committees?

35. (If yes to Q34) Which committees?

36. Was the file brought to the attention of other Justice officials/structures?

37. (If yes to Q36) Which ones?

38. Did counsel consult with specialized sections within Justice?

39. (If yes to Q38) Which ones:

40. What were the specialized sections consulted about? (Check all that apply):

41. Were other potentially affected departments and agencies consulted?

42. If yes to Q41, which one(s):

43. (If yes to Q41) What were they consulted about? (Check all that apply):

Questions 44 – 46 are for litigation files only.

44. At what stage were dispute resolution options considered? (Check all that apply.)

45. At what stage were dispute resolution options used?

46. What dispute resolution options were used?

Questions 47 – 50 are for advisory files only.

47. Is there documented evidence in the file that the client department considered the BRLP legal advice in program and policy development?

48. (If yes to Q47) What evidence is there in the files?

49. Is there documented evidence in the file that the client department considered BRLP legal advice to prevent, mitigate, and/or manage legal risk?

50. (If yes to Q49) What evidence is there in the files?

51. Is there documentation in the file that shows that any of the following were used/prepared?

52. Was the file included in any of the following?

Risk assessment [from file or from text fields in iCase (background, impact, and status)]

53. What legal risks are identified? Please check all that apply. Only include if there is documentation that specifies risks (in iCase or in file); do not try to interpret information (e.g., counsel indicates difficult facts in memo in file; the researcher should not make their own decision that facts are difficult). You do not have to enter risks that are already listed under potential client impact in iCase. You should include other risks that might be identified in the Background, Impact, and Status sections of iCase as well as risks identified in the paper files.

54. Is there a discussion/indication of risk level indicated in the file?

55. What is the initial (or only) risk level

56. Date of initial (or only) risk assessment:

Questions 56 – 60 are for litigation files only.

57. At what stage in the case was the initial (or only) risk assessment done?

58. Was risk reassessed?

59. At what stage in the case was the risk reassessed?

60. If case was reassessed to a higher risk level, did any of the following occur after the reassessment?

61. What language was used to describe the level of legal risk to clients? Please give verbatim examples that show the range of language used.

62. Any additional comments? (indicate applicable question number, if appropriate).

For all files where Q12 is yes (was appealed)

iCase information (from Appeal file)

63. Potential client impact:

64. Risk level (1-9):

64b. Earlier risk level (if available through iCase)

65. Complexity:

Survey of BRLP Legal Counsel

Introduction

Your response is very important to us. All information you provide is confidential and only aggregate results will be included in the evaluation report.

1. When did you first join the Department of Justice Canada?

2. How long have you worked in the Business and Regulatory Law Portfolio?

3. Where do you work?

4. (LSU counsel) Please specify the legal services unit to which you currently belong:

5. (BRLP counsel in regional offices) In which regional office are you currently located?

6. What is your classification?

7. What kinds of files do you work on most often? (Check all that apply)

8. Based on the files you have worked on over the last two years, how often were the following goals achieved?

9. Please provide your level of agreement with the following statements.

10. (LSU counsel) When you work together on a file, how effective is the collaboration with the other BRLP areas listed below? Please explain or qualify your response in the space provided below.

11. (BRLP counsel in regional offices) When you work together on a file, how effective is the collaboration with the other BRLP areas listed below? Please explain or qualify your response in the space provided below.

12. (Commercial Law Section counsel) When you work together on a file, how effective is the collaboration with the other BRLP areas listed below? Please explain or qualify your response in the space provided below.

13. When you work together on a file, how effective is the collaboration with the other areas of Justice listed below? Please explain or qualify your response in the space provided below.

14. (LSU counsel) When you work together on a file, are the roles and responsibilities clear between your DLSU and the other BRLP areas listed below? Please explain or qualify your response in the space provided below.

15. (BRLP counsel in regional offices) When you work together on a file, are the roles and responsibilities clear between your Regional Office and the other BRLP areas listed below? Please explain or qualify your response in the space provided below.

16. (Commercial Law Section counsel) When you work together on a file, are the roles and responsibilities clear between the Commercial Law Section and the other BRLP areas listed below? Please explain or qualify your response in the space provided below.

17. When you work together on a file, are the roles and responsibilities clear between your DLSU, Regional Office or Section and the other areas of Justice listed below? Please explain or qualify your response in the space provided below.

18. To what extent do you find the following tools, structures and processes useful to your work?

19. Are there other tools, structures or processes that you have found useful in managing your work?

20. In your opinion, are there other tools or support that would assist you in your work?

21. How satisfied are you with the professional development opportunities that are available to you?

22. In your opinion, do any gaps exist in the professional development opportunities available to you?

22. Have you provided BRLP-related training or information sessions over the last five years?

24. To whom did you provide this training or information session(s)? (Check all that apply)

25. Please select the topic area(s) of this training or information session(s). (Check all that apply)

26. Based on the files you have worked on over the last two years, how often...

27. Based on the files you have worked on over the last two years, how often did paralegals provide support when needed?

28. Based on the files you have worked on over the last two years, how often have paralegals been used for the following activities?

29. What factors contribute to the BRLP's ability to provide high-quality legal services?

30. What factors constrain the BRLP's ability to provide high-quality legal services?

31. What suggestions, if any, do you have for improving the legal services provided by the BRLP?

32. Do you have any other comments?