Tax Law Services Portfolio Evaluation

Appendix B: Data Collection Instruments

Key Informant Interview Guide for Department of Justice

The Department of Justice has hired PRA Inc. to conduct an evaluation of the Tax Law Services (TLS) Portfolio. One aspect of the study is key informant interviews with representatives of Canada Revenue Agency (CRA) and the Department of Justice.

The information we gather through this interview will be summarized in aggregate form. Interview notes will not be shared outside of PRA and the Department of Justice Evaluation Division. You will have an opportunity to review our written summary of the interview and make any corrections or additions.

Introduction

1. Please describe briefly your current roles and responsibilities. If you are involved in any TLS Portfolio committees or CRA committees, please include them in your response.
2. Please describe briefly your involvement with the TLS Portfolio.

Rationale and Relevance

3. Have you noticed any changes in the demand for TLS Portfolio services in the last five years? Please consider the volume, type of legal issues, complexity, and legal risk level, or any other characteristics of your work. How has the TLS Portfolio responded to these trends?
4. In your view, how have the CRA's priorities regarding tax law services changed over the last five years (e.g., a focus on appeals versus compliance)? How has the TLS Portfolio responded to meet CRA's priorities?
5. Have federal government priorities in the area of taxation law and policies changed over the past five years? How has the TLS Portfolio responded to meet federal priorities?

Performance – Effectiveness

6. As you know, the TLS Portfolio is guided by service standards in its dealings with the CRA. These standards include:
- Provision of legal services in either official language
- Courteous and respectful treatment
- Timely response to legal service requests
- Negotiation of and attention to deadlines
- Provision of clear and practical guidance on resolving legal issues
- Provision of ongoing feedback respecting CRA's requests for service
- Development of drafting options and solutions appropriate to CRA's policy and program objectives
- Early identification of means to prevent and resolve legal disputes
- Provision of advice or recommendations regarding implementation options (e.g., advisory versus legislative or regulatory) for policies and programs
In your view, are these service standards being met? In instances where the service standards are not met, please identify which are not, and what you believe are the most common contributing factors.
7. Please describe how the TLS Portfolio and the CRA work together to identify and assess legal risks. In your answer, please include TLS involvement in CRA risk committees or other committees. In your opinion, how effective is this collaboration?
8. In your opinion, has the CRA's level of awareness and understanding of legal risks improved over the last five years? If yes, to what extent do you feel this improvement is attributable to the TLS Portfolio or TLS regional sections? If no, why do you say that?
9. Based on your experience, do you think the Portfolio uses consistent language to describe legal risk? In your opinion, are the legal risk descriptions useful to the CRA, the regional office(s) involved, and the Department generally? Why or why not?
10. Do you find the Department and Portfolio's processes, tools and standards for assessing legal risk helpful? Please explain.
11. In general, are the consultations/collaboration between the CRA and the TLS Portfolio effective? Why or why not? In your response, please consider whether the consultations assist the TLS Portfolio in understanding the CRA's policy and program objectives; whether the consultations enhance the CRA's understanding of legal issues, their implications, and potential risks as well as possible options to manage those risks; and how the consultations/collaboration affect the progress and ultimate outcome of the file.
12. In your experience, to what extent is the advice provided by the TLS Portfolio considered by the CRA in developing legal strategies and making decisions? What factors make it more or less likely that advice will be considered?
13. In general, are the consultations/collaboration between the TLS Portfolio and the specialized sections in Justice effective? Why or why not? In your response, please think of the various ways the Portfolio and the specialized sections may work together such as identifying and assessing legal risks, managing and mitigating legal risks, determining litigation strategies, and assessing legal options.
14. How well does the coordination/consultation within the Portfolio work, including across regional offices and between regional offices and headquarters? Please explain.
15. In your view, how well does the briefing process work (e.g., in terms of clarity of the process and level of effort required)? In your answer, please consider the processes relevant to your work (e.g., Early Warning Notes, Adverse Reporting Procedure, process through the National Coordination Network). Are senior managers/officials in Justice made sufficiently aware of complex and high profile cases? Please explain.
16. How helpful are the following Portfolio tools and structures in managing the Portfolio's files?
- National Coordinating Committees
- Resources on the Fiscal Path
- Justipedia
- Documents from the Good Practices and Communication Committee
- Practice directives
- Structured review and approval of facta
- Resolution process for cases before the Tax Court of Canada
- File assignment process
17. What is your opinion of the current training provided to TLS Portfolio counsel? Are any training needs not being met?
18. Please describe the strategies used by the TLS Portfolio to ensure a nationally coordinated approach to legal issues. In your opinion, are these strategies effective? What, in your opinion, is the purpose of/need for a national approach in the provision of legal services to the CRA?
19. What factors contribute to or constrain the TLS Portfolio's ability to provide timely, high-quality, cost effective legal services?

Performance – Efficiency and Economy

20. In your opinion, are adequate resources (e.g., human, financial, technological) in place to support the work of the TLS Portfolio? How does the TLS Portfolio manage resource challenges? (Regional Directors) How do the TLS regional sections manage their resource challenges?
21. In your opinion, what role does the CRA play in improving the effectiveness, efficiency and economy of legal services? In your response, please consider issues such as:
- The role and responsibility of the CRA in managing demand for legal services
- The involvement of the CRA in TLS Portfolio practice groups and committees
- The involvement of TLS Portfolio counsel in CRA risk committees and other committees
- The use of early dispute resolution or alternative dispute resolution processes
- The timeliness in which the CRA involves the TLS Portfolio in files
22. In your opinion, are the legal services provided by the TLS Portfolio cost-effective? In your response, please consider issues such as:
- Use of alternative dispute resolution practices and early resolution strategies, when appropriate
- The hours spent on a case/file in proportion to legal risk and/or complexity
- Availability of counsel at the appropriate level for the file risk/complexity
- Other tools or practices used to reduce the cost of legal services
23. What, if any, suggestions do you have for improving the efficiency or cost effectiveness of legal services provided by the TLS Portfolio?

Alternatives

24. In your opinion, how would a private sector approach to the provision of legal services to the CRA differ from the TLS Portfolio approach? What are the advantages and disadvantages of using the TLS Portfolio to provide services to the CRA, as opposed to using a private law firm? Do the advantages outweigh any disadvantages? Why or why not?

Conclusion

Thank you. We greatly appreciate your participation.

Key Informant Interview Guide for Canada Revenue Agency

Introduction

1. Please describe briefly your current roles and responsibilities. How do they relate to the TLS Portfolio?

Rationale and Relevance

2. Have you noticed any changes in the demand for TLS Portfolio services in the last five years? Please consider the volume, type of legal issues, complexity, and legal risk level, or any other characteristics of your work. How has the TLS Portfolio responded to these trends?
3. In your view, how have the CRA's priorities regarding tax law services changed over the last five years (e.g., a focus on appeals versus compliance)? How has the TLS Portfolio responded to meet CRA's priorities?
4. Have federal government priorities in the area of taxation law and policies changed over the past five years? How has the TLS Portfolio responded to meet these priorities?

Performance – Effectiveness

5. As you know, the TLS Portfolio is guided by service standards in its dealings with the CRA. These standards include:
- Provision of legal services in either official language
- Courteous and respectful treatment
- Timely response to legal service requests
- Negotiation of, and attention to, deadlines
- Provision of clear and practical guidance on resolving legal issues
- Provision of ongoing feedback respecting CRA's requests for service
- Development of drafting options and solutions appropriate to CRA's policy and program objectives
- Early identification of means to prevent and resolve legal disputes
- Provision of advice or recommendations regarding implementation options (e.g., advisory versus legislative or regulatory) for policies and programs
In your view, are these service standards being met? In instances where the service standards are not met, please identify which are not, and what you believe are the most common contributing factors.
6. Please describe how/if the TLS Portfolio and the CRA work together to identify and assess legal risks, and how/if they work together to develop options to manage or mitigate those risks. In your answer please include TLS involvement in CRA risk committees or other committees. In your opinion, how effective is this collaboration?
7. In your opinion, has the CRA's level of awareness and understanding of legal risks improved over the last five years? If yes, to what extent do you feel this improvement is attributable to the TLS Portfolio? If no, why do you say that? If you are aware of the training provided by the TLS Portfolio, please include your assessment of it in your response.
8. How satisfied are you with how legal risks are communicated to the CRA in terms of timeliness, clarity and consistency? In your opinion, are the legal risk descriptions useful to the CRA? Why or why not?
9. In general, are the consultations/collaboration between the CRA and the TLS Portfolio (or TLS regional sections) effective? Why or why not? In your response, please consider whether the consultations assist the TLS Portfolio in understanding the CRA's policy and program objectives; whether the consultations enhance the CRA's understanding of legal issues, their implications and potential risks; and whether the consultations/collaboration affect the progress and ultimate outcome of the file.
10. In your experience, to what extent is the advice provided by the TLS Portfolio (or TLS regional sections) considered in the legal strategies pursued and decisions made by the CRA? What factors make it more or less likely that advice will be considered?
11. In your experience, how well coordinated is the TLS Portfolio in working on matters that involve more than one CRA office and/or multiple regional offices within the Department of Justice?
12. How does the TLS Portfolio (or TLS regional sections) support the briefing process within CRA? Are senior managers/officials in CRA made sufficiently aware of complex and high profile cases? Please explain.
13. In your view, how important to the CRA is a nationally coordinated approach to legal services? Is the TLS Portfolio successful in ensuring a nationally coordinated approach to legal services? Why or why not?

Performance – Efficiency and Economy

14. In your view, how accurate is the CRA's approach to forecasting demand for legal services? Are the measures that the CRA currently has in place to manage demand for legal services effective? Why or why not?
15. Does the TLS Portfolio have adequate resources (e.g., human, financial, technological) in place to support the CRA's Portfolio-related work? What, if any, resource challenges has the CRA encountered in working with the TLS Portfolio?
16. Based on your experience, are appropriate counsel assigned to files (in terms of counsel years of experience and areas of expertise, and the level of complexity of files)? Please explain.
17. In what ways (if any) does the CRA contribute to the effectiveness, efficiency, and economy of legal services? In your response, please consider issues such as:
- The degree to which the CRA is involved in TLS Portfolio practice groups, committees, and working groups
- The involvement of TLS Portfolio counsel in CRA risk committees and other committees
- The use of early dispute resolution or alternative dispute resolution processes, including the degree to which the CRA tries to resolve disputes prior to involving the TLS Portfolio or regional sections
- The degree to which the CRA involves the TLS Portfolio in files in a timely manner
18. In your opinion, are the legal services provided by the TLS Portfolio cost effective? In your response, please consider issues such as:
- Use of alternative dispute resolution practices and early resolution strategies, when appropriate
- Whether the time spent on a case/file is in proportion to legal risk and/or complexity
19. What factors contribute to or constrain the TLS Portfolio's ability to provide timely, high-quality, cost effective legal services?
20. What, if any, suggestions do you have for improving the efficiency or cost effectiveness of legal services engaged by the CRA?

Alternatives

21. In your opinion, how would a private sector approach to the provision of legal services to the CRA differ from the TLS Portfolio approach? What are the advantages and disadvantages of using the TLS Portfolio to provide services to the CRA, as opposed to using a private law firm? Do the advantages outweigh any disadvantages? Why or why not?

Conclusion

22. Do you have any other comments?

Thank you. We greatly appreciate your participation.

Key Informant Interview Guide for Finance –Tax Counsel Division

The Department of Justice has hired PRA Inc. to conduct an evaluation of the Tax Law Services (TLS) Portfolio. One aspect of the study is key informant interviews with representatives of the Department of Finance Legal Services Unit as well as the Department of Justice Litigation Branch and Public Law Sector.

Not all questions will be relevant to your work. We have noted below which questions we believe to be applicable to each group. However, please feel free to answer any of the questions.

Introduction

1. Please describe briefly your current roles and responsibilities. What type of work do you do with the TLS Portfolio?

Performance — Effectiveness

2. Please describe how/if the TLS Portfolio and your group work together. Are risks discussed? If not, why?
3. If so, to what extent has this collaboration affected the level of awareness and understanding of legal risks in matters where the Department of Finance and the Canada Revenue Agency have a shared interest? In your response, please consider the effects on the level of awareness and understanding of legal risks for the Finance LSU, the TLS Portfolio, and their client departments/agencies. We understand that you may only be able to comment on some of these groups.
4. In areas where the Finance LSU and the TLS Portfolio work collaboratively (e.g., amendments to legislation and regulations, policy amendments), how satisfied are you with the way legal risks are communicated between the Finance LSU and the TLS Portfolio in terms of timeliness, clarity and consistency?
5. In general, are the consultations/collaboration between your group and the TLS Portfolio effective? Why or why not? In your response, please consider whether the consultations enhance the understanding of legal issues, their implications and potential risks; and whether the consultations/collaboration affect the progress and ultimate outcome of the file.
6. Please describe the type and frequency of consultations that occur between your group and the TLS Portfolio. Based on your experience, do they occur when they should? Are they effective? How, if anything, could the TLS Portfolio improve these consultations?
7. Do you have anything you would like to add about your group's relationship with the TLS Portfolio?

Thank you. We greatly appreciate your participation.

Key Informant Interview Guide for the Department of Justice Public Law Sector

The Department of Justice has hired PRA Inc. to conduct an evaluation of the Tax Law Services (TLS) Portfolio. One aspect of the study is key informant interviews with representatives of the Department of Finance Legal Services Unit as well as the Department of Justice Litigation Branch and Public Law Sector.

Not all questions will be relevant to your work. We have noted below which questions we believe to be applicable to each group. However, please feel free to answer any of the questions.

Introduction

1. Please describe briefly your current roles and responsibilities. What type of work do you do with the TLS Portfolio?

Rationale and Relevance

2. Have you noticed any changes in the demand from the TLS Portfolio in the last five years? Please consider the volume, type of legal issues, complexity and legal risk level, or any other characteristics of your work.

Performance — Effectiveness

3. In general, are the consultations/collaboration between your group and the TLS Portfolio effective? Why or why not? In your response, please consider whether the consultations enhance the understanding of legal issues, their implications and potential risks; and whether the consultations/collaboration affect the progress and ultimate outcome of the file.
4. Please describe the type and frequency of consultations that occur between your group and the TLS Portfolio. Based on your experience, do they occur when they should? Are they effective? How, if anything, could the TLS Portfolio improve these consultations?
5. In your experience, to what extent is the advice provided by your group considered in the development of legal strategies by the TLS Portfolio? Are there any systemic barriers to advice being incorporated into decision making? What factors make it more or less likely that advice will be incorporated?
6. Do you have anything you would like to add about your group's relationship with the TLS Portfolio?

Thank you. We greatly appreciate your participation.

Key Informant Interview Guide for the Department of Justice Litigation Branch

The Department of Justice has hired PRA Inc. to conduct an evaluation of the Tax Law Services (TLS) Portfolio. One aspect of the study is key informant interviews with representatives of the Department of Finance Legal Services Unit as well as the Department of Justice Litigation Branch and Public Law Sector.

Not all questions will be relevant to your work. We have noted below which questions we believe to be applicable to each group. However, please feel free to answer any of the questions.

Introduction

1. Please describe briefly your current roles and responsibilities. What type of work do you do with the TLS Portfolio?

Rationale and Relevance

Performance — Effectiveness

2. In general, are the consultations/collaboration between your group and the TLS Portfolio effective? Why or why not? In your response, please consider whether the consultations enhance the understanding of legal issues, their implications and potential risks; and whether the consultations/collaboration affect the progress and ultimate outcome of the file.
3. Please describe the type and frequency of consultations that occur between your group and the TLS Portfolio. Based on your experience, do they occur when they should? Are they effective? How, if anything, could the TLS Portfolio improve these consultations?
4. How does the TLS Portfolio support the briefing process within Justice? Are senior managers in Justice made sufficiently aware of complex and high profile cases? Please explain. (Prompt: include in your response the TLS Portfolio's participation in the National Litigation Committee).
5. In your view, how important is a nationally coordinated approach to litigation services in Justice? Is the TLS Portfolio successful in contributing to a nationally coordinated approach to litigation? Please explain.
6. Do you have anything you would like to add about your group's relationship with the TLS Portfolio?

Thank you. We greatly appreciate your participation.

Case Study Guide for the Department of Justice

The Department of Justice has hired PRA Inc. to conduct an evaluation of the Tax Law Services (TLS) Portfolio. One aspect of the study is an in-depth review of three files to provide more detailed information of the TLS Portfolio's approach to managing files. Each case study includes interviews with representatives of the Canada Revenue Agency (CRA) and the Department of Justice, as well as a review of the file, which is conducted by TLS Portfolio staff.

1. What was your role on the file that we will be discussing today?
2. If you are aware of it, please describe the process used to engage the TLS Portfolio's legal services in this file. In your opinion, was the Portfolio engaged in a timely manner? Why or why not?

Legal Risk Management

3. Were you involved in the identification and assessment of legal risk on the file? If yes, were others from the Portfolio, Justice regional offices, specialized sections of Justice, and/or the CRA consulted to assist in the identification and assessment of the legal risk? How were they involved? Was this involvement effective in identifying and assessing the legal risks?
4. To the best of your recollection, how were the legal risks described in this file in terms of risk level or other language used? Based on your experience on this file, do you think the Portfolio uses consistent language to describe legal risk? In your opinion, were the legal risk descriptions useful to the CRA? Why or why not?
5. In this file, how was TLS Portfolio legal advice used to manage and mitigate legal risks? In your response, consider how the TLS Portfolio responded to manage and mitigate legal risks and how the CRA used TLS Portfolio (or TLS regional section) advice to manage and mitigate legal risks.
6. Did you find the Department's and Portfolio's processes, tools, and standards for assessing legal risk helpful?
7. Was a contingency plan developed for this file? Why or why not? Was the Portfolio consulted in the preparation of the contingency plan? If so, to what extent?

Consultation and Communication

8. How often and for what purposes did the TLS Portfolio consult with the CRA on this file? To what extent was the CRA involved in the development of legal strategy and positions?
9. Was the file brought to the attention of CRA risk committees or other committees? Why or why not? To what extent, if any, did committee discussions concerning the file affect the approach or strategy taken by the CRA or the TLS Portfolio in dealing with the file?
10. In general, was the consultation/collaboration between the CRA and the TLS Portfolio on this file effective? Why or why not? In your response, please consider whether the consultations assisted the TLS Portfolio in understanding the CRA's policy and program objectives; whether the consultations enhanced the CRA's understanding of legal issues, their implications, and potential risks; and whether the consultations/collaboration affected the progress and ultimate outcome of the file.
11. Were any of the specialized sections of Justice consulted on this file? Please explain why the specialized sections were consulted, and what value, if any, this consultation brought to the file.
12. Were other Justice regional offices consulted on this file? Why or why not? In your opinion, what value, if any, did this consultation bring to this file?
13. What other offices and structures within the TLS Portfolio were involved in the file, if any? How well did the coordination/consultation within the Portfolio work? Was your experience in this file consistent with the TLS Portfolio's national approach to legal issues and legal service delivery? Please explain.
14. What, if any, briefing or reporting was done on this file (e.g., Early Warning Notes, Adverse Reporting Procedure, reporting through the National Coordination Network)? If none occurred, please explain why. If briefing occurred, please consider who was briefed, and when and why they were briefed. In your view, how well did the briefing process work (e.g., in terms of clarity of the process and level of effort required)? Were senior managers/officials made sufficiently aware of this file? Please explain.

Resources

15. What, if any, resources challenges (human, financial, technical) did you experience in your work on this file? How were these challenges managed?
16. Did you use any of the following Portfolio tools and structures on this file and, if so, were they helpful in managing the file and obtaining a satisfactory result?
- National Coordinating Committees
- Resources on the Fiscal Path
- Justipedia
- Documents from the Good Practices and Communication Committee
- Practice directives
- Structured review and approval of facta
- Resolution process for cases before the Tax Court of Canada

Results

17. (For litigation files only) Were early resolution strategies or alternative dispute resolution strategies considered or used in this file? Why or why not? If they were used, what caused them to be successful or unsuccessful in resolving the file?
18. In your opinion, was this file handled in a cost-effective manner? What, if anything, could have been done differently by the TLS Portfolio and/or the CRA to reduce costs?
19. To what extent was the legal advice provided by the TLS Portfolio considered in the legal strategies pursued and decisions made by the CRA in this file? Please explain.
20. Do you believe that the Portfolio provided timely, responsive, high-quality legal services on this file? Why or why not? What feedback, if any, did the TLS Portfolio team receive from the CRA on your legal services?
21. Are there any best practices/lessons learned from this file that you would like to share?
22. Based on your legal experience, how do the TLS Portfolio's legal services differ from what the private sector would provide the client? Please answer based on this file and your work with the TLS Portfolio more generally.

Conclusion

23. Do you have any other comments?

Thank you. We greatly appreciate your participation.

Case Study Guide for Canada Revenue Agency

1. What was your role on the file that we will be discussing today?
2. If you are aware of it, please describe the process used to engage the TLS Portfolio's legal services in this file. In your opinion, was the Portfolio engaged in a timely manner? Why or why not?

Legal Risk Management

3. Were you involved in the identification and assessment of legal risk on the file? If yes, were others from the Portfolio, regional offices, specialized sections of Justice, and/or the CRA consulted to assist in the identification and assessment of the legal risk? How were they involved? Was this involvement effective in identifying and assessing the legal risks?
4. To the best of your recollection, how were the legal risks described in this file in terms of risk level or other language used? Based on your experience on this file, do you think the Portfolio uses consistent language to describe legal risk? In your opinion, were the legal risk descriptions useful to the CRA? Why or why not?
5. In this file, how was TLS Portfolio legal advice used to manage and mitigate legal risks? In your response, consider how the TLS Portfolio responded to manage and mitigate legal risks and how the CRA used TLS Portfolio (or TLS regional section) advice to manage and mitigate legal risks.
6. Was a contingency plan developed for this file? Why or why not? Was the Portfolio consulted in the preparation of the contingency plan? If so, to what extent?

Consultation and Communication

7. How often and for what purposes did the TLS Portfolio consult with the CRA on this file? To what extent was the CRA involved in the development of legal strategy and positions?
8. Was the consultation/collaboration between the CRA and the TLS Portfolio on this file effective? Why or why not? In your response, please consider whether the consultations assisted the TLS Portfolio in understanding the CRA's policy and program objectives; whether the consultations enhanced the CRA's understanding of legal issues, their implications, and potential risks; and whether the consultations/collaboration affected the progress and ultimate outcome of the file.
9. Did the TLS Portfolio and the CRA work across regional offices on this file? If so, in your view, was the collaboration between regional offices effective? Why or why not? To what extent would you say that national, as opposed to regional, legal issues and legal service delivery was maintained? Please explain.
10. Was the file brought to the attention of CRA risk committees or other committees? Why or why not? To what extent, if any, did committee discussions concerning the file affect the approach or strategy taken by the CRA or the TLS Portfolio in dealing with the file?
11. To your knowledge, what, if any, briefing or reporting was done on this file? If none occurred, please explain why. If briefing occurred, did the TLS Portfolio (or TLS regional sections) support the briefing process within CRA? Were senior managers/officials in CRA made sufficiently aware of this file? Please explain.

Resources/Counsel

12. In your opinion, were appropriate counsel assigned to this file (considering the years of experience of the counsel and the level of complexity of the file)? Did the TLS Portfolio assign adequate resources to undertake the work required? Please explain. Were you aware of any other resources challenges (human, financial, technical) in your work on this file? How were these challenges managed?

Results

13. (For litigation files only) Were early resolution strategies or alternative dispute resolution strategies considered or used in this file? Why or why not? If they were used, what caused them to be successful or unsuccessful in resolving the file?
14. Have you ever worked on a legal matter for the CRA with the private bar? Based on your experience(s), what are the advantages and disadvantages of a private sector approach, as opposed to the TLS Portfolio approach to legal services? If this file had been handled by the private bar, as opposed to the TLS Portfolio, do you think that it would have been handled better, worse, or similarly? Why or why not?
15. In your opinion, was this file handled in a cost-effective manner? What, if anything, could have been done differently by the TLS Portfolio and/or the CRA to reduce costs?
16. To what extent was the legal advice provided by the TLS Portfolio considered in the legal strategies pursued and decisions made by the CRA in this file? Please explain.
17. Do you believe that the Portfolio provided timely, responsive, high-quality legal services on this file? Why or why not? In your opinion, did the CRA provide feedback to the TLS Portfolio on the quality of its legal services?
18. Are there any best practices/lessons learned from this file that you would like to share?

Conclusion

19. Do you have any other comments?

Thank you. We greatly appreciate your participation.

Date modified:: 2022-05-13