Tax Law Services Portfolio Evaluation

Appendix B: Data Collection Instruments

Survey of Legal Counsel Tax Law Services Portfolio

The Department of Justice is conducting an evaluation of the Tax Law Services (TLS) Portfolio. This evaluation comes in response to the Treasury Board Secretariat's Policy on Evaluation, which requires government departments to evaluate all direct expenditures (including legal services) every five years. The Department of Justice has hired PRA Inc., an independent research company, to assist in conducting the evaluation. The purpose of the evaluation is to assess the extent to which the TLS Portfolio provides effective and responsive legal services to the Canada Revenue Agency (CRA). In addition to this survey of legal counsel, the evaluation will also rely on other lines of evidence, including a review of documents and data, key informant interviews with representation from the Department of Justice and the CRA, a file review, case studies, and focus groups.

Your response is very important to us. We want to hear from all TLS Portfolio counsel. The survey is easy and can be answered quickly. Most questions only ask you to click on the appropriate response. The questionnaire should take about 20 minutes to complete. All information you provide is confidential and will be used only to create aggregate results to be included in the evaluation report. No individual responses will be identified. We would appreciate receiving your completed questionnaire by Date.

If you have questions about the survey, please call Amy Richmond of PRA at 1-888-877-6744 (toll-free) for service in English, Francois Dumaine of PRA at 1-866-422-8468 (toll-free) for service in French, or email your questions to admin@pra.ca. If you encounter any technical problems with the questionnaire, please contact Jo-Anne Chrétien at 613-957-9610. For questions about the evaluation in general, you may contact Marilyn Doering, Evaluation Division, Department of Justice, at 613-954-3963.

Background

The following questions will be used to establish a profile of survey respondents.

1. When did you first join the Department of Justice?

2. Where do you work?

3. What is your classification level?

4. Which kind of files do you work on most often?

Consultations and Collaborations

5. Thinking of files you have been involved with in the last two years (where the CRA–Justice resolution process did not apply), how often have you or a member of the legal team…

  Frequently (80%-100% of files) Regularly (50%-79% of files) Occasionally (25%-49% of files) Rarely (1%-24% of files) Never (0%) Don’t know Not applicable to my work
a) consulted with CRA to understand the nature of the legal problem?              
b) provided CRA with regular updates/ progress reports?              
c) involved CRA in the development of legal strategies and positions?              
d) provided advice or recommendations regarding legal options?              
e) consulted with CRA about its policy and/or program objectives?              
f) worked with CRA to identify legal risks?              
g) worked with CRA to discuss the impact of legal risks?              
h) worked with CRA to discuss options to manage legal risks?              
i) consulted with specialized sections within the Department of Justice?              
j) consulted or briefed the CRA Legal Services Unit?              
k) consulted with other potentially affected departments/ agencies?              
l) briefed or reported on your files to senior management either in the TLS Portfolio, Regional Director, or Department of Justice Headquarters?              

Adverse Reporting Procedure

6. To what extent do you agree with the following statements about the Adverse Reporting Procedure?

  Strongly agree Agree Neutral Disagree Strongly disagree Don’t know Not applicable to my work
a) The process is effective in notifying senior management about significant files.              
b) The process is too onerous.              
c) The process is clear in terms of to whom to brief/report.              
d) The process is clear in terms of when to brief/report.              
e) The process is clear in terms of what to include in briefs/reports.              
f) The process creates delays in handling the file.              

Early Warning Notes

7. To what extent do you agree with the following statements about Early Warning Notes?

  Strongly agree Agree Neutral Disagree Strongly disagree Don’t know Not applicable to my work
a) The process is effective in notifying senior management about significant files.              
b) The process is too onerous.              
c) The process is clear in terms of to whom to brief/report.              
d) The process is clear in terms of when to brief/report.              
e) The process is clear in terms of what to include in briefs/reports.               
f) The process creates delays in handling the file.              

Assessment of Portfolio Work

8. How would you assess the work of the Portfolio generally in terms of the following areas?

  Excellent Above average Average Below average Poor Don't know Not applicable to my work
a) Responding to legal service requests by CRA in a timely manner.              
b) Meeting CRA deadlines.              
c) Meeting internal Department of Justice deadlines.              
d) Providing legal services in both official languages.              
e) Involving/consulting with specialized sections within the Department of Justice when appropriate.              
f) Involving/consulting with other regional offices or headquarters when appropriate.              
g) Involving/consulting with the CRA Legal Services Unit.              
h) Involving/consulting with the CRA.              
i) Involving/consulting with other potentially affected departments/agencies.              
j) Using consistent language to communicate legal risk so that CRA can understand the comparative legal risks across files.              
k) Using consistent legal risk ratings across Portfolio files.              
l) Providing timely assessments of legal risk so that it can be factored into decision making.              
m) Reassessing legal risk when factors change that affect the level of legal risk.              

Tools and Structures

9. How useful have you found the following TLS Portfolio tools or structures in managing your work?

  Very useful Useful Neutral Not very useful Not at all useful Have not used
a) Legal risk management assessment grid/matrix            
b) Effective Communication of Legal Risk (intranet tool)            
c) Fiscal Path (intranet site)            
d) Justipedia            
e) TLS Portfolio National Coordination Committees            
f) Structured review of facta/approval of facta            
g) Resolution process for cases before the Tax Court of Canada            
h) Practice directives            
i) TLS Portfolio training opportunities            
j) File assignment process            

9a. Are there other tools or structures that you have found useful in managing your work?

Please specify:

9b. Why have you not used the legal risk management assessment grid/matrix yet? (Check all that apply.)

9c. Why have you not used the Effective Communication of Legal Risk (intranet tool) yet? (Check all that apply.)

9d. Why have you not used the Fiscal Path (intranet site) yet? (Check all that apply.)

9e. Why have you not used Justipedia yet? (Check all that apply.)

9f. Why have you not used the TLS Portfolio National Coordination Committees yet? (Check all that apply.)

9g. Why have you not used the structured review of facta/approval of facta yet? (Check all that apply.)

9h. Why have you not used the resolution process for cases before the Tax Court of Canada yet? (Check all that apply.)

9i. Why have you not used the practice directives yet? (Check all that apply.)

9j. Why have you not used the TLS Portfolio training opportunities yet? (Check all that apply.)

9k. Why have you not used the file assignment process yet? (Check all that apply.)

CRA Working Groups

10. Are you involved in any CRA committees or working groups?

10a. Please specify the CRA committees or working groups with which you are involved.

10b. How useful have you found your participation in these groups to be in managing/conducting your work?

Outcomes

11. Please provide your level of agreement with the following statements. The TLS Portfolio…

  Strongly agree Agree Neutral Disagree Strongly disagree Don't know Not applicable to my work
a) takes a consistent approach to legal issues.              
b) has the structures in place to make strategic decisions regarding legal issues.              
c) has the structures in place to manage the demand for legal services.              
d) provides the Government of Canada and CRA with a whole-of-government approach to legal issues.              
e) has contributed to enhancing CRA’s understanding of legal issues and their implications.              

Dispute Resolution

12. Use of Dispute Resolution (DR) includes negotiated settlement of files, as well as other DR processes such as mediation and pre-trial settlement. Thinking of the files you have been involved with in the last two years, how often has a member of the litigation team…

  Frequently (80%-100% of files) Regularly (50%-79% of files) Occasionally (25%-49 of files) Rarely (1%-24% of files) Never (0%) Don’t know Not applicable to my work
a) considered using DR options?              
b) used neutral evaluation to try to resolve/ settle a file?              
c) used negotiation to try to resolve/ settle a file?              
d) used voluntary mediation to try to resolve/ settle a file?              
e) used court-mandated mediation to resolve/ settle a file?              

13. In your opinion, are DR processes:

CRA-Justice Resolution Process

14. In your opinion, how effective has the CRA–Justice resolution process been in facilitating early settlement of eligible files?

14a. What could be done to improve the effectiveness of the CRA–Justice resolution process?

Legal Training for CRA Officials

15. Have you been involved in training CRA officials on legal issues and risks? (Check all that apply.)

Level of Understanding of Legal Issues by CRA Officials

16. Based on your experience over the past two years, how would you assess the level of understanding of CRA officials with whom you have worked, with respect to the following?

  Excellent Above average Average Below average Poor Don't know Not applicable to my work
a) The most frequently encountered legal issues for CRA              
b) The importance of identifying legal risks              
c) The potential impacts of legal risks              
d) When to engage the TLS Portfolio              
e) When to use early dispute resolution options to avoid the need to engage the TLS Portfolio              
f) How to manage the demand for legal services generally              

Legal Risk Management

17. Thinking of files you have been involved with in the last two years, how often did CRA consider the legal advice/legal risk...

  Frequently (80%-100% of files) Regularly (50%-79% of files) Occasionally (25%-49% of files) Rarely (1%-24% of files) Never (0%) Don’t know Not applicable to my work
a) to take steps to prevent, mitigate or manage legal risk?              
b) in its discussions with the Portfolio regarding litigation strategies?              

Consultations and Collaborations

18. Thinking of files you have been involved with in the last two years (where the CRA–Justice resolution process did not apply), how often have you or a member of the legal team…

  Frequently (80%-100% of files) Regularly (50%-79% of files) Occasionally (25%-49% of files) Rarely (1%-24% of files) Never (0%) Don’t know Not applicable to my work
a) consulted with CRA to understand the nature of the legal problem?              
b) provided CRA with regular updates/ progress reports?              
c) involved CRA in the development of legal strategies and positions?              
d) provided advice or recommendations regarding legal options?              
e) consulted with CRA about its policy and/ or program objectives?              
f) worked with CRA to identify legal risks?              
g) worked with CRA to discuss the impact of legal risks?              
h) worked with CRA to discuss options to manage legal risks?              
i) consulted with specialized sections within the Department of Justice?              
j) consulted or briefed the CRA Legal Services Unit?              
k) consulted with other potentially affected departments/ agencies?              
l) briefed or reported on your files to senior management either in the TLS Portfolio, Regional Director Office, or Department of Justice Headquarters?              

Assessment of Portfolio Work

19. How would you assess the work of the Portfolio generally in terms of the following areas?

  Excellent Above average Average Below average Poor Don't know Not applicable to my work
a) Responding to legal service requests by CRA in a timely manner              
b) Meeting CRA deadlines              
c) Meeting internal Department of Justice deadlines              
d) Providing legal services in both official languages              
e) Involving/consulting with specialized sections within the Department of Justice when appropriate              
f) Involving/consulting with other regional offices or headquarters when appropriate              
g) Involving/consulting with the CRA Legal Services Unit              
h) Involving/consulting with the CRA              
i) Involving/consulting with other potentially affected departments/agencies              
j) Using consistent language to communicate legal risk so that CRA can understand the comparative legal risks across files              
k) Using consistent legal risk ratings across Portfolio files              
l) Providing timely assessments of legal risk so that it can be factored into decision making              
m) Reassessing legal risk when factors change that affect the level of legal risk              

Tools and Structures

20. How useful have you found the following TLS Portfolio tools or structures in managing your work?

  Very useful Useful Neutral Not very useful Not at all useful Have not used
a) Legal risk management assessment grid/matrix            
b) Effective Communication of Legal Risk (intranet tool)            
c) Fiscal Path (intranet site)            
d) Justipedia            
e) TLS Portfolio National Coordination Committees            
f) Practice directives            
g) TLS Portfolio training opportunities            
h) File assignment process            

20a. Are there other tools or structures that you have found useful in managing your work?

Please specify:

20b. Why have you not used the legal risk management assessment grid/matrix yet? (Check all that apply.)

20c. Why have you not used the Effective Communication of Legal Risk (intranet tool) yet? (Check all that apply.)

20d. Why have you not used the Fiscal Path (intranet site) yet? (Check all that apply.)

20e. Why have you not used Justipedia yet? (Check all that apply.)

20f. Why have you not used the TLS Portfolio National Coordination Committees yet? (Check all that apply.)

20g. Why have you not used the practice directives yet? (Check all that apply.)

20h. Why have you not used the TLS Portfolio training opportunities yet? (Check all that apply.)

20i. Why have you not used the file assignment process yet? (Check all that apply.)

CRA Working Groups

21. Are you involved in any CRA committees or working groups?

21a. Please specify the CRA committees or working groups with which you are involved.

21b. How useful have you found your participation in these groups to be in managing/conducting your work?

Outcomes

22. Please provide your level of agreement with the following statements. The TLS Portfolio…

  Strongly agree Agree Neutral Disagree Strongly disagree Don't know Not applicable to my work
a) takes a consistent approach to legal issues.              
b) has the structure in place to make strategic decisions regarding legal issues.              
c) has the structures in place to manage the demand for legal services.              
d) provides the Government of Canada and CRA with a whole-of-government approach to legal issues.              
e) has contributed to enhancing CRA’s understanding of legal issues and their implications.              

Legal Training for CRA Officials

23. Have you been involved in training CRA officials on legal issues and risks? (Check all that apply.)

Level of Understanding of Legal Issues by CRA officials

24. Based on your experience over the past two years, how would you assess the level of understanding of CRA officials with whom you have worked, with respect to the following?

  Excellent Above average Average Below average Poor Don't know Not applicable to my work
a) The most frequently encountered legal issues for CRA              
b) The importance of identifying legal risks              
c) The potential impacts of legal risks              
d) When to engage the TLS Portfolio              
e) When to use early dispute resolution options to avoid the need to engage the TLS Portfolio              
f) How to manage the demand for legal services generally              

Legal Risk Management

25. Thinking of files you have been involved with in the last two years, how often did CRA consider the legal advice/legal risk...

  Frequently (80%-100% of files) Regularly (50%-79% of files) Occasionally (25%-49% of files) Rarely (1%-24% of files) Never (0%) Don’t know Not applicable to my work
a) to take steps to prevent, mitigate or manage legal risk?              
b) in the development or implementation of CRA policies or programs?              

Management of Portfolio Files

26. Thinking of files you have been involved with in the last two years, how often …

  Frequently (80%-100% of files) Regularly (50%-79% of files) Occasionally (25%-49% of files) Rarely (1%-24% of files) Never (0%) Don’t know Not applicable to my work
a) were files completed in a timely manner without undue delays within the control of the Portfolio?              
b) were files conducted in a cost-effective manner?              
c) were files assigned to the appropriate level of counsel given the legal risk/complexity of the file?              
d) were tasks allocated appropriately (level and experience) within the team assigned to manage the file?              
e) was appropriate mentoring and/or supervision included in managing your files?              

Factors Contributing to the Provision of High-Quality Legal Services

27. What factors contribute to the TLS Portfolio's ability to provide timely, high-quality, cost-effective legal services?

Factors Constraining the Provision of High-Quality Legal Services

28. What factors constrain the TLS Portfolio's ability to provide timely, high-quality, cost-effective legal services?

Suggestions for Improvement

29. Do you have any other suggestions for how to improve the legal services provided by the TLS Portfolio?

File Review

Overview

1. File Code Number:

2. Date file opened: (mm/dd/yy)
Date file closed: (mm/dd/yy)

3. Lead counsel
Organizational unit:

4. Case type:

5. Number of counsel on file:

6. CRA Branch:

7. CRA Order:

Questions 8 and 9 are for litigation files only.

8. CRA is

9. Level of court:

10. Legal issue and brief description of the nature of the file (i.e., constitutional issue, relating to a particular section of the Income Tax Act, etc.), without waiving solicitor-client privilege. Do not reference any specific tax payer information.

Questions 11 and 12 are for litigation files only.

11. What was the outcome of the case?

12. Has the case been appealed?

Question 13 is for advisory files only.

13. What was the outcome of the file? (Check all that apply.)

iCase information (to extent available)

14. Potential client impact:

15. Risk level (1-9):

15b. Earlier risk level (if available through iCase)

16. Complexity:

17. What was the seniority level of lead counsel:

Contingent Liability files only

18. Amount claimed ($):

19. Possibility of settlement:

20. Settlement estimate ($): to

21. Amount at risk ($):

22. Contingent gain ($):

To be completed by the Business Management Section (Q23-25)

23. How many hours did lead counsel spend on the file? hours

24. Indicate the number of additional counsel on the file by seniority level and indicate the number of hours spent on the file:

1 LA0

2 LA0

3 LA0

4 LA0

5 LA0

6 LA0

7 LA0

25. Indicate the number of paralegals on the file and indicate the number of hours spent on the file by each paralegal:
Hours for

Information from file [Answer the following questions based on documents in the file]

26. Did TLS Portfolio miss any client-imposed deadlines?

27. (If yes to Q26) How many times, and for what reason(s) (if an explanation to the client is available on file)?

28. (Litigation files only) Did TLS Portfolio miss any court deadlines?

29. (If yes to Q28) How many times?

30. (If yes to Q28) Were additional court procedures required (e.g., motions)?

31. Approximately, how often did TLS Portfolio consult with CRA? Consultations can include oral/written updates or oral/written discussions of possible strategies, options, approaches to the file (please check the most appropriate choice):

32. Is there documentation in the file that shows what the CRA was consulted about? (Check all that apply.) (If none apply, go to Q34.)

33. (If you identified any categories listed in Q32) What evidence is there in the files?

34. How often did the TLS Portfolio respond to documented client requests?

35. On average, how soon after documented client requests did the TLS Portfolio respond?

36. Was the file brought to the attention of any CRA committees?

37. (If yes to Q36) Which committees?

38. Was the file brought to the attention of other Justice officials/structures?

39. (If yes to Q38) Which ones? (Check all that apply.)

40. Is there documented evidence in the file that the TLS Portfolio considered CRA policy and program objectives when developing legal strategies?

41. (If yes to Q40) What evidence is there in the files?

42. Did lead counsel work with other Tax Law Services units within Justice?

43 (If yes to Q42) Which ones?

44. How were the other units involved? (Check all that apply):

45. Did counsel consult with specialized sections within Justice?

46. (If yes to Q45) Which ones:

47. Were the specialized sections consulted about? (Check all that apply):

48. Were other potentially affected departments and agencies consulted?

49. (If yes to Q48) Were they consulted about? (Check all that apply):

Questions 50-53 are for litigation files only.

50. At what stage were dispute resolution options considered? (Check all that apply.)

51. At what stage were dispute resolution options used?

52. What dispute resolution options were used?

53. (Advisory files only) Is there documented evidence in the file that CRA considered TLS Portfolio legal advice in program and policy development?

54. (If yes to Q53) What evidence is there in the files?

55. Is there documented evidence in the file that CRA considered TLS Portfolio legal advice to prevent, mitigate, and/or manage legal risk?

56. (If yes to Q55) What evidence is there in the files?

57. Is there documentation in the file that shows that any of the following were used/prepared?

58. Was the file included in any of the following?

Risk assessment [from file or from text fields in iCase (background, impact, and status)]

59. What legal risks are identified? Please check all that apply. Only include if there is documentation that specifies risks (in iCase or in file); do not try to interpret information (e.g., counsel indicates difficult facts in memo in file; the researcher should not make their own decision that facts are difficult). You do not have to enter risks that are already listed under potential client impact in iCase (see Q14). You should include other risks that might be identified in the Background, Impact, and Status sections of iCase as well as risks identified in the paper files.

60. Is there a discussion/indication of risk level indicated in the file?

61. What is the initial (or only) risk level (1-9)?

62. Date of initial (or only) risk assessment: (mm/dd/yy)

63. (Litigation only) At what stage in the case was the initial (or only) risk assessment done?

64. Was risk reassessed?

65. (Litigation only) At what stage in the case was the risk reassessed?

66. If the case was reassessed to a higher risk level, did any of the following occur after the reassessment?

67. What language was used to describe the level of legal risk to clients? Please give verbatim examples that show the range of language used.

68. Was it consistent?

For all files where Q12 is yes (was appealed)
iCase information (from Appeal file)

69. Potential client impact:

70. Risk level (1-9):

70b. Earlier risk level (if available through iCase)

71. Complexity:

Contingent Liability files only

72. Amount claimed ($)

73. Possibility of settlement:

74. Settlement estimate ($): to

75. Amount at risk ($):

76. Contingent gain ($):