Tax Law Services Portfolio Evaluation

Appendix A: Evaluation Matrix

Relevance
Issues/Questions Indicators Data Sources Timing of Data Collection Responsibility for Data Collection
1. Is there a continued need for TLS Portfolio legal services?
TB core issue 1
  • Trends in demand for TLS Portfolio services
    • Number of active files by fiscal year and type of service (litigation, advisory, other)
    • Total hours of legal services by fiscal year and type of service (litigation, advisory, other)
  • Legal practice trends (type of legal issues by fiscal year)
  • Trends in file complexity and risk level by fiscal year and type of service (litigation, advisory, other)
 Document review (e.g., iCase) Ongoing TLS Portfolio
  • Legislative and policy requirements
  • Evolving legal needs in the area of taxation
 Document review Evaluation Evaluation Division
Assessment of continued need for TLS Portfolio services in all current areas of practice Key informant interviews Evaluation Evaluation Division
2. Does the delivery of legal services by the TLS Portfolio continue to respond to federal government priorities?
TB core issue 2
  • Alignment of the TLS Portfolio mandate and activities with federal priorities/policy commitments in the area of taxation
  • Assessment of how the TLS Portfolio contributes to the priorities and objectives of the federal government
  • Document review
  • Key informant interviews
 Evaluation Evaluation Division
3. Is the provision of legal services by the Portfolio consistent with the Department of Justice Act and the departmental mandate?
TB core issues 2 and 3 		Assessment of extent to which TLS Portfolio legal services are consistent with the Department of Justice Act and the departmental mandate Document review Evaluation Evaluation Division
Performance of the TLS Portfolio (effectiveness): TB core issue 4
Immediate Outcomes
Issues/Questions Indicators Data Sources Timing of Data Collection Responsibility for Data Collection
4. Does the TLS Portfolio provide timely, responsive high-quality litigation services and legal advice?
  • Compliance with TLS Portfolio service standards
  • Assessments by the TLS Portfolio and the CRA on litigation services and legal advice in terms of:
    • clarity
    • timeliness
    • meeting deadlines
    • understanding nature of problem
    • providing updates/progress reports
    • involving the CRA in development of legal strategy and positions
    • developing legal strategies appropriate to CRA policy and/or program objectives
    • effectively working with the CRA to identify and manage legal risks
    • overall quality
 Document review (Client Feedback Survey; semi-annual reports on client feedback) Ongoing TLS Portfolio
  • Key informant interviews
  • Legal counsel survey
  • File review
  • Case studies
  • Focus groups
 Evaluation Evaluation Division
5. Are legal risks identified and assessed in a timely and consistent manner? Use of processes, tools and standards for assessing legal risk
  • Document review
  • Key informant interviews
  • Legal counsel survey
  • File review
  • Case studies
 Evaluation Evaluation Division
  • Number and percentage of files "unable to assess" and length of time before assessment entered
  • Average number of risk assessments/reassessments per file
  • Document review (iCase reports)
  • File review
  • Case studies
 Evaluation Evaluation Division
  • Consistency of language used in assessing and communicating legal risk
  • Consistency of risk ratings
  • Timeliness of assessments/reassessments of legal risk
  • Key informant interviews
  • Legal counsel survey
  • File review
  • Case studies
  • Focus groups
 Evaluation Evaluation Division
6. Has the TLS Portfolio contributed to the enhanced understanding of legal issues, their implications, and potential risks by the CRA?
  • Level of TLS Portfolio involvement in training of appropriate CRA officials on legal issues and risks
  • Nature and extent of TLS Portfolio involvement on regional risk committees and Tri-departmental Risk Committee
  • Assessment of the TLS Portfolio contribution to improved understanding of legal issues and legal risks by the CRA
  • Document review
  • Key informant interviews
  • Legal counsel survey
  • File review
  • Case studies
  • Focus groups
 Evaluation Evaluation Division
7. Do counsel have the appropriate competencies, tools, structures and resources to conduct their work?
  • Nature and type of tools and structures
    • Fiscal path
    • Best practices
    • Practice directives
    • Teams
    • Committees
    • Structured review and approval of facta
    • Resolution process for cases before the Tax Court of Canada
  • Use of tools and structures
  • Satisfaction with tools, structures and resources
  • Document review
  • Key informant interviews
  • Legal counsel survey
  • File review
  • Case studies
 Evaluation Evaluation Division
  • Nature and frequency of training
  • Number of counsel and support staff trained
 Document review Ongoing TLS Portfolio
Satisfaction with training
  • Key informant interviews
  • Legal counsel survey
 Evaluation Evaluation Division
Level and sufficiency of resources (human, financial, information, training, technological) to support TLS Portfolio work
  • Document review
  • Key informant interviews
  • Legal counsel survey
 Evaluation Evaluation Division
8. Does the TLS Portfolio strategically manage legal issues?
  • Provision of guidance to counsel of a national approach to legal issues
  • Consistency in approach to legal issues
  • Assessment of the Portfolio's ability to make strategic decisions regarding legal issues and strategies
 Document review Ongoing TLS Portfolio
  • Key informant interviews
  • Legal counsel survey
  • Case studies
  • Focus groups
 Evaluation Evaluation Division
9. What is the role of the CRA in the effectiveness of legal services? Involvement of the CRA in TLS Portfolio practice groups and committees
  • Document review
  • Key informant interviews
  • Legal counsel survey
  • Case studies
  • Focus groups
 Evaluation Evaluation Division
Intermediate Outcomes
Issues/Questions Indicators Data Sources Timing of Data Collection Responsibility for Data Collection
10. Is the TLS Portfolio able to achieve appropriate resolution of litigation cases in a timely, cost-effective manner?
  • Cost of legal services relative to file outcome (success rate), impact and tax amount
  • Hours spent per closed file by complexity and legal risk
  • Number and percentage of files settled/closed without going to court
  • Results of tracking closed litigation files for the potential need for management action
 Document review (iCase) Ongoing TLS Portfolio
  • Use and effectiveness of strategies to promote timely, cost-effective resolution
  • Early resolution strategies
  • Alternative DR practices, where appropriate
  • Assessment of whether litigation cases are resolved in a timely, cost-effective manner
  • Key informant interviews
  • Legal counsel survey
  • File review
  • Case studies
  • Focus groups
 Evaluation Evaluation Division
11. Does the TLS Portfolio protect the interests of the Crown?
  • Counsel work collaboratively within Justice
  • Counsel consult with specialized sections within Justice, where appropriate
  • Assessment that an integrated, whole-of-government approach is used in provision of legal services
  • Document review
  • Key informant interviews
  • Legal counsel survey
 Ongoing TLS Portfolio
  • File review
  • Case studies
  • Focus groups
 Evaluation Evaluation Division
12. Is the TLS Portfolio legal advice considered in program and tax law policy development and decision-making?
  • Active participation of the Portfolio in CRA committees and management structures
  • Consideration of TLS Portfolio legal advice by decision-makers in program and policy development
  • TLS Portfolio and CRA assessment of legal advice in terms of:
    • involving the CRA in development of legal strategy and positions
    • developing legal strategies appropriate to CRA policy and/or program objectives
  • Document review (Client Feedback Survey)
  • Key informant
  • Legal counsel survey
  • File review
  • Case studies
  • Focus groups
 Evaluation Evaluation Division
13. Are legal risks managed and mitigated effectively?
  • Consideration of legal risk in the development and implementation of CRA policies, programs and litigation strategies
  • Use of legal advice to prevent, mitigate and/or manage legal risk
  • Use of contingency plans for high-risk files
  • Document review (Client Feedback Survey, iCase)
  • Key informant interviews
  • Legal counsel survey
  • File review
  • Case studies
  • Focus groups
 Evaluation Evaluation Division
14. Is demand for legal services met in a responsive and cost-effective manner?
  • Mix of legal resources (FTEs/hours by group and level) by file complexity and level of risk
  • Resources are allocated in line with Portfolio Business Plan and Service Agreement with the CRA
 Document review (iCase reports) Ongoing TLS Portfolio
  • Measures in place to manage demand for legal services and their effectiveness
  • Assignment of counsel to files based on complexity and level of risk
  • Assessment of the Portfolio's ability to meet demand for legal services in terms of:
    • Human resources
    • Financial resources
    • CRA priorities
  • Document review
  • Key informant interviews
  • Legal counsel survey
  • File review
  • Case studies
  • Focus groups
 Evaluation Evaluation Division
Ultimate Outcomes
Issues/Questions Indicators Data Sources Timing of Data Collection Responsibility for Data Collection
15. To what extent does the TLS Portfolio contribute to a federal government that is supported by effective and responsive legal services? Contribution of the TLS Portfolio to provision of effective, responsive legal services
  • Document review
  • Key informant interviews
  • Legal counsel survey
  • File review
  • Case studies
  • Focus groups
 Evaluation Evaluation Division
16. What factors are contributing to or constraining the achievement of expected outcomes?
  • Constraints on ability to achieve outcomes
  • Factors supporting achievement of outcomes
 Document review
  • Key informant interviews
  • Legal counsel survey
  • Case studies
  • Focus groups
 Evaluation Evaluation Division
Performance of the TLS Portfolio (efficiency, economy): TB core issue 5
Issues/Questions Indicators Data Sources Timing of Data Collection Responsibility for Data Collection
17. Is the TLS Portfolio facing any challenges in terms of available funding, organizational structure/staffing, or internal support (training, technological, research) that are affecting its ability to achieve its expected outcomes?
  • Forecasted resource needs compared to available resources
  • Identified resource challenges
  • Assessment of the effectiveness of the TLS Portfolio organizational structure
  • Document review
  • Key informant interviews
  • Legal counsel survey
  • Case studies
  • Focus groups
 Evaluation Evaluation Division
18. Could the work of the TLS Portfolio be undertaken/ conducted more efficiently and economically?
  • Mix of legal resources (FTEs/hours by group and level) by file complexity and level of risk
  • Hours by types of files
  • Cost by types of files
  • Standard of 1300 hours of legal services per FTE is met
  • Caseload per FTE by complexity
  • Contingent liability trends
 Document review (iCase reports) Ongoing TLS Portfolio
  • Measures in place to manage efficiency and cost of legal services and their effectiveness (e.g., Law Practice Model; practice groups; use of standardized processes, practices, approach to assigning work and determining when to outsource, information management tools such as iCase)
  • Suggestions for improvements in efficiency
  • Key informant interviews
  • Legal counsel survey
  • Case studies
  • Focus groups
 Evaluation Evaluation Division
  • Alternative approaches to service delivery
  • Trends in law practice/service delivery models
 Literature review Evaluation Evaluation Division
19. What is the role of the CRA in the efficiency and economy of legal services?
  • Role and responsibility of the CRA in managing demand for legal services
  • Involvement of the CRA in TLS Portfolio practice groups and committees
  • Assessment of the CRA's effectiveness in managing its demand for legal services
  • Number and proportion of files settled in early DR by the CRA that do not require TLS Portfolio services
  • Timely involvement of the TLS Portfolio by the CRA
  • Document review
  • Key informant interviews
  • Legal counsel survey
  • Case studies
  • Focus groups
 Evaluation Evaluation Division