Aboriginal Affairs Portfolio Evaluation

Appendix B: Evaluation Matrix

Evaluation Issue	Evaluation Questions	Performance Indicator	Data Sources/Methods
Relevance
1. Continued Need for the AAP^{Footnote 19}	To what extent does the AAP address the needs of the Government of Canada and federal client departments and agencies?	1. Extent, scope and nature of demand for AAP national strategic coordination, litigation, Aboriginal law practice management and legal advisory services	AAP iCase reports^{Footnote 20}/analysis AAP administrative files/analysis AAP staff/interviews AAP clients/interviews
		2. Extent, scope and nature of past, current and projected legal risk related to Aboriginal issues of national/federal relevance	AAP administrative files/analysis AAP iCase reports/analysis AAP staff/interviews AAP clients/interviews
		3. Extent, scope and nature of past, current and projected legal risks related to Northern development issues of national/federal relevance	AAP files/analysis AAP iCase reports/analysis AAP staff/interviews AAP clients/interviews
		4. Monetary value of Aboriginal issues related to AAP work (e.g. monetary value of class actions, claims, settlements, government spending related to fiduciary responsibilities of Crown)	Main estimates/analysis AAP administrative files/analysis
		5. Trends (past, current, forecasts) on the extent, scope and nature of Aboriginal legal, legal policy and Aboriginal justice policy issues of national/federal relevance^{Footnote 21} that AAP are within the mandate and scope of AAP^{Footnote 22}	Environmental scan and forecasts^{Footnote 23}/analysi AAP staff/interviews AAP clients/interviews
		6. Extent to which AAP is perceived as relevant to the needs of the Government of Canada	AAP staff/interviews AAP clients /interviews
2. Alignment with Government Priorities	To what extent are the activities of the AAP aligned with 1. Government of Canada priorities and 2. The strategic priorities of the Department of Justice?	7. Extent to which the AAP activities are aligned with 1. Government of Canada priorities and 2. The strategic priorities of the Department of Justice	Speech from the Throne, Federal Budget/analysis JUS strategic priorities/analysis AAP Business Plan/analysis
3. Alignment with Federal Roles and Responsibilities	To what extent do the activities of the AAP align with federal roles and responsibilities?	8. Extent to which AAP activities are aligned with federal roles and responsibilities and legal obligations in relation to Aboriginal peoples	Constitutional/Statutory Authorities/analysis PAA/analysis Analysis
Portfolio Design
1. AAP Design	Are the mandate and objectives of AAP clear?	9. Evidence, scope and clarity of AAP mandate and objectives	AAP administrative files^{Footnote 24}/analysis AAP staff/interviews AAP clients^{Footnote 25}/interviews AAP legal counsel survey
	Are governance structures, roles and responsibilities appropriate, clear, and being fulfilled as intended?	10. Planned versus actual organizational structure and scope of implementation	AAP governance structure/analysis AAP administrative files/analysis AAP staff/interviews
		11. Appropriateness and clarity of structures, roles and responsibilities within AAP	AAP administrative files/analysis AAP staff/interviews AAP legal counsel survey
	Is the AAP structured in such a way that it is likely to achieve its objectives?	12. Appropriateness of portfolio design to support the achievement of objectives 13. Factors contributing to/detracting from the achievement of AAP objectives	AAP structure/analysis AAP administrative files/analysis AAP staff/interviews AAP legal counsel survey
2. AAP Performance Monitoring and Evaluation Capacity	Is there sufficient capacity within AAP to support ongoing quality improvement, performance measurement and evaluation?	14. Sufficiency of capacity/resources/ 15. Methods/systems to support ongoing quality improvement, performance measurement & evaluation	AAP administrative files/analysis AAP staff/interviews AAP legal counsel survey
Performance
1. Achievement of Expected Outcomes
Direct Outcomes
	To what extent has the AAP produced high quality^{Footnote 26} legal services related to Aboriginal law and northern development issues?	16. Nature and scope of AAP legal activities/ services^{Footnote 27} undertaken in relation to Government of Canada needs/requirements	AAP administrative files/analysis AAP legal files/analysis iCase reports/analysis Litigation inventory/analysis
		17. #/ files being litigated by AAP by scope of complexity and legal risk	AAP administrative files/analysis AAP legal files/analysis iCase reports/analysis Litigation inventory/analysis
		18. Level of effort dedicated to the provision of AAP legal services (by file complexity and legal risk for files related to 1.Aboriginal law and 2. northern development issues)	AAP administrative files/analysis AAP legal files/analysis iCase reports/analysis Litigation inventory/analysis
		19. Extent to which AAP clients are satisfied with the timeliness, responsiveness and overall quality of AAP legal services related to Aboriginal law and northern development issues	AAP client survey^{Footnote 27}/analysis AAP clients/ interviews or focus groups
		20. Extent to which timeliness, responsiveness and level of quality of AAP legal services is demonstrated in selected case studies	AAP legal files/analysis AAP case studies^{Footnote 28}/analysis
	To what extent is AAP contributing to enhanced capacity to deliver legal services related to Aboriginal law and northern development issues in an integrated/consistent and responsive manner?	21. #/type/nature of quality improvement efforts planned/undertaken to 1. enhance Aboriginal law practice within the Government of Canada and 2. enhance AAP capacity to deliver legal services related to Aboriginal law and northern development issues in an integrated, consistent and responsive manner	AAP staff /interviews AAP administrative files/analysis AAP legal counsel survey
		22. #/type/nature of internal structures, processes, tools and products developed/applied to enhance AAP capacity to strategically coordinate and deliver litigation and legal advisory services in an integrated, consistent and responsive manner (e.g. training, knowledge practice management tools & products)	AAP administrative files/analysis AAP staff/interviews AAP legal counsel survey
		23. Nature and extent to which new technology has been planned/applied to enhance capacity (e.g. virtual teams)^{Footnote 29}	AAP administrative files/analysis AAP staff/interviews AAP legal counsel survey
		24. #/type/nature of training sessions, presentations planned/delivered by AAP (within and outside the Portfolio)	AAP administrative files/analysis
		25. Participant experience/satisfaction with AAP training and presentation events (within and outside the Portfolio)	AAP training session feedback reports^{Footnote 30}/analysis
		26. Extent to which AAP capacity enhancements are demonstrated as effective in realizing integrated/consistent and responsive strategic coordination and services in selected case studies	AAP legal files/analysis AAP case studies/analysis
	To what extent is the AAP contributing to enhanced awareness/ understanding within the federal government, of issues, options, approaches to Aboriginal legal, legal policy, and Aboriginal Justice policy and northern development legal issues?	27. Evidence/effectiveness of AAP structures and processes that 1. Promote and 2. enhance awareness understanding internally within the Portfolio and horizontally across government e.g. AADM Joint Steering Committee, Client Claims Advisory Committee, Interdepartmental Working Groups	AAP administrative files/analysis Committee and working group minutes/analysis AAP staff/interviews AAP clients/interviews AAP case studies/analysis AAP legal counsel survey
		28. Level/nature of awareness/understanding of AAP clients of issues, options, approaches to legal, legal policy and Aboriginal Justice policy issues	AAP clients/interviews or focus groups AAP training session feedback reports^{Footnote 31}/analysis AAP legal files/analysis AAP case studies/analysis
	To what extent is AAP contributing to Aboriginal law and policy issues and claims being effectively addressed, litigated and resolved across government?	29. Extent to which clients perceive that AAP is contributing to evolution of Aboriginal law, to Aboriginal legal policy issues and to claims being effectively addressed, litigated and resolved across Government of Canada	AAP administrative files/analysis AAP client interviews /analysis AAP legal files/analysis AAP case studies/analysis Retrospective review of Aboriginal law and policy issues /analysis
		30. Extent to and nature in which AAP influence and expertise is demonstrated	AAP legal files/analysis AAP case studies/analysis
		31. Extent to and nature in which AAP role in litigation and resolution is demonstrated	AAP legal files/analysis AAP case studies/analysis
	To what extent is AAP contributing to the ability of government departments and agencies to manage their legal risks on Aboriginal issues?	32. #/% of files where legal risk is reported as high, and nature of strategies AAP identifies/advises to manage/mitigate risk	AAP litigation inventory/ analysis AAP client interviews/analysis AAP legal files/analysis AAP case studies/analysis
		33. Extent to which clients perceive AAP is contributing to the identification and assessment of legal risks related to Aboriginal issues	AAP administrative files/analysis AAP legal files/analysis AAP case studies/analysis AAP clients/interviews or focus groups
		34. Perceptions of JUS officials of the AAP contribution to legal risk management of Aboriginal issues	AAP Justice officials and other Justice officials outside the APP/interviews AAP legal counsel survey
		35. Extent to which AAP is perceived by Government decision makers as contributing to the decision-making process and to legal risk management of Aboriginal issues	AAP clients/ interviews or focus groups AAP legal files/analysis AAP case studies/analysis
Ultimate Outcome
	To what extent is AAP contributing to a fair, relevant and accessible justice system that reflects Canadian values?	36. Cumulative	Cumulative/contribution analysis
	To what extent is AAP contributing to a federal government that is supported by high quality legal services?	37. Cumulative	Cumulative/contribution analysis
2. Demonstration of Efficiency and Economy	Has the AAP resource^{Footnote 32} utilization been appropriate, in relation to the resources planned, and allocated, to activities and outputs produced and progress toward expected outcomes?	38. Budget and work plan priorities and activities in relation to outputs delivered by AAP national strategic coordination and legal risk management, litigation services, Aboriginal law practice management and legal advisory services	AAP administrative files/analysis AAP client agreements/analysis AAP staff/interviews AAP legal counsel survey
		39. Nature/extent of efforts to undertake AAP activities and deliver services cost-effectively (e.g. past, current and planned efforts)	AAP administrative files/analysis Cost Structure Working Group reports/analysis
		40. AAP human resource plans/requirements versus actual HR profile	AAP HR Needs Assessment/analysis HR plan/analysis
		41. AAP HR recruitment, deployment, engagement and retention trends	AAP HR Plan/analysis AAP staff/interviews
	Is the AAP capacity to provide national strategic coordination and legal risk management, and to deliver litigation and legal advisory services appropriate/sufficient, given the current and future demand for strategic coordination, legal risk management and service delivery?	42. Level, scope and nature of AAP activities in relation to demand for national strategic coordination, legal risk management and litigation/legal advisory services (past, current, projected)	AAP administrative files/analysis AAP staff/interviews AAP legal counsel survey
	Are AAP activities sufficiently integrated/coordinated to support the achievement of intended results?	43. Level, scope and nature of integration and coordination within AAP, JUS and with client departments	AAP administrative files/analysis AAP staff/interviews JUS staff/interviews AAP client s/interviews or focus groups Case studies/analysis AAP legal counsel survey
	Are the funding models utilized by AAP appropriate and sufficient to meet current and future demand for services?	44. Level of satisfaction that AAP and its clients have with the 1. AAP funding models generally and 2. the clients specific agreements	AAP staff/interviews AAP clients/interviews or focus groups
	Are there alternative ways of delivering the same services?	45. Trends in law practice/service delivery models (within JUS and within private sector as relevant)	AAP Cost Structure Working Group Report/analysis AAP staff/interviews AAP legal counsel survey

Footnotes

Footnote 19

Throughout this table, AAP refers to the portfolio at both national and regional levels, unless otherwise noted.

Return to footnote 19 referrer

Footnote 20

These are aggregate legal file reports. Specific legal case files will not be directly accessed by the evaluators, due to solicitor/client privilege. Where required, AAP will provide gather and required information according to the evaluation requirement (see Data Sources and Methods for more information).

Return to footnote 20 referrer

Footnote 21

This includes a range of issues related to the Government of Canada lawful obligations with respect to Aboriginal peoples, claims, settlements, etc. AAP indicated it does scanning and forecasting.

Return to footnote 21 referrer

Footnote 22

This includes issues where AAP plays a role in strategically coordinating, addressing legal risk management as well as providing litigation and legal advisory services.

Return to footnote 22 referrer

Footnote 23

AAP currently undertakes some trends analysis and forecasting; it will be important to assess this data source and determine whether supplemental scanning would be required for evaluation purposes.

Return to footnote 23 referrer

Footnote 24

AAP administrative files generally include administrative files, policy and planning documents, work plans, budgets and information management related files. Specific AAP legal file data sources, such as iCase files are noted where applicable.

Return to footnote 24 referrer

Footnote 25

AANDC is the primary, but not exclusive client. It will be important to have a full list of most relevant clients beyond AANDC, e.g. environment, natural resources.

Return to footnote 25 referrer

Footnote 26

High quality in litigation services to be defined generally as per the ongoing Justice Canada Client Feedback Survey conducted periodically by the Performance Management and Reporting Division, Management Sector; definition to be further explored in case studies, on a case-by-case basis to ensure that the AAP role in relation to Aboriginal law practice management as a whole is captured.

Return to footnote 26 referrer

Footnote 27

Legal services include litigation and legal advisory services.

Return to footnote 27 referrer

Footnote 28

A survey of AAP clients was recently conducted as part of the standard Department of Justice legal services survey Department of Justice Canada. 2011. “Department of Justice Canada Client Feedback Survey”. [AAP_Client_Feedback_Report_Final_March2011.pdf]. Supplementary interviews or focus groups will be conducted with AANDC and a sample of AAP clients to provide further probe the results of this survey, track/assess changes since that survey was completed, and provide additional qualitative information in support of the evaluation.

Return to footnote 28 referrer

Footnote 29

It should be noted that AAP has undertaken some case studies; these case studies will be reviewed as part of the evaluation, however it is likely that enhanced or supplemental case studies would be required for evaluation purposes to cover AAP services for both Aboriginal law and northern development issues. This comment applies throughout where case studies are identified. Further it should be noted that due to solicitor-client privilege, the evaluators will not have direct access to the legal case files. AAP is also indicated as responsible for data collection wherever case studies are identified. Required information will be gathered by AAP according to a data collection template to be developed by the evaluators. See Data Sources and Methods for more detailed information about the case study approach.

Return to footnote 29 referrer

Footnote 30

This could be subsumed in the indicator above, however it seemed to be important to have this as a “stand alone”

Return to footnote 30 referrer

Footnote 31

AAP training feedback reports address participant satisfaction in terms of whether expectations were met and course delivery.

Return to footnote 31 referrer

Footnote 32

AAP training feedback reports could be modified to a pre/post format, to measure changes in awareness and understanding, and also to ask not only whether participants will apply what they’ve learned, some feedback on how they will apply it.

Return to footnote 32 referrer

Date modified:: 2022-05-13

Language selection

Search

Phone Scam Warning

Aboriginal Affairs Portfolio Evaluation

Appendix B: Evaluation Matrix