Audit of Legal Knowledge Management

Approved June 1, 2021
Internal Audit Branch
Internal Audit and Evaluation Sector

Table of contents

• 1 Executive Summary
  • 1.1.1 Introduction
  • 1.1.2 Strengths
  • 1.1.3 Areas for Improvement
  • 1.1.4 Audit Opinion and Conclusion
  • 1.1.5 Management Response
• 2 Statement of Conformance
• 3 Acknowledgement
• 4 Background
  • 4.1 What is Legal Knowledge Management
  • 4.2 Why is LKM Important
  • 4.3 LKM Responsibilities at Justice
• 5 Audit Objective
• 6 Audit Scope
• 7 Audit Approach
• 8 Findings, Recommendations and Management Response and Action Plan
  • 8.1 Governance
  • 8.2 Whole-of-Justice Approach
• 9 Audit Opinion
• Appendix A: Audit Criteria

1 Executive summary

1.1.1 Introduction

Legal knowledge management (LKM) is best understood as the formal organization of practices, procedures, knowledge, and expertise across an organization to bring about better quality, consistency, and efficiency in the provision of legal work. It is, in other words, how legal service providers organize and leverage the collective knowledge of their organizations.

Leveraging the collective legal knowledge of the Department of Justice (the Department) is essential to its operations. Without collaboration and sharing, the Department increases its risk of losing legal knowledge – one of the Department’s most important assets. The loss of legal knowledge, whether through the departure of legal professionals or by way of poor information management practices, can have serious impacts on the quality and efficiency of legal work at the Department. It can also increase duplication in the workplace.

For LKM to be successful, the Department must have the means to connect people, and to connect people with knowledge. This requires that the Department create an environment in which collaboration and the sharing of knowledge is supported and encouraged.

Responsibility for LKM at the Department rests with the Legal Practices Branch (LPB), which reports to the Office of the Chief Financial Officer and Assistant Deputy Minister, Management Sector. As the functional authority for LKM, LPB is responsible for the development and coordination of national policies and processes to manage the Department’s collective legal knowledge. The LPB is also responsible for developing tools and strategies that facilitate LKM.

1.1.2 Strengths

The Department has successfully implemented Justipedia and other information systems to support and enable LKM activities. Roles and responsibilities for LKM are well defined, and the Department has a comprehensive and robust policy suite in place to encourage and support collaboration and knowledge sharing. Management has cultivated a culture of collaboration across sectors and portfolios.

1.1.3 Areas for Improvement

The development of Justipedia has long been a strategic objective of the Department and the centerpiece of its LKM strategy. Now that the system has been successfully implemented and that policies and procedures are in place to support its use, there is a need to renew the Department’s overall vision for LKM. With the development of a new vision and strategy for LKM, there is an opportunity to study and review the Department’s LKM governance model to ensure that it can support the Department’s long-term LKM goals and objectives, and the achievement of industry best practices.

In addition to renewing its LKM strategy, the Department should pay greater attention to the capture of tacit and explicit knowledge^{Footnote 1} and to the processes and controls needed for knowledge curation. The Department should also revisit and re-align the performance measures and policy mechanisms used to incentivize LKM practices.

1.1.4 Audit Opinion and Conclusion

There are sufficient policies, procedures, processes, and systems in place to support the Department’s LKM activities, and the delivery of high-quality legal services, policies and programs. There are however, opportunities to improve the effectiveness of the Department’s LKM efforts through strategic planning and a review of LKM processes, controls, and structures.

1.1.5 Management Response

Management is in agreement with the audit findings, has accepted the recommendations included in this report, and has developed a management action plan to address them. The management action plan has been integrated in this report.

2 Statement of Conformance

In my professional judgment as Chief Audit and Evaluation Executive, the audit conforms to the International Standards for the Professional Practice of Internal Auditing, as supported by the results of the Quality Assurance and Improvement Program.

Submitted by:

3 Acknowledgement

The Chief Audit and Evaluation Executive would like to thank the audit team and those individuals who contributed to this engagement, in particular employees who provided insights and comments as part of this audit.

4 Background

4.1 What is Legal Knowledge Management

Legal Knowledge Management (LKM) is at its core about improving the practice of law. Although there is no single universal definition of LKM, it can be understood as the formal organization of practices, procedures, knowledge, and expertise across an organization to bring about better quality, consistency, and efficiency in the provision of legal work. It is, in other words, how legal service providers organize and leverage the collective knowledge of their organizations.

4.2 Why is LKM Important

Leveraging the collective legal knowledge of the Department is essential to its operations. The Department employs nearly 2,500 legal professionals across Canada in 44 different areas of law. Capturing and sharing legal knowledge allows the Department to provide high-quality litigation, legislative, policy, and advisory services, and to recognize and build expertise, in keeping with the Department’s Vision. Capturing and sharing knowledge also allows legal professionals to share best practices and to develop partnerships across practice areas and regions, speaking with one voice and providing a consistent message to clients.

Without collaboration and sharing, the Department increases its risk of losing legal knowledge. It may also lose its ability to maintain a highly skilled and knowledgeable work force. At last count, the average age of both the Department’s law practitioner (LP) and law management (LC) groups, being 52 years and 55 years respectively, was higher than the average age of legal personnel in Canada as a whole,^{Footnote 2} and very close to retirement. The loss of legal knowledge arising from the departure of legal counsel can seriously impact the quality and efficiency of legal work at the Department. It can also increase duplication in the workplace. This is evermore true as legal issues and the practice of law become more complex.^{Footnote 3}

For LKM to be successful, the Department must have the means to connect people – and to connect people with knowledge. This requires that the Department create an environment in which collaboration and the creation and sharing of knowledge is supported and encouraged. LKM can be fostered through leadership and strategic planning, and supported by way of policies, procedures, and the deployment of accessible and well-designed systems. LKM practitioners and leaders must also be adequately equipped and resourced.

4.3 LKM Responsibilities at Justice

Responsibility for LKM at the Department rests with the Legal Practices Branch (LPB), which reports to the Chief Financial Officer and Assistant Deputy Minister (CFO-ADM), Management Sector. As the functional authority for LKM, the LPB is responsible for the development and coordination of national policies and processes to manage the Department’s collective legal knowledge. The LPB is also responsible for developing tools and strategies that facilitate the capture, organization and sharing of legal knowledge. Tools include the platform Justipedia – the Department’s official repository of legal precedent information and its single national portal to all legal reference documents useful to practicing law at the Department. Justipedia includes (among other things) legal opinions, pleadings and facta, annotated legislation, practice tools, and jurisprudence. They also include the Department’s Knowledge Management Toolkit, which offers practical tools for managers and employees to facilitate knowledge transfers, along with procedures to assist in the identification of knowledge transfer needs.

5 Audit Objective

The objective of this audit was to provide assurance that LKM policies, procedures, processes, and systems at the Department are adequate and effective in supporting the delivery of high-quality legal services, policies and programs.

For purposes of this audit, ‘systems’ means internal tools, platforms, technologies, and techniques used to support the accomplishment of LKM objectives. This includes, but is not limited to, internal systems such as Justipedia and LEX, as well as third-party systems such as Westlaw and Quicklaw.

6 Audit Scope

The scope for this audit included all policies, procedures, processes and systems within the Department that support LKM, in place during fiscal years 2018-19 to 2019-20 and Q1-2 of 2020-21.

The audit did not include an assessment of the accuracy, completeness, quality, or legal merits of information within the Department’s LKM systems.

7 Audit Approach

This engagement was conducted in accordance with the Treasury Board of Canada Secretariat’s Policy on Internal Audit and Directive on Internal Audit, and the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing. These standards require that the engagement be planned and performed in such a way as to obtain reasonable assurance that the objectives of the engagement are achieved.

8. Findings, Recommendations and Management Response and Action Plan

This section provides the findings and recommendations resulting from the audit work carried out by the Internal Audit and Evaluation Sector. The audit was conducted based on the lines of enquiry and audit criteria identified in the planning phase which are presented in Appendix A to this report.

8.1 Governance

The Department’s strategy for the consolidation of LKM tools into a single national repository (Justipedia), and its efforts to support knowledge sharing have generally been successful. Roles and responsibilities for LKM are well defined (if not well known), and senior management has fostered a culture of collaboration across sectors and portfolios. There is a need however to renew the Department’s overall LKM vision and strategy, and to review the Department’s LKM governance model to ensure that it can support the Department’s long-term LKM goals and objectives. This should include a review of performance measures and policy mechanisms used to incentivize LKM practices.

The Department’s LKM strategy should be renewed

The Department’s first LKM strategy was developed by the Department’s then Chief Knowledge Officer and Knowledge Management and Law Practices Division in 2009-2010. That strategy focused on the consolidation of the Department’s existing LKM tools into a single national repository for all legal opinions, pleadings and facta – a repository or system now known as ‘Justipedia’. The strategy also included plans for the development of a ‘Knowledge Transfer Toolkit’, a consortium of practical tools and strategies to foster knowledge transfer and succession planning, and to help managers and employees better integrate knowledge management into their daily business.

Since 2009-2010, and the introduction of Justipedia in 2012, there has not been a concerted effort undertaken at the Department to redefine and establish its overall goals and objectives for LKM. Although LPB is engaged in strategic and planning activities, its focus over the past several years has been on operational issues. In 2016, LPB issued a formal Governance Framework for Justipedia. It establishes Justipedia as the national portal for the consolidation of all legal reference documents and the coordination and communication of legal developments. Justipedia continues to align and contribute to the Department’s vision of recognizing and building expertise. While the Framework remains a critical governance tool, it does not set out (or purport to establish) the Department’s broader vision and mission for LKM. Nor does it set out the Department’s LKM goals and objectives.

Recognizing that Justipedia has long been the focus of the Department’s LKM strategy, now that the system has been successfully implemented there is a critical need to revisit and reaffirm the Department’s overall mission and vision for LKM, and its strategic goals and objectives. A formal LKM strategy would help re-establish the importance of LKM. It would also help inform stakeholders of the direction in which the Department is headed so as to create awareness of LKM goals and objectives. With a clear LKM strategy, LPB may be better positioned to help legal professionals understand what is being done to foster and support the sharing of knowledge. A formal LKM strategy might also help LPB garner support for planned LKM initiatives, programs and training.

The Department’s LKM model should be revisited

To better understand the LKM models and best practices being used by other Canadian law organizations, interviews were held with a broad range of senior lawyers and non-lawyers involved with LKM at provincial attorney general offices, quasi-governmental bodies, and private national law firms. Interviews were also held with leading experts and third-party vendors of LKM systems in Canada. While most of the interviews were Ontario-based, interviews were also conducted in Quebec and British Columbia.

Among organizations with relatively well-defined knowledge management practices, one of two recurring approaches to LKM were observed. In the first case (in what may be referred to as a centralized LKM model) LKM experts and legal researchers were combined and situated within a dedicated knowledge management group which subsumed the organization’s library. In this model, knowledge managers acted as LKM promoters, educators, advocates, advisors, and legal researchers.

In the second case (a de-centralized LKM model), dedicated LKM lawyers were deployed or embedded within each practice area of the organization. Their role was to solicit, capture, retain, curate, and actively disseminate knowledge specific to their practice area or portfolio. Organizational LKM policies and practices existed, unified across the organization through a general LKM framework, but LKM lawyers tailored their work to the needs of the legal professionals within their specific groups.

Although there appear to be operational advantages to each approach, the de-centralized LKM model seems to have had much more success in bringing about a general awareness of LKM and in eliciting the support of legal practitioners in LKM activities. In many cases, the centralized LKM model was characterized by the absence of dedicated LKM professionals and advisors, leading to the under-resourcing of LKM responsibilities in favour of active case files and case management activities (i.e., legal and operational matters).

While LKM at Justice more closely resembles the de-centralized model for LKM – with policy support, guidance, and systems support centralized within LPB – departmental practice areas do not have dedicated LKM lawyers. The role of LKM lawyers is expected to be fulfilled by so-called ‘super- contributors’ and Justipedia’s editorial board members. In practice however, that work generally falls to paralegals or administrative staff who are not always well equipped or positioned to know what is of importance in relation to a specific subject, or how it might be of benefit to others. Knowledge creation, curation, retention, and sharing is thus left to legal professionals, with limited time to undertake their LKM responsibilities.

Roles and responsibilities for LKM are clearly defined but not well known

Roles and responsibilities for LKM are set out in the Department’s Governance Framework for Justipedia (the Framework). The Framework outlines both corporate responsibilities, and the professional responsibilities of lawyers, paralegals, and managers. Although roles and responsibilities are defined primarily in relation to the use and management of content in Justipedia, they are helpful in establishing what kinds of knowledge should be retained and shared, when, and by whom.

Despite the clarity and precision with which roles and responsibilities are defined in the Framework, LKM responsibilities are not well known or understood. According to the Framework, legal professionals are expected to store legal documents in Justipedia which are “valuable for legal knowledge management”, that is documents that could be useful to other legal counsel at Justice. Legal professionals are also expected to retain and store jurisprudence useful to the practice of law. According to the Framework, all such legal documents are expected to be entered into Justipedia “within thirty days after being finalized”.

A targeted review of the contents of Justipedia found that in many cases the legal knowledge stored in Justipedia was inconsistent with departmental expectations. For example, the prescribed time limits for uploading content to Justipedia were rarely if ever observed. In addition, interviews with both junior and senior legal professionals revealed that in practice, the policies and processes for LKM as well as the use of Justipedia varied by practice area, region, and legal professional – a finding generally supported by the results of a survey conducted by LPB in 2020 in relation to the use of Justipedia.

While it is difficult to ascertain why roles and responsibilities – despite being clearly defined and documented – vary so much by sector and portfolio, the LKM practices of Justice legal professionals appear to vary according to the expectations established by individual groups and managers. Such expectations are not uniform or standardized across the Department, a function perhaps of the absence of a unifying vision and strategy for LKM. Although the release of the Department’s Governance Framework for Justipedia in 2016 has helped to improve legal professionals’ use and understanding of Justipedia – a key LKM tool – there is still a need for a broader LKM vision and strategy to help increase awareness of LKM efforts and expectations.

Performance measures and policy mechanisms supporting LKM vary

For LKM efforts to be successful there must be mechanisms in place to encourage collaboration and the creation and sharing of knowledge. ‘Knowledge hoarding’ is in most cases a natural human behaviour, such that knowledge workers (including legal professionals) may not be naturally disposed to share their knowledge without proper incentives.^{Footnote 4} Incentives, from a public sector context refer to mechanisms created to support LKM activities from a performance measurement and/or a policy lens (as opposed to financial rewards which are typically used in the private sector).

Although departmental performance measures and policy mechanisms exist at Justice to encourage and support LKM activities, they tend to vary across the organization. In practice areas or sectors/portfolios with established LKM practices, LKM and Justipedia contributions are included in legal professionals’ annual performance evaluations and formally tracked with standardized metrics. In practice areas or sectors/portfolios with less established LKM practices, LKM activities are not in any way prioritized, and managers do not formally track or recognize LKM contributions. These differences in practices appear to have a direct impact on knowledge creation and sharing (certainly as measured by contributions to Justipedia).

Just as performance measures and LKM policies appear to impact the use of Justipedia, so too do practices surrounding the training of new hires. According to the Department’s Governance Framework for Justipedia, managers are responsible for ensuring that all new employees receive training on Justipedia ‘quickly’. While the term ‘quickly’ is not defined in the Framework, it underscores the importance of Justipedia training early in an employee’s tenure in support of LKM activities. Whereas Justipedia serves as the Department’s central repository for legal knowledge –, and remains the most important system for the creation, dissemination, and transfer of legal knowledge –, ensuring that all legal professionals at the Department have sufficient training in Justipedia is critical.

Despite the above, interviews with many of the recently hired and senior counsel determined that they had not received sufficient training on Justipedia (and no formal training had been planned). This finding was consistent with the results from LPB’s 2020 Justipedia Survey, which considered the training of legal professionals. In that survey, 13% of law managers indicated that new employees had not received training on Justipedia within the first year of their arrival, and 43% of survey respondents indicated that they had not participated in or received training in Justipedia in the last 5 years. Of those respondents who had received training in Justipedia, 42% indicated that the training received was too general in nature.

A lack of intensive training on Justipedia can have a significant and trailing effect on the use of the system, and on LKM efforts more broadly. More specifically, employees who are well trained on the use of Justipedia, and who develop some comfort in its use early in their career, will tend to use the system more often and with more ease. However, for those employees who do not receive sufficient training on the system there is a risk that they will shy away from its use as an LKM tool for the duration of their career.

Recommendation # 1

It is recommended that the Chief Financial Officer and Assistant Deputy Minister, Management Sector reaffirm the relevance of LKM by developing a formal LKM strategic plan. The plan should:

• Clearly set out the Department’s vision for LKM, along with its short, medium, and long-term LKM goals and objectives;
• Take into consideration the Department’s LKM model and associated structures to ensure that they are sufficient to support the Department’s strategic plan and the achievement of the Department’s LKM vision and objectives;
• Be widely shared and disseminated so as to re-iterate key roles and responsibilities for LKM; and
• Incorporate performance measures and policy mechanisms to encourage knowledge sharing and collaboration.

8.2 Whole-of-Justice Approach

We found that the Department has a comprehensive and robust policy suite in place to encourage and support collaboration and knowledge sharing. The audit also revealed that the Department has sufficient systems in place to support and enable LKM activities. However, there is a need to ensure that practices and processes for the capture and curation of tacit and explicit legal knowledge are working as intended.

There is a comprehensive and robust policy suite in place to support LKM practices

LPB is responsible for developing and providing advice on policies, directives, guidelines, and tools relating to the practice of law and the management of legal services in the Department. This includes policies on LKM. LPB is also responsible for monitoring compliance with these policies, and for ensuring that they remain current and up to date.

According to LPB, there are four primary policy instruments governing legal knowledge management activities at the Department: the Governance Framework for Justipedia (created in 2016), Guidance for Canada’s Legal Team on Providing Legal Advice (created in 2014 and last updated in 2019), the Practice Directive on Sharing of Legal Opinions (created in 2015 and last updated in 2017), and the Knowledge Management Toolkit (developed in 2014 and last updated in 2018). There are also several directives and protocols in place related to information and file management which guide the storage and retention of legal information and which contribute to and support LKM. Together, these policies, directives, and guidelines provide both a comprehensive and robust framework for LKM practices at the Department.

In support of the policy instruments and guidelines developed by LPB for LKM, many sectors and portfolios within the Department have developed their own practices, protocols, and tools to help guide legal professionals in the storage and sharing of knowledge. These practices range in kind – from formal and informal mentoring, to the establishment of knowledge sharing forums, and to the use of standard forms or templates to capture and retain legal knowledge, including legal reference documents saved in Justipedia. Unfortunately, these good LKM practices are not broadly shared across sectors/portfolios or practice areas, leaving some groups to contend with ad hoc or informal LKM practices while sectors/portfolios or practice areas with more mature LKM models thrive.

Systems are in place to support LKM and the delivery of high-quality legal services

While Justipedia was launched in 2012, a formal governance framework to support the system as the Department’s official repository for legal precedent information was not forthcoming until 2016. From this point forward, the system has served as the Department’s official repository for legal precedent information. It serves as a single on-line portal and national LKM platform for all legal professionals, bringing together legal reference documents useful to the practice of law. Its holdings include (but are not limited to) legal opinions, pleadings and facta, annotated legislation, practice tools and jurisprudence. Justipedia also serves as the Department’s official platform for legal study and discussion groups, allowing for the scheduling of knowledge sharing meetings and the storage of a discussion group’s legal reference documents in one place.

Although Justipedia serves as the Department’s official LKM portal, other systems contribute to the sharing of legal knowledge and legal information at the Department. LEX, for example, is the Department’s new legal file management system, having officially replaced iCase, the Department’s former legal case management system, in 2018/2019. This system helps legal staff manage their workload and workflow. LEX also includes advanced search capabilities, and the ability to link or share records with Justipedia.

SharePoint is the platform that underlies LEX (which is used for legal documents) and Digital Workspace (DW) (which is used for corporate documents, not legal documents). DW is a document management system with basic collaboration and search features. It has multiple components, including JustME (a space for personal documents and a microblogging forum) and collaboration sites (which practice areas and sectors/portfolios may use to collaborate on documents, calendars items and tasks). In 2017, DW was linked with GCdocs, the Government of Canada’s solution for meeting legal and policy requirements for the management of electronic and paper documents and records. GCdocs is the official file and record management, and storage system, allowing for the search, retrieval, and lifecycle management of information resources. DW has also been linked to LEX, resulting in an integrated information platform for Justice employees.

Together, Justipedia, LEX, DW and GCDocs serve as enablers for knowledge sharing. Although user experiences vary by system (both in terms of likes and dislikes), each system offers sufficient features and functionality to support LKM. With the advent of new features and functionality in Justipedia and new investments in on-line collaboration tools, user experiences with LKM systems and connectivity are expected to improve. Training and awareness in relation to these systems and their capabilities will however remain critical to long-term acceptance and use.

Knowledge capture, conversion, and curation practices should be reviewed

Generally speaking, the quality and kind of knowledge stored in an organization’s LKM systems will depend on the LKM processes and practices in place to support good information management. Successful LKM also pre-supposes the existence of an adequate infrastructure to support the capture, conversion, and transfer of tacit legal knowledge. Tacit knowledge is ‘knowing-how’, the knowledge that is embedded in the minds of legal professionals and gained through their experiences and jobs. Explicit knowledge, or ‘knowing-that’, is knowledge codified and digitized in organizational records.

Overall, the audit determined that controls are in place to help ensure that explicit legal knowledge is captured and stored in internal LKM systems, and that such knowledge is complete, up-to-date, and available in a timely manner. This is not to suggest that the recorded contents in LKM systems are always in keeping with user expectations. The audit did not assess the relevancy or quality of knowledge stored in Justipedia and other LKM systems, only whether there were processes in place to guide contributions and the open sharing of legal knowledge. In practice, however, interviews with selected Editorial Board members from the Justipedia Areas of Practices indicated that not all contributions in Justipedia are up-to-date or complete. Nor is legal information always uploaded to Justipedia within 30 days of its finalization, as required by the Governance Framework for Justipedia.

While pleadings, facta, legal opinions, jurisprudence, and other explicit legal knowledge is for the most part diligently stored in Justipedia, experiences, practice tools, and other tacit legal knowledge is not. In the review of Justipedia holdings, LEX and DW, it was noted that tacit legal knowledge was often missing. This gap in legal knowledge capture was frequently mentioned in interviews with selected senior counsel, who were concerned about the decline in opportunities for job shadowing, file sharing, and in-person mentoring. As a result, there is a risk that the Department is not sufficiently capturing tacit legal knowledge – knowledge of arguably the highest value – in its LKM systems and sharing that knowledge with others.

Recommendation # 2

It is recommended that the Chief Financial Officer and Assistant Deputy Minister, Management Sector assess whether the Department’s knowledge capture and retention objectives are being achieved, and whether new practices, processes, or controls are needed to support the capture and curation of tacit and explicit legal knowledge.

9 Audit Opinion

There are sufficient policies, procedures, processes, and systems in place to support the Department’s LKM activities, and the delivery of high-quality legal services, policies and programs. There are however opportunities to improve the effectiveness of the Department’s LKM efforts through strategic planning and a review of LKM processes, controls, and structures.

Appendix A: Audit Criteria

Line of Enquiry 1 – Governance Structure

Criterion 1.1 - An effective governance structure is in place within the Department to support LKM.

Line of Enquiry 2 – Whole-of-Justice Approach

Criterion 2.1 - LKM policies, procedures, processes and systems are adequate and effective to support collaboration and the sharing of knowledge.