The Child-centred Family Law Strategy,
Summative Evaluation

5.   IMPLEMENTATION


5. IMPLEMENTATION

Four evaluation questions in the evaluation framework address the implementation of the Strategy. The key areas under this issue included Strategy design and delivery, and follow-up on the performance measurement issues identified in the 2005 Formative Evaluation. The evaluation questions specific to the implementation issue are as follows:

5.1. Design and Delivery

Although some elements of the Strategy could not be implemented as intended, the activities were re-focused, while the broad objectives remained the same.

As noted in Section 2, the Strategy was originally to consist of three independent pillars, one of which was the legislative component. However, as it became apparent that the proposed legislation would not go forward during the course of the CCFLS, the FCY Section identified non-legislative ways to address the objectives of the Strategy and subsequently adjusted the activities of the units involved in the delivery of the Strategy. The FCY Section also revised the CCFLS logic model and RMAF to accurately reflect the non-legislative activities that continued under the Strategy. The current Summative Evaluation is based on the revised logic model and RMAF.

Partnerships and collaborative working relationships have been a key element in the implementation of the Strategy.

The integrated team approach adopted in earlier strategies to deliver/support the CCFLS was noted by a few Department of Justice officials as a key element facilitating effective coordination of the CCFLS activities by the FCY Section. The collaborative approach is supported, in part, through a number of information-sharing/collaborative processes and structures.

Overall, Department of Justice and provincial/territorial officials interviewed felt that the CCSO-FJ and its sub-committees and working groups help support the achievement of the objectives of the CCFLS by acting as the forum where discussions can take place and issues can be resolved. The annual reporting requirements of the provinces/territories on outcomes and the seven primary areas of funding activity also support the partnerships in terms of information/data sharing, under the Family Justice Initiative (FJI) component of the Strategy. In addition, the Department of Justice works in close collaboration with the Canada Revenue Agency (CRA) with regard to support enforcement activities such as using CRA databases to trace persons in default of family maintenance obligations as well as the diversion of tax returns payable to the defaulter. Other less formal relationships noted by Department of Justice officials included the good working relationship with the Canadian Bar Association and with the judiciary.

A number of partnerships established under previous initiatives have continued and new ones have been developed between the FCY Section, other federal departments, provincial/territorial governments and NGOs. Specific examples of partnerships and collaboration include the following:

An important outcome of these relationships noted by some of provincial/territorial officials interviewed is the information sharing that occurs between the provinces/territories outside of the various committees. For example, the British Columbia Web site for children and teens (www.familieschange.ca) was designed by the province, in partnership with the Law Courts Education Society of British Columbia, and contains generic information about provincial family law so that the site would be applicable Canada-wide. Other means of sharing experiences occurs through various pilot projects. These projects are subject to evaluations and the reports are generally made public and/or shared between jurisdictions. During interviews with various family justice service providers and family law professionals, many of the key informants expressed an interest in what is being done in other provinces.

The partnerships and collaborative efforts between the FCY Section and other federal departments are particularly important for enforcement activities.

Department of Justice officials felt that the activities undertaken as part of the CCFLS complemented or supplemented those of a number of other federal departments. This was particularly the case with respect to enforcement activities. Some of the tangible benefits in the area of enforcement that resulted from the communications/liaisons activities with other federal departments noted by Department officials included:

Like the collaboration between the Department of Justice and other federal departments, Quebec has in place a collaborative approach in the province's support-payment collection program, Fonds des pensions alimentaires. This program was launched in 1995 following the adoption of the Act to Facilitate the Payment of Support, which applies to all judgments rendered on or after December 1, 1995. The Act led to the creation of the Fonds des pensions alimentaires, which is administered by Revenu Quebec in collaboration with the Ministère de la Justice. Revenu Quebec was brought in to provide its expertise in debt collection, and thereby, to facilitate the payment of support. The Ministère de la Justice is responsible for informing Revenu Quebec of all judgments rendered in which support is granted since the adoption of the Act, regardless of the need for enforcement. The program's procedures for handling the file and emphasizing early communication with the clients are felt to be innovative features, as well as its "hands on" approach of direct involvement of employers to assist with collection of regular support payments and arrears.

There do not appear to be any areas of significant duplication or overlap between the federal and provincial/territorial activities.

In general, Department of Justice and provincial/territorial officials felt that activities between the two levels of government do not overlap. All provincial/territorial and federal officials interviewed felt that the federally funded activities undertaken as part of the CCFLS complemented or supplemented their own family justice related activities, those of other partners and those of other federal departments.

5.2. Performance Measurement

Although there have been improvements to the performance measurement activities and reporting on the part of the provinces, there remain challenges.

The development and implementation of RMAFs and performance measurement and evaluation strategies (PMES) for public programs was relatively new when the CCFLS was first established in 2003. In 2004, a needs assessment project was conducted to determine what assistance, if any, the provinces/territories would need to help them meet the performance measurement and reporting requirements of the CCFLS under the funding agreements.

A number of recommendations resulted from the consultation, including that a reference tool be developed to guide the provinces/territories in data collection and reporting. In response, the Program Development Unit produced the Performance Measurement Handbook to assist the provinces/territories to better report on performance indicators. In addition to the Handbook, staff from the Research and Program Development Units are available to help the provinces/territories develop or refine their performance measurement and evaluation plans, and to assist in developing the data collection tools/approach.

All Department of Justice officials who commented agreed that there has been an improvement in the quality of the information provided by funding recipients over the past two years. Examples of improvements given by officials included providing more detail and building evaluation activities into projects/programs. This observation is partially supported by a comparison of provincial/territorial data submissions from 2003-2004 FY, 2004-2005 FY and 2005-2006 FY, which revealed that more measures have been implemented in all reporting provinces and measurement activities have moved from the development stage to the implementation stage.

Despite the progress made, Department of Justice officials felt that there is still room for improvement in the PMES activities of the provinces/territories. A review of the project files and provincial/territorial annual reports confirmed that the PMES reports are, to a large extent, incomplete and do not allow for a comprehensive analysis of the status of performance measurement in the provinces/territories. In accommodating provincial/territorial differences with regard to programs and administration, reports vary substantially and/or are too incomplete to be able to draw consistent conclusions on performance measurement activities.

The PMES is complex, requiring tracking and reporting on a large number of indicators.

Generic logic models and PMES were developed for the five major types of initiatives funded under the CCFJF. These were used by the provinces/territories to identify which performance indicators to measure and report. Overall, due to the complexity and broad scope of the Strategy, the number of measures that provinces/territories have to report on annually is large. The issue of performance measurement is further complicated by the fact that each province/territory implements their own version of PMES, has a unique combination of family justice services as well as their own methods of information management and data reporting.

A key lesson learned in the area of performance measurement is that, to obtain adequate information from the provinces/territories, the reporting process needs to be simple, clear and as standardized as possible. Department of Justice officials suggested that a future initiative should develop an RMAF that has fewer, more inter-related and clearer indicators. This would allow for streamlining the process and cutting back on the information that is required. Efforts are currently underway to create a database that will support comprehensive analysis for on-going performance measurement and evaluation activities in the future.

Pilot Project Agreements require that an evaluation be completed at the end of the pilot with results reported by the funding recipients. Funding for the evaluation component of the pilot project is provided to help meet this obligation. Overall, the quality of the evaluation research varies; however, many report only usage numbers and satisfaction levels. Few apply a design that permits rigorous net impact estimation, as it can be difficult or complicated to apply an experimental or quasi-experimental design. A few Department of Justice officials indicated that, in general, too much process data is being collected instead of data on long-term impact of the services. However, as noted in reviews of Canadian and international family justice services evaluation and research, this situation is not atypical, as few evaluations implement rigorous methodological designs to measure net impacts.[27]

NGOs that receive Public Legal Education and Information and Professional Training (PLEI/PT) funding, on the other hand, are required to submit descriptive or narrative reports on the application of the funds at the conclusion of the project. Department of Justice officials noted that NGOs that received funding for PLEI/PT projects tend to provide detailed quality information in these reports.

Capacity issues limit the ability of the provinces/territories to meet the current reporting requirements.

Provincial/territorial officials identified a number of challenges that they face in collecting and providing data and other information (e.g., evaluation/research reports) to the Department of Justice. These included insufficient time to conduct the work, the costs associated with collecting and reporting the information, the lack of available consultants or experts to assist, and potential privacy issues. Several respondents mentioned that, in jurisdictions where there are third-party family justice service providers, it is the contracted organizations that collect the information, limiting the amount of control provinces/territories have over the collection of information. It was also noted that some jurisdictions do not have staff assigned specifically to data collection and reporting activities.

Some of the challenges faced by the Program Development Unit in obtaining information from the jurisdictions included the lack of uniformity in programs and services offered across the country, and the perception or reactions of the provinces/territories to the reporting requirements. Although the Performance Measurement Handbook was developed to simplify the information collection and reporting process, the PMES requirements were not well received by all provinces/territories.

Another challenge was that, although the provinces/territories often provide a lot of information, it is not necessarily the information needed to track/evaluate performance. Capacity limitations and resistance to change may be keeping some jurisdictions from being able to adapt to the process and to keep up with the federal performance measurement needs. However, it was felt that, while not everyone appeared to see the benefits in collecting this information, attitudes were starting to change.

5.3. Financial Resources

Annual budget information indicates that sufficient funds were allocated to carry out the Strategy in terms of its operational, management and salary requirements.

At the outset of the CCFLS in 2003, the funding allocation was to be $163 million over the five years. Of the planned budget, 28% was to go toward the expansion of UFCs. However, the expansion of UFCs did not occur; therefore, the funding allocation was reduced by $46,863,010. These adjustments resulted in an overall decrease to the funding allocation to the Strategy from the anticipated $163M to just over $128M.[28]

Summarized in Table 5-1, are the planned and actual financial resources allocated to the operation and management of the CCFLS for each year of the Strategy. Note that surplus funds were allocated to the P/Ts for supplementary projects identified under the FJI component.

Table 5-1: Budget Submissions and Expenditures
(2003/04 Fiscal Year to 2007/08 Fiscal Year)

2003/2004 (1)
  Initial Allocation Revised Budget Actual Expenditure Actual Expenditure as % of Revised Budget Variance
O & M 4,973,998 4,451,114 2,594,817 58.3% 1,856,298
Salaries 4,220,755 -- 3,857,196 91.4%. 363,559
Total 2003/2004 9,194,753 8,671,870 6,452,013 74.4% 2,219,857

2004/2005 (1)
  Initial Allocation Revised Budget Actual Expenditure Actual Expenditure as % of Revised Budget Variance
O & M 4,408,676 3,901,576 2,610,055 66.9% 1,291,521
Salaries 4,536,384 4,380,692 4,238,888 96.8% 141,804
Total 2004/2005 8,945,060 8,282,268 6,848,943 82.7% 1,433,325

2005/2006(2)
  Initial Allocation Revised Budget Actual Expenditure Actual Expenditure as % of Revised Budget Variance
O & M 3,527,097 3,092,097 3,067,262 99.2% 24,835
Salaries 4,507,198 4,368,199 4,472,503 102.4% (104,304)
Total 2005/2006 $8,034,295 $7,460,296 $7,539,765 101.1% ($79,469)

2006/2007 (3)
  Initial Allocation Revised Budget Actual Expenditure Actual Expenditure as % of Revised Budget Variance
O & M 2,434,375 2,050,375 2,302,017 112% (251,642)
Salaries 4,653,813 4,064,851 3,936,140 96.8% 118,145
Total 2006/2007 7,088,188 6,115,226 6,238,157 102% (133,497)

2007/2008 (3)
  Initial Allocation Revised Budget Actual Expenditure Actual Expenditure as % of Revised Budget Variance
O & M 2,524,726 2,086,716 2,011,918 96.4% 74,798
Salaries 4,653,813 4,064,851 3,983,250 97.9% 81,601
Total 2006/2007 7,178,539 6,151,567 5,995,168 97.4% 156,399

Approximately two-thirds of the total Strategy budget went to the provinces/territories to operate, expand or enhance their family justice services.

FJI funds distributed to the jurisdictions through the CCFJF are based on an allocation model developed by the CCSO-FJ and approved by the FPT Deputy Ministers. The grants and contributions made under the FJI for each year of the Strategy are summarized in Table 5-2.

Table 5-2: Grants & Contributions Allocation By Fiscal Year
Fiscal Year Grants and Contributions Expenditures CCFLS
Expenditures		 Grants and Contributions as Percentage of Total CCFLS Expenditures
2003/2004 15,159,000 25,385,447 63.7%
2004/2005 16,079,500 25,012,113 64.3%
2005/2006 16,244,855 24,218,359 67.1%
2006/2007 16,050,027 23,627,967 67.9%
2007/2008 16,050,027 23,128,246 69.4%

Source:  Department of Justice IFMS and SMS data

A total of $2.05 million was distributed for pilot projects and a further $2.6 was allocated to NGOs for PLEI/PT projects. The funds available for PLEI/PT projects were fully utilized, as all resources requested by eligible NGOs for the funds were approved and distributed. It should be noted that, in addition to the professional training funds distributed for special projects, the provinces/territories also allocated funds received from the CCFJF towards training activities for family justice staff and professionals initiated in their own jurisdictions.

5.4. Observations and Conclusions

Based on the results reported in this section, the following observations and conclusions have been drawn: