The Child-centred Family Law Strategy,
Summative Evaluation

3.   EVALUATION METHODOLOGY


3. EVALUATION METHODOLOGY

This section of the report summarizes the objectives, methodological approach and research activities undertaken to complete the Summative Evaluation of the CCFLS.

3.1. Evaluation Objectives

The Summative Evaluation is based on the revised CCFLS RMAF, logic model and evaluation framework developed by the FCY Section in 2006. The stated purpose of the Summative Evaluation is twofold. The primary goal of the evaluation is to assess the extent to which the Strategy was successful in achieving the intermediate and final outcomes identified in the RMAF in order to inform the direction of future departmental participation in the family law area. The secondary purpose of the evaluation is to examine how well the performance measurement mechanism worked to ensure that the required data were systematically collected, a recommendation resulting from the Formative Evaluation. The evaluation framework identifies four key issues to examine in the evaluation, namely: relevance, design and delivery, achievement of objectives and cost effectiveness. The evaluation framework is contained in Appendix C.

3.2. Research Design and Activities

As described in Section 2, the CCFLS consists of a number of different components that, in combination, are meant to achieve the main objectives of the Strategy. The evaluation used multiple lines of evidence to examine the outcomes associated with the various activities undertaken by the FCY Section in support of the Strategy and integrated these findings to assess their collective contribution to achieving Strategy objectives.

Two types of research activities were completed, often referred to as primary and secondary research. Primary research entails collecting data specifically for the study at hand (e.g., through surveys or interviews), whereas secondary research involves the review and analysis of information/data collected for other purposes, such as Statistics Canada Census data or program-related administrative information. Both quantitative and qualitative methods were utilized.

In the present study, primary research activities included a client follow-up survey of 375 people who had used family justice services, 51 key informant interviews with a range of stakeholders, a service provider survey with 69 people involved in the delivery of family justice services, such as mediation, parent education and case management/referral, support recalculation, focus groups with mediators, and case studies of family justice initiatives delivered by the provinces/territories, as detailed in Table 3-1.

Table 3-1: Summary of Primary Research Activities
Primary Research Activities Research Completed
Key Informant Interviews DOJ officials 8
P/T officials 13
Members of the Judiciary 4
Lawyers 8
Subject Experts 8
Other 10
Client Follow-up Survey 375
Service Provider Survey 69
Mediator Focus Groups
  • Winnipeg (11 participants)
  • Victoria (5 participants)
  • Vancouver (4 participants)
Case Studies

Secondary research activities included reviewing a wide range of documents and databases to obtain contextual and background information on the CCFLS as well as information on its impact. In addition to documents detailing the Strategy itself, such as the RMAF, documentation associated with various activities (e.g., conferences and committee meetings) was provided by the various units as evidence of activities, outputs and outcomes. Research reports and literature were reviewed to identify issues facing separating/divorcing families, recent trends in Canadian families and approaches to family justice taken in other countries.

The following secondary data sources and reports were also accessed and reviewed for the evaluation:

3.3. Methodological Challenges and Limitations

A key strength of the evaluation is the number of sources of information that were used and cross-referenced to assess outcomes of the Strategy and the large sample sizes for many of the data sources. However, a number of challenges and methodological limitations to the evaluation remained, as presented below.

3.3.1. Evaluation of a Strategy with Multiple Components

There are difficulties that are often inherent in the evaluation of strategies/initiatives with multiple components and/or program or service elements. As already noted, a key element of the evaluation was to examine the five family justice services supported through the Child-centred Family Justice funding mechanism and the activities/outputs of the units in implementing the Strategy. Each unit is responsible for a range of activities and each unit maintains multiple information sources. The evaluation framework included a large number of indicators, in an attempt to capture the full range of activities of the Strategy. The challenge, therefore, was to identify the collective outcomes related to the multiple components of the CCFLS and to present a cohesive and integrated set of findings that adequately reflects the complexity and broad scope of the Strategy.

3.3.2. Attribution of Outcomes

A related challenge of the evaluation was the inability to delineate the extent to which the various federal activities and funding mechanisms contributed to outcomes in isolation from provincial/territorial and other partners' activities. Due to the division of responsibilities for family justice between and among the federal and provincial governments, both levels of government contribute to the support of the family justice system. Therefore, many of the outcomes associated with the CCFLS are the result of a culmination of activities of multiple players making it difficult to determine the extent to which the CCFLS alone contributed to outcomes. This was also an issue for the cost-effectiveness analysis, as many of the outcomes could not be attributed directly (or only) to the Strategy.

A further challenge to attributing outcomes directly to the Strategy was the fact that the CCFLS followed 18 years of initiatives designed to expand or support the family justice system. These initiatives provided a base upon which the Strategy was to build, making it difficult to isolate the effects of the CCFLS from the effects of its forerunners. Where possible, change over time was examined. In general, results of the evaluation should be kept within the context of previous initiatives in family law.

3.3.3. Information Gaps and Data Limitations

As discussed later in the report (see Section 5.2), there are gaps in the performance measurement information due to issues related to the PMES and provincial/territorial reporting practices. As a result, there are many areas where only partial data were available. It should be noted that RMAFs and PMES were newly introduced to the public service at the time of the inception of the Strategy and experience in developing and using them was limited. Improvements have been realized in many areas since that time.

There are also some limitations associated with some of the data sources. For example, the court file review sample was not meant to be a representative sample and therefore has limits in terms of generalizing the results to all divorce cases. In addition, data for post-Strategy cases were only available for two sites, meaning that any pre-post comparisons can only be interpreted in relation to those two court locations. However, due to the size of the total sample and the number of locations included in the pre-strategy review, those data can be used to support general findings generated through other lines of inquiry and evidence.

Similarly, the Survey on the Practice of Family Law that was used in the evaluation is based on a sample of attendees at the Federation of Law Societies of Canada's National Family Law Program and is therefore not representative of all Canadian legal professionals. However, in noting these limitations to a more generalized interpretability of available data, sources were used only in combination with other lines of evidence to provide support or substantiation.