Evaluation of Public Legal Education And Information: An Annotated Bibliography
- 4.1 PLEI Evaluations
4. Evaluation Reports
4.1 PLEI Evaluations
The reports annotated in this section are actual evaluations, both internal and external, of recent PLEI initiatives. Based on the selection, it can be noted that PLEI organizations that perform comprehensive evaluations do so in a methodical and consistent manner. The approach, documentation and analysis in many of these reports can be applied to various PLEI programs in other jurisdictions.
Burtch, B. & Reid, K. (1993). Discovering barriers to legal education: First-generation immigrants in greater Vancouver. Vancouver: The People's Law School.
General Overview
This evaluation report was produced for the People's Law School ("PLS") in Vancouver, British Columbia. The objectives of this assessment were three-fold: (i) to determine who is using the PLEI programs provided by the school; (ii) what barriers exist for first generation immigrants to accessing legal information resources and how PLS can overcome these barriers; and (iii) to measure the role played by PLEI in immigrant settlement and adaptation. The primary method of PLEI provision addressed is the school's free law classes that are offered as part of the Cultural Minorities Program. The study used a control group approach whereby participants were divided on the basis of whether or not they had attended PLS free law classes. The study is intended to improve the delivery of services offered by PLS and to provide "information of direct relevance to immigrant-serving agencies in B.C., as well as people concerned with policy development and the administration of justice in B.C."
Much of this report may be useful to organizations that encounter first generation immigrant populations in their work. However, the actual questionnaire and Interviewer's Manual may be very helpful to a wider range of PLEI organizations. In particular, the "rules for interviewing" provide some simple directions for eliminating bias in questioning and prompting. These instructions are especially pertinent where consistency concerns arise due to multiple interviewers. Furthermore, the background information and legal needs questions may be easily adapted to accommodate different populations and other topics of legal information.
Common Components of Evaluation
- External evaluation
- Four objectives for the study are clearly articulated in the document and are linked to the conclusions
- Recommendations include a number of suggestions for further research/assessment projects
The Challenges of PLEI Evaluation
- Evaluators found it difficult to generate a population sample. Most of those approached were suspicious of the law, often because of political repression experienced in their country of origin, or thought that the data would be used for government purposes. Moreover, many participants perceived the questions as dealing with private matters and were hesitant to discuss income levels and attitudes toward policies.
Evaluation Methodologies
- The study was conducted by way of semi-structured interviews that followed a lengthy questionnaire. The questionnaire was divided into two parts: (i) 24 questions regarding background information (personal characteristics); and (ii) 18 questions regarding legal information needs.
- The questionnaires in all five languages were pre-tested and modified where necessary.
- Interviewers were recruited from the five ethnic communities surveyed. All interviewers were bilingual and attempts were made to match genders between the interviewer and interviewee. Interviewers participated in a training session on the questionnaire and were given an Interviewer's Manual that included general rules for interviewing as well as specific background information and probing techniques for most of the questions.
- Hypothetical legal scenarios were used to test knowledge differentials for those who have attended free law classes in each of the five cultural groups.
Curtis, C. K. & Meehan, G. (2001). Courtlink auto crime prevention program: Evaluation report. Vancouver: Law Courts Education Society of British Columbia.
General Overview
This 13-hour program was designed for senior elementary classes and alternative schools in British Columbia and was developed by the Law Courts Education Society of B.C. The goals of the evaluation were two-fold: (i) to identify reactions of experiences with the program; and (ii) to assess the degree to which the program met its two stated goals of increasing knowledge and promoting positive attitudes. Different evaluation tools were used to measure the two goals. This report exhibits an extremely thorough understanding of evaluation methodologies and included a detailed analysis of findings. At every stage of the report the authors explain why particular choices were made in the process and offer suggestions as to effective methods for developing PLEI evaluations for different populations. Similar to other formal, external evaluations, the methodologies sections provide useful information for subsequent PLEI evaluations. The steps taken in developing both the survey research and the pre-test/post-test instruments are explained in detail and may be adapted to other educational programs. Research questions and various assessment instruments that were developed are all appended to the report and would be a good starting point for any school-based PLEI programs.
Common Components of Evaluation
- External evaluation
- Built into the project design and conducted pre-program, concurrently and post-program
- Goals of the evaluation and the educational program were articulated in the report and evaluation methods were matched to different goals.
- One of the recommendations suggested an annual review and update of materials used by teachers to ensure that they are time sensitive. This is a common way that projects can perform ongoing monitoring without incurring great expenses of hiring external evaluators.
The Challenges of PLEI Evaluation
- Open-ended questions are often desirable to elicit emotional, psychological and intuitive responses, however, it is important for PLEI providers to rank the preferred or "more desirable" answers to questions in order to clarify intended outcomes.
- Limitations placed on feedback surveys may encourage consistency in reporting. It is important to offer an opportunity to discuss "factors" that affect the administration of the program somewhere on the questionnaire. For example, it is not sufficient to ask whether a particular activity was useful or effective without knowing how much time was allocated to that activity or if it was appropriate to the particular students involved.
- One draw back of pre-testing is that participants have knowledge of the study which may threaten the validity of the tests.
Connections between Goals of PLEI and Evaluation
- Goals of PLEI programs should be realistic and sensitive to target audiences and environments. For example, if students show possession of prior knowledge about a particular topic, facilitators should not expect to see "substantial gains in knowledge" which may be a stated goal.
Evaluation Methodologies
- Questionnaires were constructed for three groups (facilitators were given two different questionnaires to complete) to measure the experiences of the program.
- Pre- and post-tests were given to students to measure change in knowledge and attitude
- All tests were pilot-tested and modifications were made accordingly
- Tests included some scenario questions that asked for responses as to what should be done and what action would likely be taken
- Evaluators also sought oral feedback from students and teachers regarding what they thought and how they felt about the program.
Ellis, D. (1997). Program review. Vancouver: The People's Law School.
General Overview
This study summarizes a comprehensive review of the People's Law School ("PLS") PLEI programs. It involved a range of PLEI stakeholders in an attempt to view PLS services from a local and provincial perspective. Thirty conclusions were drawn, which were grouped into eight categories of inquiry, as well as five recommendations directed specifically toward the staff and Board of PLS. Although the "findings and learnings" collected for each population or method of evaluation were intended for the staff and Board of PLS, other PLEI organizations will benefit from reviewing these findings. The evaluation is clearly written and the structure of the report allows for easy access to relevant information, especially the bulleted section that follows each question entitled "What do we/can we learn from these findings?" The tools used in the various evaluation components are appended to the report and may be useful to PLEI providers in designing evaluations. These include samples of the interview guide, client feedback sheet, resource/pamphlet feedback sheet and focus group guides with topics and questions.
Common Components of Evaluation
- External evaluation
- Conducted post-program
- Program review had clear, but broad objectives:
"to analyze the efficiency and effectiveness of the present programs of the People's Law School with regard to current and emerging needs for PLEI and to consider the Society's role in relation to other PLEI providers in the province"
(p.8). - No comment on further or ongoing evaluations
The Challenges of PLEI Evaluation
- It is difficult to assess whether there is overlap between programs for a number of reasons:
- PLEI providers believe that need is so great that there can never be enough PLEI
- No commitment to sharing information in order to decrease duplication
- Not enough trust between organizations to effectively coordinate initiatives
- Belief that innovation is more likely to occur with a larger number of smaller organizations than with larger amalgamations
Challenges of Funders' Evaluation Requirements
- Despite the reticence to discuss overlap, funders are highly concerned with duplication of efforts and program overlap between PLEI providers and want this type of analysis included in evaluations.
- Suggestions for better coordination between core PLEI providers came primarily from funders even though there is reluctance from PLEI organizations.
Evaluation Methodologies
- Identified methodology was a qualitative, participatory approach
- Two focus groups: (i) staff at PLS and (ii) Board of PLS
- Materials/pamphlet feedback from PLEI distributors
- Interviews with funders, groups with unmet PLEI needs, co-sponsors of PLS programs and observer-experts in the PLEI field all conducted in informant's native language
- Client feedback surveys (684 were returned)
Law Courts Education Society of British Columbia. (2000). Journeys of justice First Nations research project: Final report. Vancouver: Law Courts Education Society of British Columbia.
General Overview
This detailed evaluation began two years after the First Nations Journeys of Justice curriculum was introduced into elementary schools in British Columbia. The study was conducted over four years and the primary objective was to examine whether the curriculum has a "positive effect on the students' knowledge, attitudes and ultimately behaviour regarding the law"
(p.4). The breadth and nature of the study required both qualitative and quantitative evaluation methods to be developed. The ways in which these two research aspects interrelate (complement or contradict) is analyzed within the reports findings. This section is particularly informative as there is usually little or no attention paid in the evaluation reports reviewed regarding how various tools interact with one another. The other section that has potential for general application to PLEI evaluation is Part 6 "Key Issues of Research Methodology"
(p.106-117). Some details of these challenges are highlighted below, but given the infrequency with which these issues are addressed in other reports, PLEI organizations may find it helpful to review this section and identify challenges faced by other educational programs. Overall this is an excellent model for evaluation. However, the extent of the research and the allocated funding are significantly greater than what is feasible for most PLEI organizations or funders.
Common Components of Evaluation
- External, committee-driven evaluation that was primarily coordinated by Law Courts Education Society of B.C. There were some First Nation education consultants included, who were also involved in designing the curriculum
- Evaluation commenced two years after the program was implemented in schools and was conducted concurrently to program delivery for grades 4,5 and 6 students
- Goals of the evaluation were clear, but expansive given the length of the study
The Challenges of PLEI Evaluation
- Due to the native content and partially native participant populations in the program, it was important to have qualitative studies conducted by local residents who had training and familiarity with the community. Even though the materials and tests were pilot-tested, this still proved to be an important issue for collecting relevant data.
- Three key issues were highlighted for future research in similar settings (longitudinal studies in elementary schools). They were:
- Change in makeup of student populations
- Change in school personnel and teaching styles
- Modifications in testing variables and curriculum
- Other issues/challenges specific to the project were grouped into three major categories:
- Local issues i.e., getting permissions, teachers/student changes during school year, travel distances, unexpected interruptions, change in research process to accommodate unique problems
- Analysis and research issues i.e. programs followed differently in schools, gaps in comparative data, lack of consistency in teaching and testing procedures
- Administrative issues i.e., due to complexity the project required more time and effort than anticipated
Evaluation Methodologies
Quantitative tools developed
- Pre/post-tests covering general and specific knowledge
- Attitude surveys (pre/post surveys were interchangeable)
- Behaviour questionnaires tailored to three population groups
Qualitative tools used
- Semi-structured interviews
- Focus groups
- Classroom observation and recorded field notes
- Classroom stories/ "lived experience stories" communicated orally by teachers, students and community members mostly related to noticed changes in behaviours and attitudes. The authors note that
"this story approach is congruent with First Nations oral tradition which utilizes life experience stories…and can be used to illustrate themes and to provide a basis or structure for data analysis"
(p.56).
Shariff, S. (2000). Identifying successful school and community programs for youth: An evaluation rubric and compendium of sources – DRAFT. Youth Justice Education Partnership, Department of Justice Canada. Retrieved December 20, 2001, from http://www.extension.ualberta.ca/youthjustice/rubric.htm.
General Overview
The two primary objectives of this document were to (i) identify sources of information on conduct intervention programs both in school and in the community (Compendium of Sources); and (ii) to develop an evaluation rubric to help identify critical intervention programs both in school and in the community (reproduced below). Twenty-two school programs were evaluated using the rubric, which was developed to be sufficiently flexible to apply to a variety of multi-disciplinary programs; here the programs were categorized under health, education and justice. The rubric forms the basis of a national standard for program analysis, and as such it should not be seen as restricted to programs directed at youth. Although the findings of these twenty-two program evaluations are divided into "didactic" and "holistic" programs, this distinction is not necessary to the value of the rubric. The structure incorporates four conceptual approaches to evaluation, which are based on the work of 5 different authors. Combining elements from these various approaches a rubric was created that is "broadly and consistently applicable to programs, policies, or evaluation studies of programs and policies"
(p.29). As mentioned in the introduction, the production of comprehensive resources that list the available PLEI materials in any given area are extremely useful and can benefit all PLEI providers nationally and internationally. The compendium of sources compiled in this research project is an excellent example of such a resource. Categories of sources include: journals, relevant journal articles and academic papers, video-tapes and databases, books relevant to schools, conference papers, relevant legislation, etc. These are further divided up under whole school programs, school-community programs and community programs and additionally there are topical lists of sources such as restorative justice and mediation and programming for youth.
One recurring practice in evaluation that was emphasized in this project was the need to understand the subject matter from a range of perspectives. This familiarity can be acquired thorough a multi-disciplinary literature review. Depending on the resources available and who is undertaking the evaluation this background work will vary in degree. For example, if a PLEI organization is conducting an evaluation of their programs or materials they may not require much extensive research into the area. On the other hand, in order to ensure that projects are addressing pressing issues and are maximally sensitive to contextual factors facing the targeted audiences, it is valuable to be alert to current, relevant research. In this vein, the project looked at the influence of administrative policies within schools, studies on boredom as it relates to youth and violence and other animating factors such as poverty and employment.
Common Components of Evaluation
- Classification of internal or external is not applicable to this research project given the mandate of developing an evaluation rubric. However, among the programs evaluated with the rubric, most had been subject to some form of evaluation with a mix of internal and external evaluators.
- Objectives of the project were clearly articulated
The Challenges of PLEI Evaluation
-
Main limitation noted throughout the project was time and as evaluation is an ongoing process there is always the ability to collect more information. For this reason, the authors noted that many exceptional programs could not be included.
-
The scope of the programs evaluated in the project made it logistically difficult to obtain data from across the country.
- Finding documented reports of programs was difficult as
" those running excellent programs are too busy to document their work because they expend all their time and energy on the young people they serve…and have too little time to record or document the philosophy, activities, or successes and failures"
(p.31). - As with most other subject areas, there is the need to attempt to overcome assumptions and rhetoric used in the language and approach to youth at risk and violence in schools.
Connections between Goals of PLEI and Evaluation
-
The second branch of the rubric requires an examination of whether the stated goals of the programs are being met, which is especially relevant to funders seeking accountability.
Evaluation Methodologies
- Methods used in the evaluation studies ranged from oral accounts of successes to formal and extensive tracking of student's lives, careers, activities and families over many years. Other common methods included: pre/post program tests; student incident reports of bullying, rebellious or delinquent behaviour; classroom observation; website feedback; home visitation studies; surveys and questionnaires; staff/teacher interviews; information gathering regarding attitude changes between pre and post program.
Useful Materials for PLEI Evaluation
The rubric is reproduced below in its entirety and can be found in the executive summary of the document.
YJEP Evaluation Rubric
Rationale and goals: What is the program trying to accomplish?
(Is it critical, ethical, transformative, educational, dialectic, legally defensible?)
- What is the underlying philosophy behind the program and what are its objectives?
Key Elements of the program: How does it meet its goals?
- What are the key elements (content, format and implementation) that contribute to success or demonstrate potential success?
Program evaluation disclosed in the research
- Describe the results of studies evaluating the program and methods used to conduct such evaluation, if available.
Examine the context: For whom, what setting and with what attitude?
- Evaluate the potential of programs and models for intervening in different settings among different populations of youth (in terms of individual characteristics of youth targeted by the program; the socio-cultural school context; antecedent factors that impact their lives such as age, family background, ethnicity, poverty, boredom, gender, sexual orientation, physical and mental health, ability or disability and existing administrative policies.)
- Implications for program success or areas of improvement:
- [Is the program approach transformative, critical, educational, ethical, dialectic, and legally defensible?]
- Analysis and discussion of program in terms of the above criteria and the literary research from multi-disciplinary perspectives: education, social psychological and transforming personas.
Sy, S. S. & Anderson, K. (1999). Violet: Learning on the net. Final report to the Office of Learning Technologies. Retrieved December 24, 2001, from http://www.acjnet.org/home.cfm.
General Overview
This is a multi-phase project which involves construction, training and evaluation of an interactive website for women who have experienced violence and their service providers. The first phase entailed constructing the website and building the capacity of women's shelters to access the Internet. The second phase, reported in this document, involved a literature review, descriptions of the web design process and the research methodologies and the findings of two evaluations. In an attempt to improve understandings of how a particular group of women (abused women and their service providers) learn about the law using Internet technologies two research strategies were used: (i) small-group qualitative study; (ii) external evaluation using expert analysis and usability testing. This document is an excellent resource for other web-based learning projects. The literature review and bibliography are extensive and multi-faceted and the evaluation findings are summarized thematically and easily transferable to other subject areas. The excerpt from the external evaluation includes numerous and detailed suggestions regarding effective methods for tailoring websites to accommodate different learning styles and contexts. Additionally, Appendix B delineates the research plan and may be a helpful guide to any organization that is designing an evaluation process.
Common Components of Evaluation
- Two separate evaluations were conducted: external and internal.
- Evaluation was built into the design of the project. This is particularly useful where a program is still in the developmental stages and where the nature of the medium used is rapidly and constantly evolving as a learning and communication tool.
- Goals of phase II are clearly articulated in the report and were instrumental in project development.
- Outlined possible follow up research questions for subsequent evaluations (internal and external).
The Challenges of PLEI Evaluation
- Using web-based delivery researchers cannot know definitively who is using the site, although there are some ways to identify users such as knowing how and to whom the site is promoted or through on-line feedback.
- Difficult to measure success and usefulness of the site for intended audience. Could institute a counting system, but this would not measure the usefulness, whether there was any change in the person's situation as a result of the program or whether it had an empowering effect on the user.
Connection between Goals of PLEI and Evaluation
- Objectives of the project, including long-term outcomes, were indicated at the start of the design process which assists with directing evaluation. The conclusions and findings were matched to the intended outcomes to determine success and effectiveness of the learning tool.
Evaluation Methodologies
Group Qualitative Study by Project Members
- Semi-structured question periods were held after training sessions with shelters and other organizations; included rural and urban shelters and emergency and second stage shelters.
- Pilot study of website conducted during half-day session with an organization that involved training, use of program and debriefing.
- No pre/post evaluation devices used to test knowledge, understanding, attitude change.
- Electronic diary was used after all testing sessions to track findings and record evaluation.
External Evaluation
- Close examination of site including content, structural design strategy and identification of situational and psychological factors that impact site navigation.
- "Talk-aloud" with users as they navigate the Internet program while observing movements and facial expressions (this was done with five women with varying degrees of education, who were randomly chosen and who had not necessarily experienced domestic violence).
- This method of evaluation has limited use i.e., not appropriate for young children.
- It is very time-consuming, though highly informative.
The People's Law School. (1994). Report of the multicultural and immigrant seniors legal education project, 1992-1994. Vancouver: The People's Law School.
General Overview
This project had a number of PLEI elements as well as a strong community development component. The target population and the specific needs they faced were identified through research of community and government agencies, and the People's Law School created the Legal Education Project to address these identified needs. The evaluation report was produced, in part, for the purpose of helping other community-based PLEI organizations in implementing their legal education programs with seniors. Five strategies were enumerated at the outset and a variety of activities were developed to achieve these objectives and strategies. These included over 90 legal education classes, professional workshops, radio programs, and two publications that were reproduced in five languages and disseminated to thousands of seniors. This report will be useful for PLEI providers and especially those whose mandate is to serve senior and/or immigrant populations. It is succinctly written and addresses a number of evaluation issues overlooked in other reports reviewed.
Common Components of Evaluation
- Evaluation was internal and was developed and written by those who designed and implemented the various programs. An external evaluation was included in the original plans but had to be abandoned for budgetary reasons.
- Assessment occurred after most of the activities were concluded, however some of the key features of the project entailed long-term, ongoing objectives such as building community networks.
- Future research projects with evaluative components were suggested.
The Challenges of PLEI Evaluation
- Advantages and disadvantages for using an internal evaluation were discussed. The authors recognized that integrated evaluations lack the objectivity gained from an evaluation that is removed from design and implementation. However, direct involvement was seen to inject the knowledge of where progression has occurred and how the project has adapted and changed over time.
- Many of the project's objectives cannot be measured by quantified means. For example, the benefits of networking between different immigrant populations about legal issues facing seniors in order to build information-sharing relationships
"cannot be measured by counting the numbers of participants at events, or copies of publications distributed" (p.14)
. - Similar to other community-based programs, the directives and directions for these projects came from participants and community members. Therefore, it is seen as inappropriate to evaluate the "success" of any given activity, as project management (from conception to completion) was administered by community members and tailor-made for unique audiences.
Challenges of Funders' Evaluation Requirements
- Report included one section on the tension of balancing the community development process, which produces intangible benefits, against "deliverables" such as events or publications for both projects and funders. The authors note that
"[i]f all or most deliverables are focused on numbers, events or publications it leaves few resources (time, energy, money, volunteers) for the community development work. Due to these pressures, the process may become the traditional ‘top down' approach in which needs are defined by those furthest away from the community, and from those most affected"
(p.21).
Evaluation Methodologies
- Feedback forms were distributed to all seniors who were involved in the project.
- Evaluation meeting/consultation was held for all committee members.
- Informal, oral feedback was given by service providers who attended professional development workshops.
- General observations of committees and events.
- Some project activities incorporated evaluation into design i.e., "breakout"/brainstorming. sessions on how to establish and improve community ties and many new initiatives evolved out of the committees and projects.
Whyte, K. J. (2000). Native youth at risk courtlink program: Pilot project evaluation report. Vancouver: Law Courts Education Society of British Columbia.
General Overview
This short evaluation report is very thorough and can serve as an excellent model for PLEI evaluation. The program evaluated is a two-day educational program for youth at risk, part of which takes place at a courthouse in the community where the school is located. The format used in collecting information closely matches the goals of the assessment such that all of the key issues are addressed. The range of evaluation techniques employed in the program review ensured that feedback could be easily translated into conclusions and recommendations. Participants were explicitly invited to share their views as to whether the program met its stated goals, yet they were also offered space in which to make general comments and suggestions for future programs. In evaluating the presentation of materials a range of variables was articulated and subdivided into categories under student and program variables. This element highlights the evaluator's recognition that how the program is delivered is equally, if not more important than the curricula and materials of the program. This report also emphasizes the crucial link between the quality and effectiveness of the program and the ability of the program's structure to adapt to a wide variance in student interest and ability.
Common Components of Evaluation
- External evaluation
- Evaluation was built into the design of the pilot project and was conducted concurrent with program delivery
- Goals of evaluation were clearly articulated in the report
- Third goal was to determine whether further evaluation regarding impact and outcome was recommended. Given that all of the information and data collected indicated that the present curricula and presentation was very successful in reaching target group, no further evaluation of present materials was recommended.
The Challenges of PLEI Evaluation
- Evaluator found that the feedback varied between the questionnaires and the personal comments of participants. For this reason, it was suggested that multiple forms of evaluation are preferred, where possible, to accommodate different levels of familiarity/comfort to alternative evaluation devices.
- Evaluator noted that the need for evaluation increases where the subject matter/topics addressed in the PLEI program touch on more sensitive issues. It was suggested that
"should this Courtlink…incorporate materials on family violence and sexual abuse, it would be important to undertake an extensive evaluation, requiring multiple program contacts with the participating youth, teachers and family members" (p.20)
.
Evaluation Methodologies
- Evaluator sat in on the full program at five sites throughout the province. Observations were recorded as well as substantive material covered in program activities.
- Informal interviews and discussions with teachers and students during and after program.
- Interview with the facilitator after each day of the program at all sites (10).
- Collection of quantitative data using written questionnaires from four groups: (all forms are appended to the report in Appendix A through D).
- Students
- All observers (teachers, teach aides, volunteers, committee members, etc.).
- Court personnel – questionnaire addressed pre-program information package.
- Teachers – evaluation of pre-program materials
- Date modified: