Evaluation of Public Legal Education And Information: An Annotated Bibliography
4. Evaluation Reports (continued)
4. Evaluation Reports (continued)
4.2 Useful Materials and Sources
Although legal aid is often seen as separate from PLEI, the two services overlap more or less directly in different provinces. Given the historical connection to government funding, legal aid services have been the subject of numerous assessments and evaluations. Due to this pattern, these types of reports (arguably beyond PLEI provision) should be deferred to in developing PLEI evaluation. The first two entries were included by way of example as they are commonly referenced in other evaluation sources.
Brantingham, P. & Brantingham, P. (1984). An evaluation of legal aid in British Columbia. Bureau of Programme Evaluation and Internal Audit, Department of Justice Canada. Available from the Legal Resource Centre, Legal Studies Program, University of Alberta.
This older report is a very comprehensive evaluation of legal aid in British Columbia, which includes "indirect" services such as PLEI. However, the aspect that pertains to PLEI evaluation is predominantly descriptive in nature and focuses on which organizations and offices at that time tended to prioritize PLEI. This evaluation was underway at the time when the province introduced budgetary cuts to legal aid, and therefore it includes some discussion of the relationship between the Legal Services Society, whose mandate includes PLEI delivery and the provision of other legal aid services. The evaluation was comprised of an accessibility analysis, an impact analysis and quality studies.
Canadian Bar Association Standing Committee, National Legal Aid Liaison Committee. (1987). Legal aid delivery models: A discussion paper. Available from the Legal Resource Centre, Legal Studies Program, University of Alberta.
This report focuses specifically on legal aid models and only peripherally addresses the impact of various models on community legal education. Here Community Legal Education is defined as the provision of information to low-income clientele who are already receiving some legal representation. Therefore, it does not include legal education to the broader public per se. However, this paper provides a good overview of various legal aid models and insofar as PLEI is included under legal aid in some provinces such as Ontario and BC, it is a useful analysis of how different models interact with complementary services such as public interest advocacy (test case litigation), law reform initiatives and community legal education.
Currie, A. (1999). Legal aid delivery in Canada: Past experience and future directions. Research and Statistics Division, Department of Justice Canada.
This report uses empirical studies to assess the cost and quality of different delivery models of legal aid. The author builds on the Canadian Bar Association's report that was published in 1987 and offers a comprehensive overview of findings and analyses of various evaluation studies conducted between 1981 and 1998.
Currie, A. & McEwon, C. (1998). Assisted self-representation in criminal legal aid: An experiment in limited service delivery. Research and Statistics Division, Department of Justice Canada.
This report examines the factors affecting self-representation against a criminal charge once an individual has been refused legal aid. The review considered who rejected explanatory materials when offered, who accepted these materials and what difficulties these individuals face in using the materials for effective self-representation. The study looked at the usefulness of both general and specific PLEI materials and identified problems such as language barriers and overly complicated court proceedings. The authors concluded that the sample size used was too small to draw sufficient data for quantitative analysis and in-depth interviews would have produced more useful information. However, they did assert that although self-representation is not a promising or viable alternative, the PLEI materials in this area could be useful in other contexts to describe general court proceedings and provide information on other resources available to those in need of legal assistance.
Currie, J. & Roberts, T. (1986). An evaluation resource book for public legal education and information organizations. Research and Statistics Section, Policy, Programs and Research Branch, Department of Justice Canada.
General Overview
This resource book is a practical evaluation tool and a source of information and ideas regarding approaches to PLEI evaluation. The book was designed to "deal with the evaluation needs of staff from large quasi-government PLEI organizations to small experimental programs; from well-developed programs with long histories to short-term focused projects"
(p.1). The book is broken into six Modules that can be used independently from one another, although Module I is generally recommended for all PLEI evaluations. Though most of the Modules are task-oriented, they also provide background information to assist in making choices about appropriate evaluation methods. Some of the Modules are quite technical and will only be useful for those who have significant time and resources to devote to evaluation. However, the whole spectrum is covered in this one resource and many of the sections can be isolated for discrete purposes. This is an excellent resource for any organization that is involved in the provision of PLEI, and despite the date of publication it remains relevant to current PLEI services and programs.
The six Modules are:
- PLANNING YOUR EVALUTION
- USING AN EVALUATION CONSULTANT
- TYPES OF EVALUTION YOU CAN DO
- GATHERING THE DATA
- ANALYZING THE DATA
- WRITING AND UTILIZING THE DATA
For consistency, examples of key issues/topics that are addressed in the resource book have been inserted under the same thematic headings that are used throughout this bibliography.
The Challenges of PLEI Evaluation
- Who will do the evaluation: internal (staff) evaluator or external evaluator
- Conflicts between the evaluator and the group
- What to do if you have no power to affect the evaluation
Connections between Goals or PLEI and its Evaluation
- Goal oriented approach
- How to develop evaluation goals and objectives
- How can you determine whether the utilization rate of your program is satisfactory
- How does the type of PLEI program affect data collection
- Relating data collection to your evaluation purposes
Challenges of Funders' Evaluation Requirements
- Establishing a budget for an evaluation
- Cutting evaluation costs
- Determining costs and benefits for PLEI programs: difficulties and solutions
Evaluation Methodologies
- Needs assessment
- Program evaluations; organizational assessment; assessment of program utilization
- Materials assessment
- Impact assessment
- Cost effectiveness assessment
- Ways of structuring the questionnaire
- Training interviewers
Hessing, M. & Thompson, L. (1984). The telephone legal information service of the legal resource centre of British Columbia: Evaluation report. Available from the Legal Resource Center, Legal Studies Program, University of Alberta.
General Overview
The legal information service is a telephone line where trained librarians offer legal-related information to the public as well as to intermediaries (businesses, government, schools, libraries and other private organizations such as NGO's). The staff librarians commissioned this evaluation study which looked at the phone lines exclusively and not the other PLEI services and programs offered in conjunction with the Legal Services Society of B.C. After many years of service they wanted statistical and qualitative data, collected from an objective party, that could identify if and how they could improve the provision of legal telephone services in the province. In addition to assessing the content of information disseminated and the utility and impact of the service provided to clients, the evaluator recognized the importance of considering other variables such as staff scheduling, the physical surroundings and office lay out, training and upgrading and quality control.
Common Components of Evaluation
- External evaluation
- Conducted after services had been in place for seven or eight years. Although staff librarians maintained a monthly record of caller statistics, no formal evaluation had occurred to date.
- Evaluator noted that current methods of quality control such as "listening in" to calls are helpful in ensuring that information offered is accurate, appropriate and sufficient and that responses are consistent. Ongoing quality control was advised and recommendations were made to increase internal monitoring through information manuals, workshops and discussions.
The Challenges of PLEI Evaluation
- Phone staff self-selected the "clients" that would participate in the study. Therefore, if someone was too emotional or there was a language barrier they were not asked to participate. This may produce some inaccuracies insofar as those who were not fluent in English or had other unique characteristics were not represented in the study.
- By involving staff in evaluation there is an increased risk that data collected is incomplete due to the added burden of collecting data and tracking their responses in addition to performing their work obligations. This was seen in a failure to include certain aspects of their "responses" to callers.
- It is difficult to ensure that methods of evaluations accommodate all the uses/users of the services. For example, although this service is primarily rendered by phone there are often written submissions for legal information that arrive via mail. These enquiries were not included in the study, as client profiles were limited to incoming calls.
Connection between Goals of PLEI and Evaluation
- Staff specified ten questions about the services they were providing that they wanted addressed in the evaluation. However, the opportunity for external evaluation was seen as a means of securing
"not only ‘factual' and quantifiable data, but also as a means of introducing a fresh and objective perspective into the work process for the staff members themselves"
(p.2).
Evaluation Methodologies
- Client information sheet used by staff during initial call (statistical element)
- Staff were trained on how to use the forms
- Detailed client assessment through phone (call back) interview
- Clients were randomly selected (1 in 10) for call back two weeks after initial call
- Legal assessment for accuracy and adequacy by staff lawyers from another office
- In-depth and informal staff interviews
- Observations
Prairie Research Associates, The Coopers and Lybrand Consulting Group & Linden, R. (1991). Evaluation of the legal aid Manitoba expanded eligibility program. Research and Development Directorate, Department of Justice Canada.
General Overview
Although this report did not seek to evaluate any aspect of PLEI, two of the "main issues" identified for evaluation commonly constitute objectives for PLEI evaluation. These are client satisfaction and attitude assessment toward the program (here the attitudes considered were about repayment of fees). Questions were divided into categories, which were further subdivided into specific issues/questions to be addressed. This is a useful approach to evaluation as "multiple lines of evidence strengthen the conclusions and provide a stronger basis for the recommendations"
(p.5). Findings were also organized by the method used to collect information i.e., survey, interview, file review, as well as which population group would be asked each question. Although it is time consuming to develop separate questionnaires for each group of informants, this is often necessary as clients and staff engage with materials and programs from unique perspectives. Furthermore, it is preferable to tailor evaluation questions to specific PLEI clients/students on the basis of characteristics that distinguish their access, knowledge and use from other audiences. The information in this report that may be of use to PLEI evaluators is the "Methodology" section. As in most cases the content must be modified to fit the project undergoing evaluation, however the report contains some elaboration on the steps involved for administering each type of assessment tool.
Common Components of Evaluation
- External evaluation
- Conducted after one year of pilot program
- Evaluation set out the main issues to be examined (similar to goals).
- Recommendations included mechanisms to assist in management efficiencies dealing, in part, with ongoing monitoring mechanisms.
The Challenges of PLEI Evaluation
- It was a difficult population to locate as there were problems in retrieving phone numbers. Approximately 40% of the informants had no telephone or an incorrect number was on file.
Evaluation Methodologies
- Data sources included: administrative file review; applications for eligibility; accounting information; case management file review; surveys; interviews.
- There was an intensive review of accounting information due to the nature of the project.
- Report outlines how different evaluation tools were used i.e., what steps were taken for administrative file review, client surveys, interviews with private Bar, etc.
- Date modified: