Evaluation of the Justice Partnership and Innovation Program

Appendix B: Data collection instruments

Key informant interview guides

Key informant interview guide for the Department of Justice Canada representatives

The Department of Justice Canada (the Department) is conducting an evaluation of the Justice Partnership and Innovation Program (JPIP or “the Program”). This program aims to assist the Department in achieving government priorities in the areas of access to justice, family violence, public legal education and information (PLEI), and missing and murdered Aboriginal women/violence against Aboriginal women and girls. The evaluation covers a four-year period (2012–13 to 2015–16) and focusses on the Program’s relevance and performance.

PRA Inc., an independent research company, has been contracted to support the evaluation. A component of the evaluation involves interviews with stakeholders who have knowledge of the JPIP. The interview will be conducted by telephone and will be audio recorded with your permission. The information we gather from the interviews will be summarized in aggregate form. Interview notes will not be shared outside of PRA Inc. and the Evaluation Division of the Department of Justice Canada. We recognize that representatives may have familiarity mainly with specific areas and/or objectives of JPIP funding. If some questions are not applicable to your area of knowledge, please let the interviewer know and they will skip to the next question.

Prior to the interview, please review the JPIP objectives provided on the last page.

Introduction

1. Please briefly describe your involvement with the JPIP, and how long you have been involved with the Program.

Relevance

2. Given the JPIP objectives (provided on the last page), to what extent do you see that the JPIP is still needed? Can you speak to the relevance of the JPIP’s objectives for meeting the justice needs of Canadians? (Probe: Do the objectives align with the justice needs of Canadians? Are there any particular areas of justice needs not being met? Are there specific groups that have greater needs? Are there emerging trends or changing needs?) (1.1.1)

3. Please explain how the JPIP aligns with or does not align with Government of Canada priorities, and specifically with Department of Justice priorities. What, if any, changes have been made to the Program to reflect changing government or departmental priorities? (2.1.1)

4. Does the funding provided by the JPIP complement initiatives funded by other federal departments or by other jurisdictions? Is there overlap with any other initiatives? (2.1.1, 5.1.1)

5. What federal roles and responsibilities does the JPIP address and how? Please give specific examples if you are able to (e.g., legislative authorities). (3.1.1)

Performance – achievement of outcomes

6. To what extent has the JPIP enabled recipients to take part in their funded initiatives/ activities? How has the funding helped recipients in building capacity in the funded areas? Please give specific examples, as you are able, at both a Canadian and international level. (Probe: Would recipients be able to continue with their funded activities without JPIP support?) (4.1.2, 4.1.3, 4.1.4)

7. Can you comment on partnerships that have been formed as a result of the JPIP? How have these partnerships contributed to increased awareness, knowledge, and understanding of the justice system? Are partners satisfied with their partnerships? (4.1.1, 4.1.7)

8. (For stakeholders familiar with PLEI activities) Please describe some of the PLEI activities undertaken by funded recipients. (4.1.8, 4.2.2, 4.2.4, 4.2.5, 4.4.1)

  1. Who are the target audiences? How successful are these activities in reaching these audiences? (Probe: What, if any, barriers exist and how were they addressed?)
  2. To what extent have these activities contributed to increased awareness and understanding of justice issues? Has this understanding changed over the past five years or so?
  3. Have they contributed to increased access to justice? If so, how? Has this changed over the past five years or so?
  4. Is there a sufficient level of this type of information available? Why or why not?
  5. Are there any barriers to accessing this information (e.g., language issues), and if so, how are these addressed?

9. (For stakeholders familiar with educational funding to support Métis and non-status Indians in pursuing studies in law) Can you comment on the number of Métis and non-status Indians that have received financial assistance through the JPIP for pursuing their studies in law? (4.2.2, 4.4.2)

  1. Are all eligible students accessing the JPIP? What, if any, barriers exist in accessing these funds?
  2. What difference does the JPIP funding make to these students?
  3. To what extent is the JPIP funding contributing to more equitable representation of Indigenous peoples in the legal profession?

10. (For stakeholders familiar with named grants) For those named grant recipients you are familiar with, please describe some of the activities undertaken and how they contribute to the JPIP objectives (see last page). How have these activities contributed to each of the below? Please give specific examples as you are able. (4.2.3, 4.2.5, 4.4.1)

  1. Increased awareness and understanding of justice issues for your target audience or in general. Has this changed over the past five years or so?
  2. Increased access to justice. Has this changed over the past five years or so?

11. (For stakeholders familiar with other types of funded projects other than those operating at the international level) For those projects you are familiar with, please describe some of the activities undertaken, their target audience, and how they contribute to the JPIP objectives (see last page). How have these activities contributed to each of the aspects below? Please give specific examples as you are able to. (4.1.8, 4.2.3, 4.2.5, 4.4.1)

  1. Increased awareness and understanding of justice issues for your target audience or in general. Has this changed over the past five years or so?
  2. Increased access to justice. Has this changed over the past five years or so?
  3. Are there any barriers to the target audience in accessing the funded activities (e.g., language issues or other challenges), and if so, how are these addressed?

12. (For stakeholders familiar with funded initiatives that operate at the international level) Are you able to comment on the JPIP’s involvement at the international level? Which JPIP-funded initiatives are you familiar with? How has this involvement contributed to each of the following? Please give specific examples as you are able to. (4,1,4, 4.3.1, 4.3.2)

  1. promoting Canadian legal interests internationally
  2. contributing to Canada’s influence at the international level
  3. contributing to unification of rules
  4. promoting international cooperation with respect to access to justice

13. Please describe, as you are able, how you see JPIP-funded initiatives contributing towards strengthening the Canadian legal framework? Towards a more harmonized legislation? Please give specific examples as you are able, both in terms of funded recipients and how their activities make such contributions. (4.5.1, 4.5.2)

14. Are you aware of any of the JPIP-funded activities that have helped to inform policy at the federal level? At other jurisdictional levels, including internationally? If yes, can you give specific examples and the applicable policy or policies? (4.5.2)

15. Are you aware of any other contributions besides those we have already discussed that funded projects have made towards increasing awareness, knowledge, and access to justice? This could be through development of informational materials; new strategies, models or tools; research; conferences; supports for victims of crime, etc. If so, please describe these and the contributions they have made. (4.2.3, 4.2.5, 4.4.1)

16. Can you comment on specific success factors that have assisted the JPIP in achieving its objectives? On specific constraints for achieving objectives? (4.6.1, 4.6.2)

  1. Have there been any unexpected outcomes from the JPIP, either positive or negative? (4.7.1)
Performance – economy and efficiency

17. Given the level of annual funding to JPIP recipients, do you believe that the benefits accruing to Canada and the Canadian public are reasonable? Please explain. (5.1.1)

18. Can you comment on the extent that funds are leveraged from other sources as a result of the funding from the JPIP? Would this other funding have occurred without the JPIP? (5.1.1)

19. Are you satisfied with the application and recipient monitoring process? Do any challenges exist with either the application or reporting process? Do you have any suggestions for making either of these processes more efficient and/or effective? (5.2.1, 5.2.2)

20. Are the Department of Justice’s staffing resources for the JPIP sufficient to manage applications and provide oversight to funded recipients? Please explain any challenges that exist. (5.2.1, 5.2.2)

21. Are you aware of alternative ways to achieve some or all of the objectives of the JPIP more cost-effectively? If yes, please describe them. (5.1.1)

22. The previous evaluation of the JPIP made the two recommendations below. Can you comment on to what extent each of these has been implemented? (5.3.1)

  1. Streamline the objectives to focus on the primary goals of the Program.
  2. Strengthen performance measurement by requiring funded organizations to collect some data from beneficiaries as a part of the funding agreement.

23. Do you have any other comments you would like to make about the JPIP?

Thank-you for your participation.

JPIP objectives

The overall objective of the JPIP is to support the policy directions of the Department of Justice on issues related to family violence, missing and murdered Aboriginal women, access to justice, and the harmonization of private international law. The specific objectives are to:

Key informant interview guide for recipients receiving funding for PLEI activities

The Department of Justice Canada (the Department) is conducting an evaluation of the Justice Partnership and Innovation Program (JPIP or “the Program”). This program aims to assist the Department in achieving government priorities in the areas of access to justice, family violence, public legal education and information (PLEI), and missing and murdered Aboriginal women/violence against Aboriginal women and girls. The evaluation covers a four-year period (2012–13 to 2015–16) and focusses on the Program’s relevance and performance.

PRA Inc., an independent research company, has been contracted to support the evaluation. A component of the evaluation involves interviews with stakeholders, including funding recipients, who have knowledge of the JPIP. This interview will be conducted by telephone and will be audio recorded with your permission. The information we gather from the interviews will be summarized in aggregate form. Interview notes will not be shared outside of PRA Inc. and the Evaluation Division of the Department of Justice Canada. If you are unable to answer any questions, please let the interviewer know and they will skip to the next question.

Prior to the interview, please review the JPIP objectives provided on the last page.

Introduction

1. Please briefly describe your organization’s involvement with the JPIP, and how long you have been involved with the Program.

Relevance

2. Can you speak to the need for PLEI in Canada in general and in your jurisdiction specifically? What are the areas of need? Are there specific groups that have greater or specific needs? Are there any emerging or changing needs? (1.1.1)

3. Given the JPIP objectives (provided on the last page), to what extent do you see that the JPIP is still needed in general? How about specifically for your organization? Can you speak to the relevance of the JPIP’s objectives for meeting Canadians’ justice needs? (1.1.1) (Probe: Do the objectives align with the justice needs of Canadians? Are there any particular justice needs not being met? Are there specific groups that have greater needs? Are there any other emerging trends or changing needs not identified in #2 above?)

Performance – achievement of outcomes

4. To what extent has the JPIP enabled organizations such as yours to take part in their PLEI activities? How has the funding helped your organization in building capacity for providing PLEI services? Please give specific examples as you are able. (Probe: Would your organization be able to continue with their PLEI activities without JPIP support?) (4.1.2, 4.1.3, 4.1.4)

5. Please describe some of the PLEI activities your organization undertakes and the objectives for these activities. (4.2.2, 4.2.4, 4.2.5)

  1. What are some of the informational materials and tools you have developed or are able to provide through the JPIP funding?
  2. In your opinion, is there a sufficient level of this type of information available in your jurisdiction? Why or why not?

6. Who are the target audiences for your PLEI activities? How satisfied are you at your organization’s ability to reach these audiences? (4.1.8, 4.2.4)

  1. What steps does your organization take to try to reach this audience?
  2. What, if any, barriers or challenges exist in reaching this audience and how are they addressed?
  3. What, if any, steps are taken for reaching those members of the target audience with specific language needs?

7. In your opinion, to what extent have these activities contributed to each of the below? (4.2.2, 4.4.1)

  1. Increased awareness and understanding of justice issues for your target audience. Has this changed over the past five years or so?
  2. Increased access to justice. If so, how? And has this changed over the past five years or so?

8. Has your organization formed any partnerships with respect to your PLEI services? If yes, could you describe the partnerships and what they contribute to your PLEI activities? (4.1.1)

9. (If have partnerships) Are you satisfied with your partnerships and how they have helped you with your PLEI activities? Why or why not? (4.1.7)

  1. Have they assisted your organization with increasing awareness, knowledge, and understanding of the justice system? Please explain.

10. Are you able to comment on the extent to which JPIP-funded initiatives have contributed towards strengthening the Canadian legal framework? Towards a more harmonized legislation? Please give specific examples as you are able, both in terms of funded recipients and the contributions they make. (4.5.1, 4.5.2)

11. Are you able to comment on whether any of the JPIP-funded activities have helped to inform policy at the federal level? At other jurisdictional levels, including internationally? If yes, can you give specific examples? (4.5.2)

12. Are you aware of any specific success factors that have assisted the JPIP in achieving its objectives? On specific constraints for achieving objectives? (4.6.1, 4.6.2)

  1. Have there been any unexpected outcomes from the JPIP, either positive or negative? (4.7.1)
Performance – economy and efficiency

13. Given the level of annual funding to JPIP recipients, such as your organization and in general, do you believe that the benefits accruing to Canada and the Canadian public are reasonable? Please explain. (5.1.1)

14. Is your organization able to leverage funding from other sources as a result of the funding from the JPIP? If yes, please describe the funding (including the source) and what it has meant to your PLEI activities? Would this other funding have occurred without the JPIP? (5.1.1)

15. Are you satisfied with the JPIP application and reporting process? Do any challenges exist for your organization with either the application or reporting process? Do you have any suggestions for making either of these processes more efficient and/or effective? (5.2.1, 5.2.2)

16. In your opinion, are the Department of Justice’s staffing resources for the JPIP sufficient to manage applications and provide any needed input or assistance to funded recipients? Please explain. (5.2.1, 5.2.2)

17. Are you aware of alternative ways to achieve some or all of the objectives of the JPIP more cost-effectively? If yes, please describe them. (5.1.1)

18. Do you have any other comments you would like to make about the JPIP?

Thank-you for your participation.

JPIP objectives

The overall objective of the JPIP is to support the policy directions of the Department of Justice on issues related to family violence, missing and murdered Aboriginal women, access to justice, and the harmonization of private international law. The specific objectives are to:

Key informant interview guide for JPIP funding recipients (other than PLEI recipients)

The Department of Justice Canada (the Department) is conducting an evaluation of the Justice Partnership and Innovation Program (JPIP or “the Program”). This program aims to assist the Department in achieving government priorities in the areas of access to justice, family violence, public legal education and information (PLEI), and missing and murdered Aboriginal women/violence against Aboriginal women and girls. The evaluation covers a four-year period (2012–13 to 2015–16) and focusses on the Program’s relevance and performance.

PRA Inc., an independent research company, has been contracted to support the evaluation. A component of the evaluation involves interviews with stakeholders, including funding recipients, who have knowledge of the JPIP. The interview will be conducted by telephone and will be audio recorded with your permission. The information we gather from the interviews will be summarized in aggregate form. Interview notes will not be shared outside of PRA Inc. and the Evaluation Division of the Department of Justice Canada. If you are unable to answer any questions, please let the interviewer know and they will skip to the next question.

Prior to the interview, please review the JPIP objectives provided on the last page.

Introduction

1. Please briefly describe your organization’s involvement with the JPIP, and how long you have been involved with the Program.

Relevance

2. Looking at the JPIP objectives listed on the last page, which of these objectives would best apply to the activities your organization undertakes with JPIP funding? What are Canadians’ justice needs related to these activities? Are there any emerging or changing needs?

3. Given the JPIP objectives, can you speak to their relevance for meeting Canadians’ justice needs? To what extent do you see that the JPIP is still needed in general? How about specifically for your organization? (Probe: Do the objectives align with Canadians’ justice needs? Are there any particular areas of justice needs not being met? Are there specific groups that have greater needs? Are there any other emerging trends or changing needs not identified in #2 above?) (1.1.1)

Performance – achievement of outcomes

4. To what extent has the JPIP enabled recipients, such as your organization, to take part in their funded initiatives/activities? How has the funding helped your organization in building capacity in your funded area? Please give specific examples as you are able, at both a Canadian and international level, if applicable. (Probe: Would your organization be able to continue with the funded activities without JPIP support?) (4.1.2, 4.1.3, 4.1.4)

5. Please briefly describe the activities your organization undertakes through JPIP funding. (4.2.5, 4.4.1)

  1. Are there any specific informational materials, strategies, models, research, or supports you have either developed or are able to make available through the JPIP funding?
  2. In your opinion, is there a sufficient level of these types of activities taking place in Canada in general? In your jurisdiction? Why or why not?

6. Who are the target audiences for your funded activities? How satisfied are you by your organization’s ability to reach these audiences? (4.1.8, 4.2.5)

  1. What steps does your organization take to try to reach these audiences?
  2. What, if any, barriers or challenges are there in reaching these audiences and how are they addressed?
  3. What, if any, steps are taken for reaching those members of the target audiences with specific language needs?

7. In your opinion, to what extent have these activities contributed to each of the below? (4.2.3, 4.2.5, 4.4.1)

  1. Increased awareness and understanding of justice issues for your target audiences or in general? Has this changed over the past five years or so?
  2. Increased access to justice? Has this changed over the past five years or so?

8. Has your organization formed any partnerships with respect to your funded activities? If yes, could you describe the partnerships and what they contribute to your JPIP-funded activities? (4.1.1)

9. (If they have partnerships) Are you satisfied with your partnerships and how they have helped you with your funded activities? Why or why not? (4.1.7)

  1. Have they assisted your organization with increasing awareness, knowledge, and understanding of the justice system? Please explain.

10. (For recipients with funded initiatives that operate at the international level) Please describe your funded initiative’s activities and involvement at the international level. How has this involvement contributed to each of the following? Please give specific examples as you are able. (4,1,4, 4.3.1, 4.3.2)

  1. promoting Canadian legal interests internationally
  2. contributing to Canada’s influence at the international level
  3. contributing to unification of rules
  4. promoting international cooperation with respect to access to justice

11. If applicable, please describe your funded activities that you believe contribute towards strengthening the Canadian legal framework? Towards a more harmonized legislation? Please give specific examples as you are able, both in terms of the activities and the contributions they make. (4.5.1, 4.5.2)

12. Are you aware of any of how your organization’s JPIP-funded activities have helped to inform policy at the federal level? At other jurisdictional levels, including internationally? If yes, can you give specific examples? Are you aware of any other JPIP-funded activities that make such contributions? If so, could you give specific examples? (4.5.2)

13. Are you aware of any specific success factors that have assisted the JPIP in achieving its objectives? Or specific constraints for achieving objectives? (4.6.1, 4.6.2)

14. Have there been any unexpected outcomes from the JPIP, either positive or negative? (4.7.1)

Performance – economy and efficiency

15. Given the level of annual funding to JPIP recipients, such as to your organization and in general, do you believe that the benefits accruing to Canada and the Canadian public are reasonable? Please explain. (5.1.1)

16. Is your organization able to leverage funding from other sources as a result of the funding from the JPIP? If yes, please describe the funding (including the source) and what it has meant to your funded activities. Would this other funding have occurred without the JPIP? (5.1.1)

17. Are you satisfied with the JPIP application and reporting process? Do any challenges exist for your organization with either the application or reporting process? Do you have any suggestions for making either of these processes more efficient and/or effective? (5.2.1, 5.2.2)

18. In your opinion, are the Department of Justice staffing resources for the JPIP sufficient to manage applications and provide any needed input or assistance to funded recipients? Please explain. (5.2.1, 5.2.2)

19. Are you aware of alternative ways to achieve some or all of the objectives of the JPIP more cost-effectively? If yes, please describe. (5.1.1)

20. Do you have any other comments you would like to make about the JPIP?

Thank-you for your participation.

JPIP objectives

The overall objective of the JPIP is to support the policy directions of the Department of Justice on issues related to family violence, missing and murdered Aboriginal women, access to justice, and the harmonization of private international law. The specific objectives are to:

Key informant interview guide for AJA PLEI recipients

The Department of Justice Canada (the Department) is conducting an evaluation of the Justice Partnership and Innovation Program (JPIP). This program aims to assist the Department in achieving government priorities in the areas of access to justice, family violence, public legal education and information (PLEI), and missing and murdered Aboriginal women/violence against Aboriginal women and girls. PRA Inc., an independent research company, has been contracted to support the evaluation.

The evaluation also covers the PLEI component of the Access to Justice Agreements (AJA). As such, the evaluation involves interviews with several representatives of Canada’s territories who can speak to the PLEI activities offered. This interview will be conducted by telephone and will be audio recorded with your permission. The information we gather from the interviews will be summarized in aggregate form. Interview notes will not be shared outside of PRA Inc. and the Evaluation Division of the Department of Justice Canada.

1. Please briefly describe your position and your role in delivering PLEI services and activities in your territory.

2. Please describe what types of PLEI services and activities are delivered in your territory.

3. How are these PLEI services delivered (e.g., who delivers them; how are materials distributed)?

4. Are there any gaps in services or any PLEI needs not being met? If yes, what are these gaps or unmet needs? What steps are taken to overcome these gaps/unmet needs?

5. What delivery challenges exist in the territories with respect to PLEI? Again, what steps are taken to overcome these challenges?

6. How have your PLEI services contributed to increased awareness and understanding of justice issues in your territory?

7. Do you have any other comments you would like to make about your territory’s PLEI services?

Thank-you for your participation.

Case study guides

Case study interview guide (PLEI recipients)

The Department of Justice Canada (the Department) is conducting an evaluation of the Justice Partnership and Innovation Program (JPIP or “the Program”). This program aims to help the Department achieve government priorities in the areas of access to justice, family violence, public legal education and information (PLEI), and missing and murdered Aboriginal women/violence against Aboriginal women and girls. The evaluation covers a four-year period (2012–13 to 2015–16) and focusses on the Program’s relevance and performance.

PRA Inc., an independent research company, has been contracted to support the evaluation.

Case studies of several funded projects/initiatives comprise a component of the evaluation. Case studies will involve a review of documents and several interviews with recipients and other relevant stakeholders. The interviews will be conducted by telephone and will be audio-recorded with your permission. Interview notes will not be shared outside of PRA Inc. and the Evaluation Division of the Department of Justice Canada. If you are unable to answer any questions, please let the interviewer know and they will skip to the next question.

Prior to the interview, please review the JPIP objectives provided on the last page.

1. Please briefly describe your organization and how its activities and objectives relate to the JPIP objectives.

  1. How long has your organization received JPIP funding for PLEI activities?

2. Can you speak to the need for PLEI in Canada in general and in your jurisdiction specifically? What are the areas of need? Are there specific groups that have greater or specific needs? Are there any emerging or changing needs? (1.1.1)

3. To what extent has the JPIP enabled your organization to take part in their PLEI activities? (4.1.2, 4.1.3)

  1. How has the funding helped your organization in building capacity for providing PLEI services? Please give specific examples as you are able.
  2. Would your organization be able to continue with their PLEI activities without JPIP support? (Probes: Would the activities have been offered/conducted at all? Offered in a different manner or scale?)

4. Please describe some of the PLEI activities your organization undertakes and the objectives for these activities. (1.2.2, 4.2.2, 4.2.4, 4.2.5, 4.4.1)

5. What are some of the informational materials and tools you have developed or are able to provide through the JPIP funding?

6. In your opinion, is there a sufficient level of this type of information available in your jurisdiction? Why or why not?

7. Who are the target audiences for your PLEI activities? How satisfied are you with your organization’s ability to reach these audiences? (1.2.1, 4.1.8)

  1. What steps does your organization take to try to reach this audience?
  2. What, if any, barriers or challenges exist in reaching this audience and how are they addressed?
  3. What, if any, steps are taken for reaching those members of the target audience with specific language needs?

8. Looking at the JPIP objectives, which of these objectives would best apply to the PLEI activities your organization undertakes? Can you describe how your funded activities have been able to contribute to those objectives? (4.2.2, 4.2.4, 4.2.5, 4.4.1)

9. In general, to what extent have these activities contributed to: (4.2.2, 4.4.1)

  1. Increased awareness and understanding of justice issues for your target audience or in general? Has this changed over the past five years or so?
  2. Increased access to justice? Has this changed over the past five years or so?

10. Has your organization formed any partnerships with respect to your PLEI activities? If yes, please describe these partnerships and what they contribute to this project (e.g., direct funding, in-kind funding and if so, what kind)? (4.1.1, 4.1.7, 5.1.1)

  1. What have your partnerships meant to your organization in terms of achieving your PLEI objectives?
  2. Were these partnerships formed because of your involvement with this JPIP project? (Probe: would they have happened without the JPIP funding?)
  3. How satisfied is your organization with these partnerships? Why or why not?

11. Does your organization receive funding from any other sources for this project other than what you might have described above under question 10? If yes please describe these other source of funding. (5.1.1)

12. Did the JPIP funding assist your organization in leveraging other sources of funds either from partners or other sources reported in the previous question? Would this funding have occurred without the JPIP funding? (5.1.1)

  1. Relative to other sources of funding (all other sources, including those from partners), what proportion of the funding for this project is from JPIP funding?

13. Are you aware of any other PLEI initiatives or activities similar to those undertaken by your organization being conducted in your jurisdiction? If yes, who funds these activities (e.g., other federal departments, other governments, or non-governmental organizations)? Are there any measures your organization takes to avoid duplication with other initiatives? (5.1.1)

14. Can you comment on specific success factors that have assisted your organization in achieving its PLEI objectives? On specific constraints for achieving objectives? (4.6.1, 4.6.2)

  1. Have there been any unexpected outcomes, either positive or negative, from the JPIP funding? (4.7.1)

15. Do you have any other comments you would like to make about your JPIP-funded project/initiative or about the JPIP in general?

Thank-you for your participation.

JPIP objectives

The overall objective of the JPIP is to support the policy directions of the Department of Justice on issues related to family violence, missing and murdered Aboriginal women, access to justice, and the harmonization of private international law. The specific objectives are to:

Case study interview guide (other than PLEI recipients)

The Department of Justice Canada (the Department) is conducting an evaluation of the Justice Partnership and Innovation Program (JPIP or “the Program”). This program aims to help the Department achieve government priorities in the areas of access to justice, family violence, public legal education and information (PLEI), and missing and murdered Aboriginal women/violence against Aboriginal women and girls. The evaluation covers a four-year period (2012–13 to 2015–16) and focusses on the Program’s relevance and performance.

PRA Inc., an independent research company, has been contracted to support the evaluation.

Case studies of several funded projects/initiatives comprise a component of the evaluation. Case studies will involve a review of documents and several interviews with recipients and other relevant stakeholders. The interviews will be conducted by telephone and will be audio-recorded with your permission. Interview notes will not be shared outside of PRA Inc. and the Evaluation Division of the Department of Justice Canada. If you are unable to answer any questions, please let the interviewer know and they will skip to the next question.

Prior to the interview, please review the JPIP objectives provided on the last page.

1. Please briefly describe your organization and how its activities and objectives relate to the JPIP objectives.

2. Please briefly describe the JPIP-funded project/initiative we are discussing today. (1.2.2)

  1. How long has your organization received funding for this project?

3. What needs related to access to justice is this project/initiative intended to meet? In your view, are there any emerging or changing needs in this area? If yes, how is this project/initiative able to respond to these emerging or changing needs? (1.1.1, 1.2.1)

4. To what extent has the JPIP funding enabled your organization to take part in the funded initiatives/activities? (4.1.2, 4.1.3, 4.1.4)

  1. How has the funding helped your organization in building capacity in the area funded through this project? Please give specific examples as you are able, at both a Canadian and international level, if applicable.
  2. Would your organization be able to continue with the funded activities without JPIP support? (Probes: Would the activities have been offered/conducted at all? Offered in a different manner or scale?)

5. Please give examples of the activities your organization undertakes through this JPIP funded project/initiative. (1.2.2, 4.2.5, 4.4.1)

  1. Are there any specific informational materials, strategies, models, research, or supports you have either developed or are able to make available through the JPIP funding?
  2. In your opinion, is there a sufficient level of these types of activities taking place in Canada in general? In your jurisdiction? Why or why not?

6. Who are the target audiences for this JPIP funded project/initiative? How satisfied are you with your organization’s ability to reach these audiences? (1.2.1, 4.1.8)

  1. What steps does your organization take to try to reach this audience?
  2. What, if any, barriers or challenges are there in reaching this audience and how are they addressed?
  3. What, if any, steps are taken for reaching those members of the target audience with specific language needs?

7. Looking at the JPIP objectives, which of these objectives would best apply to this project? Can you describe how your funded project/initiative has been able to contribute toward those objectives? (4.2.3, 4.2.5, 4.4.1)

8. In general, to what extent have the activities funded through this JPIP project contributed to: (4.2.3, 4.4.1)

  1. Increased awareness and understanding of justice issues for your target audience or in general? Has this changed over the past five years or so?
  2. Increased access to justice? Has this changed over the past five years or so?

9. Has your organization formed any partnerships for delivering these activities? If yes, please describe these partnerships and what they contribute to this project (e.g., direct funding, in-kind funding and if so, what kind)? (4.1.1, 4.1.7, 5.1.1)

  1. What have your partnerships meant to your organization in terms of achieving the objectives of this funded project/initiative?
  2. Were these partnerships formed because of your involvement with this JPIP project? (Probe: would they have happened without the JPIP funding?)
  3. How satisfied is your organization with these partnerships? Why or why not?

10. Does your organization receive funding from any other sources for this project other than what you might have described above under question 12? If yes, please describe these other source of funding. (5.1.1)

11. Did the JPIP funding assist your organization in leveraging other sources of funds either from partners or other sources reported in the previous question? Would this funding have occurred without the JPIP funding? (5.1.1)

  1. Relative to other sources of funding (all other sources, including those from partners), what proportion of the funding for this project is from JPIP funding?

12. Are you aware of any other initiatives or activities similar to those funded by this JPIP project being conducted elsewhere? If yes, who funds these activities (e.g., other federal departments, other governments, or non-governmental organizations)? Are there any measures your organization takes to avoid duplication with other initiatives? (5.1.1)

13. Can you comment on specific success factors that have assisted your JPIP-funded project/initiative in achieving its objectives? On specific constraints for achieving objectives? (4.6.1, 4.6.2)

  1. Have there been any unexpected outcomes, either positive or negative, from this JPIP-funded project/initiative? (4.7.1)

14. Do you have any other comments you would like to make about your JPIP-funded project/initiative or about the JPIP in general?

Thank-you for your participation.

JPIP objectives

The overall objective of the JPIP is to support the policy directions of the Department of Justice on issues related to family violence, missing and murdered Aboriginal women, access to justice, and the harmonization of private international law. The specific objectives are to:

Case study interview guide for partners of JPIP recipients

The Department of Justice Canada (the Department) is conducting an evaluation of the Justice Partnership and Innovation Program (JPIP or “the Program”). This program aims to help the Department achieve government priorities in the areas of access to justice, family violence, public legal education and information (PLEI), and missing and murdered Aboriginal women/violence against Aboriginal women and girls. The evaluation covers a four-year period (2012–13 to 2015–16) and focusses on the Program’s relevance and performance.

PRA Inc., an independent research company, has been contracted to support the evaluation.

Case studies of several funded projects/initiatives comprise a component of the evaluation. Case studies will involve a review of documents and several interviews with the project recipients as well as with partners that they have worked with on some of the activities that are funded through the JPIP. The interviews will be conducted by telephone and will be audio-recorded with your permission. Interview notes will not be shared outside of PRA Inc. and the Evaluation Division of the Department of Justice Canada. If you are unable to answer any questions, please let the interviewer know and they will skip to the next question.

1. Please briefly describe your organization and the work it does.

2. (Interviewer: name the organization receiving the JPIP funding and the funded initiative that you would like to discuss with the participant) Can you briefly describe how you are involved with this organization and this JPIP-funded initiative?

3. What does your organization contribute to this initiative? (Probes: cash funding, in-kind resources and what kind?)

  1. How important to this initiative is your partnership with the organization we are talking about today?

4. What are some of the activities related to this initiative that your organization is involved in? What needs is this project intended to meet?

5. Who are the target audiences/beneficiaries for this initiative? How do they benefit from the initiative? How satisfied are you with the extent this audience has been reached? What challenges are there, if any, in reaching this audience?

6. In general, to what extent would you say the activities conducted through this initiative have contributed to:

  1. Increased awareness and understanding of justice issues? Please give examples if possible.
  2. Increased access to justice? Please give examples if possible.

7. Can you comment on specific success factors that have assisted this initiative in achieving its objectives? On any specific constraints for achieving objectives?

8. Do you have any other comments you would like to make about this initiative or your partnership with the organization we are discussing today?

Thank-you for your participation.

Case study questions for international organizations

The Department of Justice Canada (the Department) is conducting an evaluation of the Justice Partnership and Innovation Program (JPIP). This program aims to help the Department achieve government priorities in the areas of access to justice, family violence, public legal education and information (PLEI), and missing and murdered Aboriginal women/violence against Aboriginal women and girls. Two areas that are funded through the JPIP are Canada’s assessed contributions to the Hague Conference on Private International Law and the International Institute for the Unification of Private Law.

PRA Inc., an independent research company, has been contracted to support the evaluation.

Case studies of several funded projects/initiatives comprise a component of the evaluation, including case studies of the two internationally funded organizations. Case studies will involve a review of documents and several interviews with recipients and other relevant stakeholders. The questions below can be responded to in writing or, if preferred, a telephone interview can be arranged with a PRA researcher. Interview notes will not be shared outside of PRA Inc. and the Evaluation Division of the Department of Justice Canada. Please indicate if you are unable to answer any of the questions.

1. Please briefly describe how Canada’s membership is important to your organization?

2. How does Canada’s financial contribution assist your organization in conducting its work and achieving its objectives?

3. Who do you see as the main beneficiaries of the work your organization conducts? How do they benefit from the organization’s work?

4. Can you comment on and give examples of how the work of your organization contributes to each of the following:

  1. Unification of rules?
  2. Increased awareness and understanding of justice issues?
  3. Increased access to justice?
  4. Promoting international cooperation with respect to access to justice?

5. Are you able to comment on how Canada’s membership in your organization facilitates the promotion of Canadian legal interests internationally? How does this contribute to Canada’s influence at the international level?

6. Are there any other comments you would like to make about Canada’s membership in the organization that have not already been covered?

Thank-you for your participation.

File review templates

File Review Template – Named Grants Recipients

The file review data will be entered into an excel spreadsheet.

Background

1. Name of organization/recipient; funding amount; file number

2. Follows JPIP report template

3. Year or years of funding (evaluation years only)

4. Information contained in recipient’s file (CHECK ALL THAT APPLY)

5. Years of summary reports available – indicate if based on only a partial year

Use the most recent summary report to complete remaining questions

6. Summary/annual report year that the remaining template questions are drawn from:

7. Type of organization

8. Grants or contribution funded

What happened?

9. Activities supported by JPIP funding – check identified activities below (CHECK ALL THAT APPLY). Also briefly summarize giving specific examples (e.g. type of conference, types of public legal education and information project, etc.).

10. Based on activities reported above what were the main areas of focus? (CHECK ALL THAT APPLY)

11. Which of the JPIP objectives do the funded activities appear to be related to (assessed by reviewer based on information in file) (CHECK ALL THAT APPLY)

12. Results achieved – check identified results achieved below (CHECK ALL THAT APPLY). Also briefly summarize verbally giving specific examples. Briefly summarize results achieved with examples.

13. Who accessed services?

14. Why they accessed the services? (DESCRIBE BRIEFLY)

Target population

15. Target population (CHECK ALL THAT APPLY)

Partnerships

16. Received funding from other sources

17. If yes, number of partners listed

18. If yes, did partners offer the below:

Cash:

Knowledge or expertise:

Materials or space:

Other (specify)

19. If yes, types of partners listed (CHECK ALL THAT APPLY)

Lessons learned

20. Anything they would do differently? (DESCRIBE BRIEFLY)

File Review Template – All Recipients Except Named Grants

The file review data will be entered into an excel spreadsheet.

Background

1. Name of organization/recipient, budget, fund, project title, file number

2. Follows JPIP report template

3. Year or years of funding (evaluation years only)

4. Information contained in recipient’s file (CHECK ALL THAT APPLY)

Questions 5 to 7 from application;

5. Program which they applied for (first page of application). Complete for multiple years if applied for different programs in different years and indicate the years.

6. Funding program priorities that relate to their project (Question 2.5 of application).

7. Briefly describe how their project is to address the priority (Question 2.5 of application)    

8. Years of summary reports available – indicate if based on only a partial year

Use the most recent summary report to complete remaining questions

9. Summary/annual report year that the remaining template questions are drawn from:

10. Type of organization

11. Grants or contribution funded

12. Type of project (CHECK ALL THAT APPLY)

Project objectives

13. Objectives given:

14. If objectives given above, based on objectives what were the main areas of focus? (CHECK ALL THAT APPLY)

15. Do the project objectives appear to align with the funding program priorities related to their project? (Compare answers to Q1 and Q14 to Q6)

16. Gives extent objectives achieved

17. Activities conducted (CHECK ALL THAT APPLY) (reviewer assess based on descriptions in reporting and application)

Target population

18. Target population (CHECK ALL THAT APPLY)

19. Reach - number of participants, numbers served, numbers of materials distributed, etc.

Results

20. Results achieved (CHECK ALL THAT APPLY)

a. Reports new skills, information, changed awareness

b. Reports that identified best practices or potential responses to priority/emerging justice

c. Reports that participants find that progress made developing best practices or responses to emerging justice issues

21. Reports unanticipated results

22. Reports positive response from target population on quality and utility of activities

23. Reports that target audience reported increased knowledge/understanding of the relevant justice issue

24. Reports on how project affected community's capacity to respond to he needs identified in the project

25. Reports on how project affected Official Minority Communities

Partnerships

26. Number of partners

27. If partners indicated, types of partners (CHECK ALL THAT APPLY)

28. What partners offered to project (CHECK ALL THAT APPLY)

29. Description of partnership arrangement (check both if they indicated short-term for some partners and ongoing for others)

30. Any potential partners suggested by their Department of Justice contact

31. If yes, did they follow up on suggestion

32. Would they do anything differently for partnerships?

Lessons learned

33. Reports on what worked well

34. Reports on what did not work so well

35. Reports on changes identified to build on lessons learned

Communication of results

36. Are project results communicated

37. If yes, how are results communicated (CHECK ALL THAT APPLY)

38. Next steps identified

Applicant survey

Applicant Survey Questionnaire

Welcome to the Justice Partnership and Innovation Program (JPIP or “the Program”) applicant survey. The survey should take approximately 15 minutes to complete. Participation in the survey is voluntary. You may leave the survey at any time and come back later to complete the questions. If you do leave the survey prior to completion, we ask that you wait approximately 15 minutes to re-enter it, in order to give the survey a chance to refresh. The survey will be online until INSERT DATE.

About your organization

Please tell us a bit about your organization

1. Which of the following best describes your organization?

2. Which of the following areas does your organization conduct work in? (Check all that apply)

3. How did you first learn about the JPIP program?

4. How familiar are you with the JPIP? (4.1.3)

5. In general, how aware of the JPIP do you believe other organizations that may potentially be eligible for JPIP funding are? (4.1.3)

6. Below lists some of the JPIP objectives. Please indicate for each the extent your organization needs or would benefit from support to help you in achieving these objectives. (4.1.3)

JPIP objectives

a. promote the equitable representation in the legal system of Métis and Non-Status Indians by encouraging them to pursue their studies in law

Extent your organization requires/could benefit from support to assist you in achieving the JPIP objectives

b. promote and encourage involvement in the identification of emerging trends, issues, and/or gaps and possible responses with respect to the justice system

Extent your organization requires/could benefit from support to assist you in achieving the JPIP objectives

c. build knowledge, awareness, and understanding among justice stakeholders on justice issues

Extent your organization requires/could benefit from support to assist you in achieving the JPIP objectives

d. build knowledge, awareness, and understanding among the public on justice issues

Extent your organization requires/could benefit from support to assist you in achieving the JPIP objectives

e. strengthen the justice system’s response to family violence

Extent your organization requires/could benefit from support to assist you in achieving the JPIP objectives

f. promote continued public awareness of family violence and public involvement in the response to family violence

Extent your organization requires/could benefit from support to assist you in achieving the JPIP objectives

g. reduce the vulnerability of young Aboriginal women to violence through the development of models, programs, tools, and resources to provide alternatives to risky behaviour and to empower Aboriginal women

Extent your organization requires/could benefit from support to assist you in achieving the JPIP objectives

7. To what extent would you say your organization needs or would benefit from support to help you in conducting the following activities for contributing towards JPIP objectives? (4.1.2)

JPIP funded activities

a. Offer workshops, conferences, symposiums

Extent to which your organization requires/could benefit from support to help you in conducting the following activities

b. Offer training sessions

Extent to which your organization requires/could benefit from support to help you in conducting the following activities

c. Pilot, demonstration, and research projects

Extent to which your organization requires/could benefit from support to help you in conducting the following activities

d. Public legal education and information projects

Extent to which your organization requires/could benefit from support to help you in conducting the following activities

e. Develop/offer resources, tools, and supports related to family violence?

Extent to which your organization requires/could benefit from support to help you in conducting the following activities

f. Culturally appropriate healing and support projects

Extent to which your organization requires/could benefit from support to help you in conducting the following activities

g. Other (please specify)

Extent to which your organization requires/could benefit from support to help you in conducting the following activities

8. In your opinion, how well are the JPIP objectives and activities aligned with the justice needs of the general public? Would you say they are… (1.1.1)

9. In your opinion, how well are the JPIP objectives and activities aligned with the justice needs of the legal community in general? Would you say they are… (1.1.1)

About your JPIP project applications

(For successful and unsuccessful applicants)

10. Since April 1, 2012 how many times have you or your organization applied to the JPIP? Please indicate the total number of times, even if the applications have been for the same project. (1.2.2)

11. If more than once, have these all been for the same project or for different projects? (1.2.2)

12. Of the total times you’ve applied to the JPIP for funding since April 1, 2012, how many times were you successful and your application approved by the JPIP? Were all, some, or no applications approved? (1.2.1)

(CONTINUE WITH ALL RESPONDENTS BUT ADD THE FOLLOWING FOR THOSE THAT SAID SOME WERE APPROVED AND SOME WERE NOT) Most of the remaining questions are regarding your applications that were approved. Please respond with these projects in mind. We ask you a few questions at the end about your applications that were not approved.

13. (Successful applicants) What was the area of focus of the project or projects for which received JPIP funding? (CHECK ALL THAT APPLY)

(Unsuccessful applicants) What was the area of focus of the project or projects for which you applied to the JPIP for funding? (CHECK ALL THAT APPLY) (1.2.1)

14. (Successful applicants) What type of activities did you take part in through JPIP funding? (CHECK ALL THAT APPLY) (1.2.2/42.4/4.2.5)

(Unsuccessful applicants) What type of activities did you plan to take part in through JPIP funding? (CHECK ALL THAT APPLY)

15. Who were the intended direct beneficiaries or target population for this project or projects? (CHECK ALL THAT APPLY) (1.2.1)

16. Besides your own organization, did you have any other partners for this project or projects? (4.1.1)

17. If yes, what were these partners to contribute to the project or projects? (CHECK ALL THAT APPLY) (4.1.1)

UNSUCCESSFUL APPLICANTS SKIP TO Q21; SUCCESSFUL AND THOSE WHO WERE BOTH SUCCESSFUL AND UNSUCCESSFUL CONTINUE

18. Overall, how satisfied are you with the partnerships you have or had for your JPIP project or projects? (4.1.7)

19. Could you please explain why you were [ENTER RESPONSE FROM PREVIOUS QUESTION] with the partnerships you had for your JPIP funded project or projects? (4.1.7)

20. (Successful applicants) Would this project have been able to proceed without the JPIP funding? (4.1.3)

[SKIP TO Q24]

21. (Unsuccessful applicants) Was this project able to proceed without the JPIP funding? (4.1.3)

22. Please explain:

  1. What changes were necessary in order for the project to proceed?
  2. How did this affect the success of the project (e.g., achievement of objectives)?

23. Were any of your funding sources for this project (e.g., cash or in-kind resources from partners or other funders) dependent on receiving the JPIP funding and unavailable due to not receiving JPIP funding?

24. Do you plan to re-apply to the JPIP in the future? (1.1.1/5.1.1)

25. If no, why not?

[IF THEY INDICATED NONE IN Q12, GO TO Q33]

Results of funded projects

[For successful applicants only]

The following section concerns the results of your JPIP-funded project(s) or activities.

26. Is/are your JPIP-funded project, or projects, completed or still ongoing?

27. To what extent was/were your completed JPIP project(s) able to achieve its/their objectives, or to what extent do you expect your ongoing JPIP project(s) to achieve its/their objectives? (4.2 to 4.5 depending on project)

28. [If it achieved at least a few objectives] Please describe the factors and best practices that have assisted your project with those objectives that have been achieved/that you expect to achieve. (4.6.2)

29. Please describe the challenges, if any, encountered and how these affected the achievement of the project objectives. (4.6.1)

30. To what extent would you say that your JPIP-funded project or projects contributed to each of the following JPIP objectives :

JPIP objectives

a. promote the equitable representation in the legal system of Métis and Non-Status Indians by encouraging them to pursue their studies in law (4.2.1)

Extent your JPIP funded project achieved JPIP objectives

b. promote and encourage involvement in the identification of emerging trends, issues, and/or gaps and possible responses with respect to the justice system (4.4.1)

Extent your JPIP funded project achieved JPIP objectives

c. build knowledge, awareness, and understanding among justice stakeholders on justice issues (4.2.3)

Extent your JPIP funded project achieved JPIP objectives

d. build knowledge, awareness, and understanding among the public on justice issues (4.2.2)

Extent your JPIP funded project achieved JPIP objectives

e. strengthen the justice system’s response to family violence (4.4.1)

Extent your JPIP funded project achieved JPIP objectives

f. promote continued public awareness of family violence and public involvement in the response to family violence (4.4.1)

Extent your JPIP funded project achieved JPIP objectives

g. reduce the vulnerability of young Aboriginal women to violence through the development of models, programs, tools, and resources to provide alternatives to risky behaviour and to empower Aboriginal women (4.4.1)

Extent your JPIP funded project achieved JPIP objectives

31. To what extent do you believe you were able to reach the intended target audience for this JPIP-funded project? (1.2.1)

32. Could you please describe how the beneficiaries/target population of your JPIP-funded activities benefited from the project activities? (4.2 to 4.5 depending on project)

Public legal education and information

33. Are you familiar with public legal education and information, or PLEI, materials and activities?

34. If yes, to what extent would you say the general public finds PLEI materials/activities useful in addressing their legal needs and concerns? (4.2.2)

35. Please explain why you believe the general public finds PLEI materials useful [INSERT RESPONSE TO PREVIOUS QUESTION]. (4.2.2)

Satisfaction with the JPIP application process

[For successful and unsuccessful applicants]

The next few questions ask about the JPIP application process.

36. How satisfied are you with each of the following with respect to the application process? (5.1.1)

Elements of the JPIP application process

a. the ease of the application process

Level of satisfaction…..

b. the clarity of the application process

Level of satisfaction…..

c. the type of information required in the application form

Level of satisfaction…..

d. the ease of interactions between applicants and JPIP representatives

Level of satisfaction…..

e. the timeliness of responses to your queries or communications with Justice Canada

Level of satisfaction…..

f. the appropriateness of selection criteria

Level of satisfaction…..

g. the timeliness of communication about the status of your application (approval decisions)

Level of satisfaction…..

h. Other aspects of JPIP application process (please specify)

Level of satisfaction…..

37. Do you have any suggestions for improving the application process? (5.1.1)

[IF ANSWERED NONE IN Q12 GO TO Q48]

38. [For successful applicants only — those who answered “All” or “Some” in Q12] How satisfied is your organization with each of the following for your successful applications? (5.1.1)

Elements of the JPIP funding, project management, and reporting process

a. the amounts awarded

Level of satisfaction…..

b. the time period of funding awarded

Level of satisfaction…..

c. the timeliness of payments

Level of satisfaction…..

d. the funding reporting requirements

Level of satisfaction…..

e. the assistance received from JPIP representatives for meeting reporting requirements

Level of satisfaction…..

f. Other aspects of JPIP funding, project management or reporting (please specify)

Level of satisfaction…..

39. Do you have any suggestions for improving the JPIP management and reporting process? (5.1.1)

Q40 TO Q0 are for respondents that said some applications were accepted and some were not in Q12; all others SKIP TO Q48

The remaining questions ask about the projects where your JPIP application was not approved.

40. What was the area of focus of the project or projects for which you applied to the JPIP for funding but were not approved? (CHECK ALL THAT APPLY) (1.2.1)

41. What type of activities did you plan to take part in through JPIP funding? (CHECK ALL THAT APPLY)

42. Who were the intended direct beneficiaries or target population for this project or projects? (CHECK ALL THAT APPLY) (1.2.1)

43. Besides your own organization, did you have any other partners for this project or projects? (4.1.1)

44. If yes, what were these partners to contribute to the project or projects? (CHECK ALL THAT APPLY) (4.1.1)

45. Was this project able to proceed without the JPIP funding? (4.1.3)

46. Please explain:

  1. What changes were necessary in order for the project to proceed?
  2. How did this affect the success of the project (e.g., achievement of objectives)?

47. Were any of your funding sources for this project (e.g., cash or in-kind resources from partners or other funders) dependent on receiving the JPIP funding and unavailable due to not receiving JPIP funding?

(All respondents)

48. Do you have any other comments you would like to share about the JPIP program?

Thank you very much for your participation in this survey