Evaluating the Success of a Homelessness Program

Outcome Evaluation for the Maryland Department of Housing and Community Development’s Homelessness Solutions Program

Introduction

Homelessness is a critical issue in Maryland, affecting individuals and families across the state. The Maryland Department of Housing and Community Development's Homelessness Solutions Program (HSP) is designed to address this problem by providing housing stability and financial education to individuals experiencing homelessness (Homeless Solutions Program, 2023). However, it is important to assess the effectiveness of the program and identify areas for improvement to ensure that it is meeting the needs of the community. This evaluation plan outlines the research design, data collection methods, and measures that will be used to assess the impact of the HSP on program participants.

Evaluation Objectives and Questions

The primary objective of the evaluation is to assess the effectiveness of the Homelessness Solutions Program in reducing homelessness in Maryland. The evaluation aims to answer the following research questions:

1. Are program participants able to secure and maintain stable housing?

2. Is there adequate and ongoing support and resources for program participants after they secure permanent housing?

3. Is the program culturally sensitive, competent, and accessible to an increasingly diverse population?

4. Are program participants consistently satisfied with the services and support they receive?

Additional questions that may be answered over the course of this evaluation include:

1. What are the specific outcomes of the program, such as the number of people who have been housed and the length of time they have remained housed?

2. What are the factors that contribute to the success or failure of the program?

3. Are there any unintended consequences or negative effects of the program?

Research Design

Quasi-experimental study using a pre-/post-test design with a comparison group will be used. This design was chosen because it allows for the comparison of program participants with a similar group of individuals who did not receive program services. A quasi-experimental study is a type of research design that aims to investigate the causal relationship between an intervention and an outcome (Bloomfield & Fisher, 2019). This type of study is similar to a randomized controlled trial (RCT) in that it involves comparing two or more groups, with one group receiving the intervention and the other(s) serving as a control group. However, unlike an RCT, participants in a quasi-experimental study are not randomly assigned to groups, but rather are assigned based on some other characteristic, such as their location or pre-existing condition. This can limit the researcher's ability to control for confounding variables, but can be useful in situations where random assignment is not feasible or ethical.

This study will compare the outcomes of the Homelessness Solutions Program to a group of individuals who did not receive the program's services. This will allow the evaluators to determine whether the program has had a significant impact on reducing homelessness in Maryland. For that reason, quasi-experimental design is appropriate. In this case, the comparison group consists of individuals experiencing homelessness who did not receive program services, and the program participants will be compared to this group at baseline and at three follow-up points. This design can help to control for some of the potential confounding factors that may affect outcomes for both groups, such as changes in economic conditions or policies that affect the availability of affordable housing.

Data Collection Method

The data will be collected through a combination of quantitative and qualitative methods, including surveys, interviews, and program records. Surveys will be used to collect quantitative data on participants' housing stability, financial literacy, and satisfaction with program services. Interviews will be used to collect qualitative data on participants' experiences with the program, including their perceptions of program effectiveness and cultural competence. In other words, the survey will be used to collect data on the program's effectiveness, while interviews will be used to gather more in-depth information about the experiences of program participants. Program records will be used to collect data on the number of participants served and the types of services provided. Data will be collected at baseline (prior to program enrollment) and at three follow-up points: 3 months, 6 months, and 12 months post-enrollment.

Sample Characteristics

The sample will consist of program participants and a comparison group (control group) of individuals experiencing homelessness who did not receive program services. The target number of participants will be 300 (150 in each group). Participants will be recruited from homeless shelters and community-based organizations serving individuals experiencing homelessness. Participants will be selected based on specific demographic characteristics such as age, gender, race, and socioeconomic status. Thus, the sample will be diverse in terms of race, ethnicity, age, gender, and disability status.

Measures

The evaluation will use the following measures to evaluate the program's effectiveness:

1. Housing stability: Participants' housing status (e.g., homeless, at risk of homelessness, housed) will be measured using a survey question adapted from the Comprehensive Assessment Tool and the Homeless System Evaluator Tool, both of the National Alliance to End Homelessness (2014, 2017).

2. Financial literacy: Participants' financial literacy will be measured using a survey adapted from the Financial Well-Being Scale (Consumer Financial Protection Bureau, 2023).

3. Satisfaction with program services: Participants' satisfaction with program services will be measured using a survey adapted from the Client Satisfaction Questionnaire (CSQ-8)

Data Collection Instruments

1. Survey: The survey will assess the program's impact on reducing homelessness in Maryland.

2. Interview Protocol: The interview protocol will be used to gather more in-depth information about the experiences of program participants and the factors that contribute to the program's success or failure.

Analysis Approach

The evaluators will use both quantitative and qualitative methods to analyze the data collected. The survey data will be analyzed using statistical methods to determine whether the program has had a significant impact on reducing homelessness in Maryland. The interview data will be analyzed using coding techniques to identify themes and patterns in the data.

More specifically, quantitative data collected through surveys will be analyzed using descriptive statistics (e.g., means, standard deviations) to assess changes in housing stability and financial literacy over time. Chi-square tests and t-tests will be used to compare differences between the program and comparison groups.

Qualitative data collected through interviews will be analyzed using a thematic analysis approach to identify recurring themes related to program effectiveness and cultural competence. Program records will be analyzed to identify patterns and trends in the types of services provided to participants. The results of the quantitative and qualitative analyses will be integrated to provide a comprehensive understanding of program outcomes and impact.

Alignment with Proposed Logic Model

The evaluation plan is aligned with the logic model in that it aims to measure the Short-Term Outcomes and Intermediate Outcomes identified in the logic model. Specifically, the evaluation plan will measure whether participation in the HSP leads to increased housing stability and improved financial literacy for program participants, which are Short-Term Outcomes. The evaluation will also provide information on the program's effectiveness in reducing homelessness, which is an Intermediate Outcome.

By measuring these outcomes, the evaluation will help to determine whether the program's Activities and Outputs are leading to the desired outcomes, as outlined in the logic model. Additionally, the evaluation will provide data on program effectiveness that can be used to make improvements to the program and ensure that it is achieving its intended goals.