Outcomes Evaluation Proposal: Maryland HSP Homelessness Program

Background and Rationale for the Evaluation

The proposed evaluation focuses on homelessness. The 2009 Homeless Emergency Assistance and Rapid Transition to Housing (HEARTH) Act defines homelessness as a situation where a family or individual lacks an adequate, regular, and fixed nighttime residence (Substance Abuse and Mental Health Services Administration [SAMHSA], 2023). This definition includes persons living in transitional housing, emergency shelters, abandoned buildings, the streets, and other places not meant for habitation. Individuals who stand to lose their nighttime residence within 14 days, or who are fleeing domestic violence and lack the resources to obtain alternative housing, are also considered homeless under the HEARTH Act of 2009 (SAMHSA, 2023).

Data from the National Alliance to End Homelessness shows that the number of homeless people in the United States has risen by approximately 6 percent since 2017 (SAMHSA, 2023). Contrary to expectations, the COVID-19 pandemic that peaked in 2020 had no significant effect on the U.S. homelessness rate (National Alliance to End Homelessness, 2023). According to a report by the Maryland Interagency Council on Homelessness (2022), this is due to increases in economic assistance programs and efforts to de-congregate shelters in line with social distancing requirements at the time. Nonetheless, a point-in-time count conducted in 2022 showed that over 582,000 people — nine in every 5,000 — were homeless across the United States (National Alliance to End Homelessness, 2023). Individual adults made up 72 percent of this population, while 28 percent lived on the streets and in other unfit environments with their children (National Alliance to End Homelessness, 2023).

In Maryland, point-in-time count data from 2020 showed that slightly over 6,300 people were homeless statewide, with Baltimore City accounting for the highest rates of homelessness at 35 percent (Maryland Interagency Council on Homelessness, 2022). The statewide annual count showed that Maryland had approximately 28,288 homeless people in 2020 (Maryland Interagency Council on Homelessness, 2022). Homelessness disproportionately affects special populations, including persons with disabilities, mentally ill individuals, veterans, unaccompanied youth, and victims of domestic violence (Maryland Interagency Council on Homelessness, 2022). Twenty-one percent of homeless people in the state, per the 2020 point-in-time count, were chronically homeless individuals who had been homeless for over a year and struggled with a physical disability, substance use disorder, or severe mental illness (Maryland Interagency Council on Homelessness, 2022). Homeless veterans made up 7.5 percent of the homeless population, while individuals experiencing domestic violence and unaccompanied homeless youth made up 5 percent and 4 percent, respectively (Maryland Interagency Council on Homelessness, 2022).

Policymakers across the U.S. need to institute ways of addressing homelessness because it increases the risk of mortality and reduces average life expectancy among affected populations (Maryland Interagency Council on Homelessness, 2022). Studies have shown that homeless people report higher rates of injury-related death and violence than the general population. Data from the Office of the Chief Medical Examiner (as cited in Maryland Interagency Council on Homelessness, 2022) shows that approximately 243 homeless people died in Maryland in 2020 — a death rate of 3,835 per 100,000 people, which is three times the death rate of the general Maryland population.

On average, homelessness reduces an individual's life expectancy by 30 years relative to the general population (Maryland Interagency Council on Homelessness, 2022). Homelessness is also associated with a higher risk of drug overdose and other complications related to substance use. For instance, 65 percent of deaths reported among homeless individuals in 2020 were due to drug overdose and related complications, with opioid overdoses accounting for 88 percent of those deaths (Maryland Interagency Council on Homelessness, 2022).

A significant body of literature exists on homelessness interventions. To identify studies for review, the evaluator searched three databases — Cochrane, PubMed, and Science Direct — for studies on homelessness prevention, case management, income assistance, and permanent supportive housing programs. Inclusion criteria required randomized controlled trials (RCTs) or systematic reviews conducted in high-income countries, published in peer-reviewed journals before 2020, and using housing stability as the primary outcome of interest.

Generally, the studies found homelessness prevention programs and interventions effective in preventing and reducing homelessness. Dwyer et al. (2023) found that unconditional cash transfers, coupled with coaching supports and workshops on plan-making, goal-setting, and self-affirmation, improved housing stability within three months better than cash transfers alone. Similarly, Phillips and Sullivan (2022) found that financial assistance coupled with case management increased housing stability and reduced evictions more than monetary assistance alone. The primary weakness of these evaluations is their narrow focus on income interventions, disregarding supportive housing programs. Aubry et al. (2020) address this gap in their systematic review by studying the combined effect of housing support and income assistance interventions. They found that, compared to usual care, housing support programs increase long-term housing stability at a rate of 1.13, while income assistance coupled with case management improved the number of days a homeless individual is housed by eight days relative to usual care.

While these evaluations provide valuable insights into the effectiveness of homeless assistance interventions, they have a limited scope, focusing on one or two interventions at a time. The proposed evaluation seeks to assess the HSP, which incorporates a diverse range of interventions. Moreover, the reviewed studies do not focus specifically on Maryland, and it would be valuable to determine whether findings from other settings can be replicated in the state.

The program being evaluated is the Homelessness Solutions Program (HSP), administered by the Maryland Department of Housing and Community Development (DHCD) as a statewide response to homelessness (HSP Policy Guide, 2022). The program provides funding and technical support to Local Homeless Coalitions (LHCs) and Continuums of Care (COCs), coordinating resources and funding for local service providers serving the homeless population. To qualify for a grant under the HSP, an applicant must be a local government or non-profit entity offering a client-centered approach to homeless services and compliant with anti-discriminatory policies (HSP Policy Guide, 2022). HSP grantees and sub-grantees must also implement formal grievance procedures, demonstrate sound financial management, and operate a sound Homeless Management Information System (HMIS).

The HSP pursues the following goals and objectives as stated in the HSP Policy Guide (2022) and the DHCD Consolidated Plan (2020):

Goal 1: Reduce the number of families and individuals at risk of homelessness by providing homeless prevention assistance and shelter diversion.

Objective 1.1: At least 50 percent of unsheltered homeless families and individuals reached through outreach programs will exit to safe emergency shelters by the end of 2024.

Objective 1.2: One hundred percent of households and individuals who become homeless as a result of losing their permanent residence will successfully exit to safe shelters by the end of 2024.

Goal 2: Reduce the duration an individual or household remains homeless by supporting them in accessing permanent housing.

Program Description and Logic Model

Objective 2.1: At least 80 percent of previously homeless individuals and households held in emergency shelters and interim housing will transition effectively to permanent housing by the end of 2024.

Objective 2.2: At least 90 percent of households and individuals who become homeless due to losing their permanent residence will successfully exit to alternative permanent housing by the end of 2024.

Goal 3: Reduce the number of individuals or households falling back into homelessness by connecting them to community-based resources, natural support networks, and income supports.

Objective 3.1: At least 10 percent of adult HSP beneficiaries report increased income by the end of 2024 due to services and support received.

Objective 3.2: At least 25 percent of HSP beneficiaries report improved mental health by the end of 2024.

While there has been a notable decline in homelessness rates in Maryland over the past several years, homelessness continues to disproportionately affect special populations, including veterans, people with physical disabilities, mentally ill individuals, and unaccompanied youth (Maryland Interagency Council on Homelessness, 2020). The 2020 point-in-time count revealed that veterans made up close to 8 percent of the homeless population, while chronically homeless persons with some disabling condition made up 21 percent (Maryland Interagency Council on Homelessness, 2020). This data accentuates the need to evaluate the HSP's effectiveness in reducing or preventing homelessness among these special populations. According to Smith and Ory (2014), program evaluation serves two purposes in public health: to assess the effectiveness of a program or policy and its interventions, and to identify areas of weakness that could be used to improve overall program quality.

The proposed evaluation would help policymakers determine how effective the HSP is for its target population, including the relevance of its interventions to selected special populations. This would increase efficiency by ensuring that program implementers identify interventions that work and commit more resources to making homelessness non-recurring, brief, and rare for all Marylanders (HSP Policy Guide, 2022). The evaluation would also help improve program quality by indicating the quality of deployed human resources and areas requiring capacity-building (Smith & Ory, 2014). The findings will guide program managers running similar programs on best practices, thereby minimizing the risk of errors and resource wastage.

As Mabhala et al. (2017) point out, "Homelessness is a more complex social and public health phenomenon than the absence of a place to live" (p. 2). Homelessness results from a combination of structural forces — lack of access to mental health services, racial disparities, poverty, and lack of affordable housing — and social factors such as poor mental health, family breakdown, domestic violence, and addiction (Mago et al., 2013). Studies have shown that a majority of homeless people have experienced traumatic events and social disadvantage from childhood, including dysfunctional families, neglect, emotional and sexual abuse, physical abuse, lack of psychological support, and disrupted schooling (Mabhala et al., 2017). All these factors serve as possible drivers of homelessness among Marylanders and may not be fully addressed by providing housing alone. The HSP's effectiveness will depend on how relevant its interventions are to the specific drivers of homelessness in the state. While the HSP may be effective in linking homeless people with interim or permanent housing, failure to adequately address the social context of homelessness may result in high rates of people falling back into homelessness upon exiting the program.

Structural factors such as ineffective housing assistance programs, budget cuts affecting federal housing, and rent increases may also increase homelessness rates (Mabhala et al., 2017). These factors are often dependent on the political environment and may affect the operations of the HSP, reducing its overall effectiveness. It would therefore be prudent to assess the possible effect of these factors during the evaluation to obtain a holistic view of the program's sufficiency and relevance.

Generally, the HSP targets the homeless population in Maryland. The program's support for homeless individuals and households is divided into three areas: Outreach, Emergency Shelter, and Housing Stabilization (HSP Quick Reference Guide, 2022). Outreach services target unsheltered homeless people to link them with critical health services, housing, and emergency shelters. Emergency Shelter services target unsheltered individuals living in places unfit for habitation, linking them with overnight shelters. The HSP supports these populations indirectly by providing essential services during their stay in shelters and contributing towards shelters' operating costs.

Housing Stabilization focuses on preventing people from falling back into homelessness by supporting them in moving into permanent housing (HSP Quick Reference Guide, 2022). This segment is divided into three support areas targeting different populations. The first, Rapid Rehousing, targets households and individuals who lack adequate, regular, and fixed nighttime residences. The second, Homelessness Prevention, targets individuals with annual incomes below 30 percent of the area median income who risk becoming homeless but lack the support and resources to prevent it. Finally, Permanent Housing Case Management targets individuals in permanent housing who risk falling into homelessness for diverse reasons.

The HSP was instituted in 2017 following a decision by Maryland's agencies to develop a central entity to coordinate activities of LHCs and COCs across the state (Maryland Interagency Council on Homelessness, 2022). The program's role was to oversee homelessness programs statewide and ensure alignment with federal and national best practices. The formative years 2017 to 2019 served as the planning phase, characterized by setting goals and objectives, defining project scope, and developing the DHCD Consolidated Strategic Plan 2020–2024. The program is currently in the implementation phase, guided by the framework, milestones, and targets laid out in the consolidated strategic plan.

The evaluation requires financial resources of at least $669,300.80. These cover staffing costs and benefits, staff training costs, and financing of equipment and supplies. Equipment and supplies will require $43,000 to cover emergency clothing and food vouchers, rental assistance vouchers, housing supplies and furniture vouchers, and participant incentives. Staff salaries and benefits amount to $620,760.80, and an additional $5,600 is required to finance training sessions for case managers engaged in the evaluation.

The evaluation will engage two full-time and eleven part-time staff members. A project director will serve as the primary contact person, responsible for recruiting case managers, training consultants, and the program analyst, and for overseeing the evaluation process. A project assistant will assist in accounting and record-keeping, data entry, and coordinating training sessions. The ten part-time case managers will conduct assessments and offer case management services, while the program analyst will collect and analyze data. The DHCD has agreed to provide 2,200 square feet of office space. Technological equipment required includes three desktop computers, one laptop for the project director, ten tablets for case managers, an office printer, and a projector.

The planned evaluation is expected to produce the following outputs: (i) outreach programs successfully identify and link homeless individuals and households with HSP services; (ii) available assessments accurately identify homeless individuals' income, housing needs, and financial literacy; (iii) short-term financial assistance successfully increases access to safe and affordable housing; (iv) long-term supportive and housing services increase access to permanent housing; and (v) counseling and financial education services effectively improve participants' mental health and financial literacy.

Short-term outcomes include: increased numbers of individuals and households receiving HSP services; participants reporting increased housing stability; participants reporting improved financial literacy; participants reporting increased access to safe and affordable housing; and participants reporting increased psychological well-being.

Intermediate outcomes include: reduced homelessness rates among program participants; increased community-wide knowledge and understanding of effective homelessness prevention interventions; and increased collaboration among local service and housing providers.

Long-term outcomes include: sustainable housing stability among HSP participants; declining inequality and poverty rates among participants; and improved overall quality of life among participants.

The logic model below summarizes the inputs, activities, outputs, and outcomes of the HSP evaluation.

Inputs: Funding allocated to the HSP; staff and volunteers trained to work with homeless participants; partnerships with stakeholders; access to affordable housing; access to financial education resources; evaluation specialist to assist with program planning and data collection.

Activities: Outreach and engagement to identify and connect individuals and families experiencing homelessness; assessment of participants' needs, including housing, income, and financial literacy; delivery of short-term financial assistance (rental and utility payments) to prevent or end homelessness; delivery of long-term housing and supportive services to help participants maintain housing stability; delivery of financial education and counseling services to build financial literacy.

Outputs: Outreach programs successfully identify and link individuals and households with HSP services; assessments accurately capture homeless individuals' income, housing needs, and financial literacy; short-term financial assistance increases access to safe and affordable housing; long-term supportive services increase access to permanent housing; counseling and financial services improve mental health and financial literacy.

Short-term outcomes: Increased number of individuals and families receiving HSP services; increased housing stability; improved financial literacy; increased access to affordable and safe housing; improved psychological well-being.

Evaluation Focus, Questions, and Design

Intermediate outcomes: Reduced rates of homelessness for program participants; increased knowledge and understanding of effective homelessness solutions among stakeholders and community members; increased collaboration among housing and service providers.

Long-term outcomes: Sustainable housing stability and financial well-being for HSP participants; reduced rates of poverty and inequality among participants; improved overall quality of life among participants.

The proposed evaluation seeks to answer the following questions:

(a) Compared to non-participants, are HSP participants more able to secure and maintain stable housing?

(b) How does the financial well-being of HSP participants compare with non-participants at program start, end, and at six-month follow-up?

(c) Is the program accessible and relevant to the diverse needs of homeless people?

(d) Are program participants consistently satisfied with the support and services they receive from the HSP?

Stakeholders in this evaluation are categorized as primary or secondary. Primary stakeholders — those with a direct stake in the evaluation — include HSP participants, the DHCD board, and staff working directly in the HSP. Secondary stakeholders — those indirectly affected by the evaluation findings — include HSP grantees and sub-grantees (LHCs and COCs), funders, and local communities.

The evaluation findings will be crucial to multiple parties. The DHCD and its staff will gain insights into the overall quality and effectiveness of the HSP's homelessness solutions and possible areas of improvement. The HSP management could use the findings to analyze staff performance and identify areas requiring capacity enhancement (Harris, 2016). Grantees and sub-grantees will learn how relevant the services and supports they offer are to participants. The federal government, as the primary funder of the HSP, will be able to assess whether the program is worthwhile by measuring its effectiveness in reducing homelessness in Maryland.

Harris (2016) notes that engaging stakeholders in program evaluation goes a long way toward creating buy-in and minimizing resistance to change. Accordingly, stakeholders will be engaged in both developing the evaluation proposal and its implementation. In developing the proposal, program staff, volunteers, grantees, and sub-grantees will help design data collection tools and determine what to include in surveys and interviews. During implementation, representatives of COCs, LHCs, and program staff will be engaged in interpreting findings and drawing conclusions. A stakeholder forum will also be organized to disseminate project findings, with stakeholders expected to share results across their respective agencies and organizations.

The evaluation uses a case study with a control group design. According to Schoch (2020), a case could be a nation, community, organization, small group, or individual; here, HSP participants serve as the cases. The case study design is appropriate because it provides a platform to study the HSP's interventions and assess their effectiveness within a real-life context (Schoch, 2020). Schoch (2020) advocates for this design when the researcher's aim is not to generalize findings but to obtain an in-depth understanding of a program's inner workings. The proposed evaluation not only measures the HSP's effectiveness but also seeks an in-depth understanding of its interventions and their relevance to the needs of homeless individuals through surveys, interviews, document reviews, and observations. The inclusion of a control group is crucial for assessing whether an association actually exists between HSP participation and reduced risk of homelessness.

One of the greatest strengths of the case study design is that it enables a researcher to focus on a specific case within defined temporal and geographic parameters, providing a deep understanding of its inner workings (Schoch, 2020). The design also allows for the collection of different types of data using a combination of methods — surveys, observations, documents, and interviews. Its greatest weakness is that findings are not generalizable to other cases (Schoch, 2020). The findings of this evaluation may not apply to homelessness prevention programs in other states. Another major weakness is the high risk of selection bias when selecting controls (Tenny et al., 2023). Controls must be similar to cases in many respects, with an equal underlying risk of experiencing the outcome; failure to ensure this similarity produces biased and misleading findings.

The evaluation will compare outcomes of HSP participants (cases) with non-participants (controls) — specifically, homeless individuals who frequent the participating shelters but are not registered to receive HSP services. Comparison will be based on financial literacy and housing stability scores at baseline, program exit, and six months post-exit.