Veteran Transition Program Outcome Evaluation Plan

Introduction and Program Overview

This paper presents an outcome evaluation plan whose aim is to assess the effectiveness of a new Veteran Transition Program administered by a community-based family services agency (referred to here as NCCFS) in order to secure additional funding. The Veteran Transition Program serves two crucial aims: (i) to help veterans transition successfully into civilian life and (ii) to improve veterans' quality of life. The paper outlines the specific strategies that the agency will employ under the CDC's six-step program evaluation framework to ensure that the evaluation exercise is a success.

Step 1: Engaging Stakeholders

Stakeholders are organizations or people affected or served by the program, those involved in program operations, and the intended users of the evaluation results. Stakeholders served by the program include local community members, veterans, and their families. The intended users of the evaluation results include funding agencies, law-enforcement representatives, program managers, and state agencies serving veterans. Stakeholders involved in program activities include program staff, faith-based organizations, state education agencies, local community members, and housing authorities.

From these groups, key stakeholders are those needed to increase the credibility of the evaluation and those responsible for making implementation, funding, or advocacy decisions (CDC, 2012). The final list of key stakeholders thus includes state and local agencies serving veterans, faith-based organizations, schools, state education agencies, program staff, housing authorities, law-enforcement representatives, the media, and funding agencies. Representatives from each stakeholder group will be engaged through focus group meetings held every two months and periodic emails. The agency will use the focus groups to engage stakeholders on why they are interested in the program, what elements of the program are most relevant to them, their expectations regarding the program, what critical questions to include in the evaluation, how they will use the evaluation results, and what specific resources they will contribute to the evaluation (Ray & Miller, 2017).

All stakeholders will be allowed to provide input on evaluation questions and the design of the data collection instrument. However, since the primary aim of the evaluation is to secure funding, stakeholders responsible for making funding decisions will be responsible for choosing the most appropriate evaluation questions, while those involved in the program — such as program staff — will be responsible for disseminating results.

Step 2: Program Description and Evidence Base

The agency's mission is to develop partnerships with the public and private sectors to enhance the quality of life for vulnerable children and their families (NCCFS, 2020). The agency recognizes the challenges that the contemporary family faces. With the help of its social workers, the agency uses workshops, seminars, and self-help groups to educate families on effective parenting, maintaining a healthy marriage, grief management, balancing work and family, and related topics (NCCFS, 2020). Social workers offer non-clinical counseling to troubled individuals and families and network with community-based counselors for long-term clinical counseling whenever necessary (NCCFS, 2020).

The agency's newest program is the Veteran Transition Program, which recognizes that veterans face significant challenges as they transition into civilian life. The program is in the implementation stage and aims to facilitate transition for veterans and their families by linking them to needed housing services, providing job training, and offering behavioral counseling services (NCCFS, 2020). To serve more veterans, the agency initiated a three-year fundraising campaign to seek funding for the program. This evaluation was commissioned to assess the program's effectiveness and establish a justification for continued funding.

Studies have shown that veterans face a host of challenges as they transition into civilian life, ranging from family reintegration issues to financial difficulties, mental health issues, and unemployment (Ahern et al., 2015). A study by Ahern et al. (2015), for instance, showed that most veterans face psychiatric difficulties, challenges securing employment, and a lack of access to affordable housing. That study found, however, that veterans with supportive social and family networks and those with access to job training services coped better than those without such resources (Ahern et al., 2015). These findings mirror those of a study by Sayer et al. (2015) involving 1,292 veterans, which found that 54 percent of veterans experience family reintegration problems, posttraumatic stress disorder, and significant distress. That study supported the use of behavioral counseling to assist at-risk veterans facing mental health issues — veterans with access to psychological support were found to cope better than those without it (Sayer et al., 2015).

Empirical literature thus supports the approach adopted by the NCCFS Veteran Transition Program. The program offers psychological support through counseling to help veterans address their mental health issues. In recognition of the housing challenge, the program offers transitional housing to at-risk veterans and their families to minimize the risk of homelessness. Further, the agency increases opportunities for employment by providing job training and equipping veterans with the skills needed to maintain employment.

Step 3: Focusing the Evaluation Design

The evaluation seeks to assess the effectiveness of the Veteran Transition Program in realizing its objectives. The central evaluation question can be summarized as follows:

How effective is the Veteran Transition Program in facilitating veterans' transition into civilian life and enhancing their quality of life?

The evaluation focuses on measuring how effectively the program has realized its intended outcomes. An outcome evaluation is the most appropriate approach because funding agencies are the primary users of the evaluation results. Such agencies are interested in assessing the feasibility of the program's activities, which is best captured through outcome evaluation. The reason for conducting the evaluation is to assess the program's effectiveness and thereby convince potential funders that the program is worth investing in.

Step 4: Gathering Credible Evidence

The evaluation will use quantitative methods — specifically, self-administered surveys — to collect data from randomly selected beneficiaries of the program. A questionnaire (included as an appendix to this paper) will be the primary data collection instrument. Quantitative techniques are preferred for this evaluation because they provide numerical data that is easily quantifiable and allows for straightforward comparison of baseline and current levels (W.K. Kellogg Foundation, 2004). Because funding agencies are the primary users of the evaluation findings, comparing participants' baseline and current scores on selected quality-of-life indicators allows those agencies to assess whether significant change has occurred as a result of participation in program activities.

Veterans who have benefited from the program over the eight months of its existence will be the primary source of evaluation data. The evaluation will engage 150 veterans selected randomly through systematic sampling. Program staff will organize beneficiaries' files in alphabetical order by family name, and every fifth file will be selected and included in the sample.

Veterans remain in the program for three years, after which they are expected to have attained enough stability to sustain themselves and their families. According to the original evaluation plan, data collection and analysis are to take a maximum of four months, beginning November 2020. During the first month, staff will focus on mailing survey instruments to respondents and following up to ensure that participants respond and return their questionnaires within four weeks. Respondents will first be notified by email that they have been selected to participate and that they may decline if they are unwilling to take part. Participants who choose to participate will receive their questionnaires in the first week of November and will be asked to complete and return them within four weeks at their convenience. Data organization and analysis are scheduled to begin in the second week of December, with the final report expected by the end of February 2021. Findings will be disseminated to participants during March 2021.

Self-administered surveys require little expertise to administer (Posavac, 2015). Program staff can effectively carry out the task of mailing questionnaires and making follow-ups. However, the agency will hire an external party to conduct data analysis and prepare the final report. The proposed evaluation budget is presented in Table 1 below. Data collection, analysis, and report preparation together account for over 70 percent of the proposed budget, at a total cost of $4,015.

Table 1: Proposed Evaluation Budget

Planning: $1,000
Preparation: $500
Data Collection: $500
Analysis: $3,000
Report Preparation and Dissemination of Findings: $500
Total Cost: $5,500

Conclusion

The agency has taken steps to ensure that the standards of utility, feasibility, accuracy, and propriety are met across all six steps of the evaluation. All key stakeholders are engaged from the outset and have the opportunity to provide their views on the data collection instrument and the overall findings. The data collection and analysis approaches selected are cost-effective for both the agency and participants, in line with the feasibility requirement. The agency has also taken steps to ensure that the data collected are accurate and reflect the program's actual effectiveness without bias. In summary, with the strategies adopted, stakeholders can be confident that the evaluation results present a fair and credible view of the Veteran Transition Program's effectiveness.

References

Ahern, J., Worthen, M., Masters, J., Lippman, S. A., Ozer, E. J., & Moos, R. (2015). The challenges of Afghanistan and Iraq veterans' transition from military to civilian life and approaches to reconnection. PLOS ONE, 10(7), 1–13.

CDC. (2012). Step 6: Ensure use of evaluation findings and share lessons learned. Centers for Disease Control and Prevention. Retrieved from https://www.cdc.gov/eval/guide/step6/index.htm

Kellogg Foundation. (2004). Logic model development guide. Battle Creek, MI: Author.

NCCFS. (2020). Resources for veterans. North Carolina Center for Child and Family Services. Retrieved from

Posavac, E. J. (2015). Program evaluation: Methods and case studies. New York, NY: Routledge.

Ray, K. N., & Miller, E. (2017). Strengthening stakeholder engaged research and research on stakeholder engagement. Journal of Comparative Effectiveness Research, 6(4), 375–389.

Sayer, N. A., Orazem, R., Noorbaloochi, S., Gravely, A., Frazier, P., & Carlson, K. (2015). Iraq and Afghanistan war veterans with reintegration problems: Differences by Veterans Affairs healthcare user status. Administration and Policy in Mental Health, 42(1), 493–503.

Stufflebeam, D. L., & Zhang, J. (2017). The CIPP evaluation model: How to evaluate for improvement and accountability. New York, NY: Guilford Press.

Walker, J. T., & Maddan, S. (2013). Understanding statistics for the social sciences, criminal justice, and criminology. Burlington, MA: Jones & Bartlett.

Appendix: Data Collection Instrument

North Carolina Center for Child and Family Services — Veteran Transition Program Evaluation Questionnaire

This questionnaire is for an evaluation that seeks to assess the effectiveness of the Veterans Transition Program, for which you are a beneficiary. All responses are entirely confidential and will be used solely for purposes of the current evaluation. The questionnaire will take between 10 and 15 minutes to complete.

1. Have you been a beneficiary of the Veteran Transition Program's transitional housing initiative, which provides temporary housing to veterans and their families at risk of homelessness?

Yes ________ No ________

2. If yes, to what extent has the program helped to minimize your risk of homelessness?

Very great extent _____ Great extent _____ Moderate extent _____ None at all _____

3. Did you know how to file for disability/educational entitlements before joining the program?

Yes __________ No __________

4. Can you now file for disability/educational entitlements using skills learned in the program?

Yes __________ No __________

5. Have you successfully filed for disability and educational entitlements for veterans using skills learned in the program?

Yes ________ No ________

6. Have you managed to secure employment as a result of the job skills learned in the program?

Yes _________ No _________

7. To what extent have the job skills gained through the program helped you in your current job?

Very great extent _____ Great extent _____ Moderate extent _____ None at all _____

8. How would you rate your mental well-being:

i) Before joining the program:
Very Good ________ Good ________ Poor ________ Very Poor ________

ii) Now:
Very Good ________ Good ________ Poor ________ Very Poor ________

9. To what extent has the program helped to improve the quality of life for you and your family, as measured by the following indicators (from "very great extent" to "none at all")?

i) Availability of basic necessities:
Very great extent ________ Great extent ________ Moderate extent ________ None at all ________

ii) Access to education for children:
Very great extent ________ Great extent ________

iii) Access to educational supports in school for children:
Very great extent ________ Great extent ________ Moderate extent ________ None at all ________

For broader context on challenges facing returning service members, see the Wikipedia overview of veteran transition to civilian life and the U.S. Department of Veterans Affairs resources on veteran homelessness.