Program evaluation in public agencies

Program Evaluation

Evaluation #1'

Operation and Performance of Virginia's Social Services System

http://jlarc.state.va.us/Reports/Rpt323.pdf

Which agency is being audited?

The Virginia Social Services System provides a variety of services for the general public. The evaluation involved site visits of the 120 local departments of Virginia's Social Services. In addition, phone interviews were conducted with 27 local departments (JLARC, 2005). In addition to these procedures, the JLARC also performed an analysis of the department's financial records, human resources, caseload, and internal performance data (JLARC, 2005). Social Services in the state of Virginia are structured so that they are administered locally, with oversight on the state level (JLARC, 2005). This allows the agency to meet the needs of its local population and tailor its services to meet those needs, yet remain compliant with state requirements. Only a handful of states use this state-supervised/locally administered approach to social services (JLARC, 2005).

Virginia provides over 50 social service programs to its citizens (JLARC, 2005). These agencies are administered on a state level by either a Commissioner of Chief Operating Officer. The Commissioner has an Advisor who reports directly to him on issues that arise within the system. Under the Commissioner's direction are the departments responsible for licensing, Child Support Enforcement, Quality Management, Audit Services, Family Services, Benefit Programs, Child Care and Development, and Community Programs (JLARC, 2005). The Chief Operating Officer is responsible for overseeing Human Resources Management, Information Systems, Appeals and Fair Hearings, Public Affairs, Finance, Budget, General Services, Legislative Affairs, Training, and Inter-Departmental Regulation (JLARC, 2005). These two directors use local agency administrators as consultants to help design the overall programs. Local agencies provide important feedback to those on the upper levels of the programs.

It is important to understand the structure of the department in order to evaluate the JLARC audit. Underneath the general headings listed above, there are many other individual services and service providers. These include the Food Stamp Program, Medicaid Eligibility Determination, and Temporary Assistance for Needy Families (TANF), Virginia Initiative for Employment, Not Welfare (VIEW), Child Care Development, Adoption and Foster Care, Child Protective Services, and Adult Service Programs (JLARC, 2005). Aside from understanding the vast array of services that the agency provides and the high degree of variation in clientele, one must also consider local population demographics in order to develop an understanding of the agency's performance.

Who performed the audit?

The Joint Legislative Audit and Review Commission of the Virginia General Assembly (JLARC) conduct performance audits on other agencies within the Virginia government. The purpose of these audits is to make certain that the agency is fulfilling its intended purpose. It makes suggestions as to how the agency could improve its services and operate more efficiently.

The purpose of the Joint Legislative Audit and Review Commission is to serve as an external auditor for the agency or service. The commission consists of nine members of the House of Delegates, of which at least five serve on the House Appropriations Committee (JLARC, 2008a). It performs audits at the request of the Virginia General Assembly. The JLARC is a branch of Virginia government, receiving funds for audits from state funds set aside for that purpose (JLARC, 2008a). Audits result from an enactment of a Joint House Resolution. In this case, the audit was performed because of House Joint Resolution 193 (JLARC, 2005).

This system is designed to eliminate bias in the evaluation process. The purpose of these audits is to provide an outside opinion to the agency as to how to operate more efficiently, resulting in considerable savings (JLARC, 2008a). The audits are also for the purpose of keeping the legislature informed so that their policies are effective and reflect the current needs of the agencies and the public (JLARC, 2008). They are also intended to assure compliance with current operating standards and policies (JLARC, 2008a).

Analysis of the JLARC Audit

This audit was a result of a 2004 House Joint Resolution 193 (JLARC, 2005). It is directed by the Virginia General Assembly as a part of a continual audit process. This process consists of regular audits designed to make certain that state agencies are serving the public good. The audit process evaluates strengths and weaknesses in the agency. The purpose of the evaluation is focused on making recommendations to the agency for improving existing services.

Audits are performed at the request of the Virginia General Assembly on a regular basis. Some audit cycles are long and some are relatively short. The last complete audit performed on Social Services in Virginia was April of 1981 (JLARC, 2008b). However, audits have been conducted periodically for various aspects of the Social Services Department. Although, the report did not explicitly state the reason for the report, it is probable that the Department was due for an audit.

The audit and the Department of Social Services have many different stakeholders on many levels. Stakeholders can be divided into different categories including policymakers and decision makers, program sponsors, evaluation sponsors, target participants, program managers, program staff, contextual stakeholders, and the research community. The study was mandated and funded by the policymakers and decision makers. This group of stakeholders has a high degree of risk in the accuracy and usefulness of the report. They will take actions and make decisions based on the outcome of the findings. This group of stakeholders also serves as the program sponsor and the evaluation sponsor.

The target participants are the patrons who use the programs and services that the department provides. They did not participate directly in the study, but they will be the ones upon which the study will have the greatest impact. Although the study was conducted by an outside agency, it resembles an internal audit, rather than an external study. The impressions and feelings of the participants were not considered in the study. The target participants will receive the greatest benefit of any of the other stakeholders in the study.

Program managers and program staff will be directly affected by the results of the study. They will be ultimately responsible to enacting the changes mandated by the study. They will be the ones that have to turn theory into actions in their daily work practices and routines. A majority of the information upon which the study was based derived from program managers and staff. Their input had a direct impact on the results of the study. They will feel the direct impact of the study more than any other stakeholder group.

The Department of Social Services does have contextual groups that will be affected by the outcome of the study. However, it is difficult to define this group of independent organizations that will be affected by the study outcome. One example may be a Women's Shelter whose clients must depend on services provided by departmental agencies. These stakeholders will receive the same impact as the target participants. The study will have an impact on the research community, as it takes the results of the study and the methods as an example, for good or bad, to design future studies in other agencies and departments.

The program evaluation is both formative and summative. In the Preface of the report, Director, Philip Leone outlines strengths, weaknesses, as well as recommendations for improvement. The study found several resource issues that were found to be lacking. It found that state supervision of local departments were inadequate. The research cites weak communication between departments. These elements of the study are summative and make critical judgments about the performance of the department.

However, the study goes beyond its summative elements and provides a formative evaluation to address these issues. The study recommends comprehensive departmental changes. It provides an exact plan for addressing the weaknesses found during the audit. It suggests a top-down approach to resolving many of the issues. However, the formative evaluation also recognizes the complexity of the issue due to differences in structure among various service providers.

This study was extremely long and provided portions that could be classified as different types of evaluations. Chapter 4: Local Administrative Funding serves as a needs assessment for the different localities that are under the jurisdiction of various agencies. This chapter serves as a justification for the need to address issues within the department. It addresses population demographics that affect programs in various areas of the state.

The remainder of the evaluation was weighted towards an assessment of program theory and an assessment of program process. Much of the theory was addressed through an examination of the history of the department. The largest portion of the study focused on assessment of program process, including the processes within various individual service agencies. This study focused on internal issues within the department; therefore, it did not include an impact assessment or an efficiency assessment. The study suggests that the improvements recommended by the JLARC will have a significant positive impact on the target population, but they provide no empirical evidence to support their opinion.

Research Questions and Methodology

One of the key weaknesses of the study is that it did not implicitly state the intended research questions. However, each chapter does include a paragraph explaining the intended purpose and outcome of the study. This study was not a traditional academic study and did not follow a standard academic format. It followed specific departmental government guidelines and stated the purpose, as directed by the General Assembly. The key research objectives focused on discovering the strengths and weaknesses of ever program or process that was examined in the study.

The evaluation utilized a combination of qualitative and quantitative research strategies. It employed the use of observational site visits to all of the local service offices in the state. It also employed interview techniques with various staff members from several offices. These qualitative techniques allowed the evaluators to place the empirical data into proper perspective.

In addition to the qualitative techniques used, the evaluators also used quantitative methods derived from a review of departmental records. Statistical methods helped the evaluators to gain credible evidence for their arguments and to provide a means to convey the study results. However, quantitative methods only answer questions regarding what happened. In order to fulfill the intended purpose of the study, more in-depth knowledge was needed in order to make recommendations. The researchers used qualitative information obtained through interviews and observation to place the quantitative information in perspective. It used quantitative information to substantiate the analyses of qualitative data. This allowed the researchers to provide a full-bodied report that provided an excellent base upon which to make recommendations for reform.

This is not an outcomes/impact assessment in terms of the agencies and their impact on the clientele that they service. However, it is an outcome/impact assessment in terms of how well the programs operate under the current program guidelines. The focus of the study is on results, rather than causality. It established key strengths and weaknesses in many areas, but did not attempt to determine causality. Many external and internal variables contributed to the findings. Therefore, any allusion to causality would only constitute conjecture. Rather than focusing on causality, the study focused on the steps that needed to be taken to fix weaknesses and enhance strengths.

Major findings and recommendations are presented in Chapter 8: Comprehensive Changes Needed. Each service provider within the department was evaluated on its own merit. The evaluation found inconsistencies between the various agencies in terms of weaknesses and strengths. Several findings were consistent throughout the various agencies. A lack of training was wound to be lacking in many service sectors. The evaluators indicate that a needs-based training program needs to be instituted in many of the service agencies. In addition to training, business processes were in need of an overhaul in many agencies. However, these process transformations will be dependent upon the characteristics of the various service agencies. Generalized training and business processes were found to be inadequate to meet the specific needs of services and their clients.

Conclusion

This evaluation illustrates many of the key terms and concepts found in the Rossi text. For instance, the study contains no hidden agendas. The evaluator has no stake in the outcome or the changes that result from the study. The evaluator is not a stakeholder in the study, other than to provide a quality report. Many of the programs within the study have been in existence for many years. However, at the time study, it was recognized that reforms were needed. It was not known what the reforms would be. This study was the first step in the reform process. It outlined the project and a possible timetable for the reforms.

The scope of this study was broad and the topic was comprehensive. This may represent one of the major shortcomings of the study. The service agencies represented in the study are varied and have different needs. They use different processes and serve different clientele. This study may have treated them too homogeneously to provide an accurate representation of the various factors within the agencies that could affect the outcome.

This study provides an excellent overview of the agencies and their positive and negative aspects. However, its perspective is general and does not address the specific needs of the agencies on a local level. It does provide direction as to future research that specifically addresses the needs of the various services. This research provides a stepping-stone to specific agency research that needs to be undertaken. The evaluation provides a general overview of the problems within the Virginia Social Services Department. However, further evaluation of the various components will be needed in order to produce changes that are effective and reflect the needs of the agencies within the department.

Evaluation #2

Special Report: State Spending on Standards of Quality (SOQ) Costs, FY 2007

http://jlarc.state.va.us/Meetings/December07/soq07.pdf

What is the agency being audited?

As mandated by Article VIII of the Constitution of Virginia, the state must determine Standards of Quality (SOQ) for the school divisions (Rotz, FY2007). Under the direction of the General Assembly, the Joint Legislative Audit and Review Commission of the Virginia Assembly (JLARC) must conduct an audit of the standards that apply to the elementary and secondary schools throughout the state. The Board of Education must meet the required SOQ, including staff and resources needed to accomplish the task. The purpose of this audit is to determine how much the State needs to provide to subsidize the needed programs for Virginia youth.

Who performed the audit?

The Joint Legislative Audit and Review Commission of the Virginia Assembly (JLARC) is an agency within the Virginia state government that is responsible for providing research reports to the General Assembly on various state agencies. The JLARC is independent of the agencies themselves and addresses the parameters of the topic as set forth by the General Assembly. Funds for this report are provided by the Virginia government, according to the mandates of the Constitution of Virginia (Rotz, FY2007).

Analysis of the JLARC Audit

This study was funded according to a state Constitutional provision that a required audit be conducted quarterly and then those results compiled annually for the previous fiscal year (Rotz, FY2007). This annual report provides an estimation of the funds that will be needed for SOQ over the next school year. The audit is an annual requirement to assess the needs of the school systems and the amount of funds needed to make them compliant.

Like many evaluation reports, a number of stakeholders are involved on a number of levels. The first level of stakeholders is the policymakers who mandate the study. The decision makers at the state level represent one level of stakeholder. They will use this information in the formation of funding policies for the coming fiscal year. This group of stakeholders serves a dual purpose, as they are also the sponsor of the study and the resultant funding program.

The ultimate target participants in the study are the students. They are the ones that will receive the benefit of better education because of the funding provided through this study. The target participants are far removed from the conduct of the study, yet they are the ones with the most at risk. The students have an indirect impact on the outcome of the study. Each pupil in the school system counts as a unit. These units are used to determine various costs associated with each student that the school system serves.

The school system has many different levels of program managers and staff. The Board of Education determines policy that will affect all of the school systems within the state. This study will have an impact on these stakeholders, as it will help them to determine how to allocate funds for the coming school year. At the school district level, the study will have an impact on programs managers and staff, as they will have to use these funds wisely so that the students receive the greatest benefit. They will have to make changes within the district according to the amount of funding that they receive.

Those program managers and staff on the school level will be the ones to feel the impact of the funding provided. They will be the ones that have to manage the impact of budget cuts or overages that result from the study. They will have to find ways to fill staff vacancies created by budget drops, or how to provide textbooks when the money is not available. The individual staff will wonder if they will be the ones to be cut, should their school lose funding.

Few contextual groups will be affected by the findings and the resulting funding decisions of the study. The impact of the study will be felt by those directly involved in the Virginia educational system. This study will have little impact on the research community. It does not provide new methodology or a new topic of interest. It is conducted according to strict guidelines that are set forth by the state. It is important to those directly affected by it, but it will not contribute significantly to those outside of the school system.

The study is formative in nature, as the results will be the sole determining factor in SOQ funding decisions. The study does not address program effectiveness nor provide a critical evaluation of the key points of the program. It is strictly financial in nature. The focus of the study is on providing accurate information upon which to base funding issues for the school system.

The SOQ finding study does not fit neatly into any one of the categories outlined in the Rossi text. It does not address program theory, program processes, program impact, or the efficiency of the program. One could call it a needs assessment, only because it determines the needs of the school system. However, this is not a traditional needs assessment where the target participant determines if a need exists. This needs assessment is based on prescribed mathematical criteria rather than examination of a social program.

Research Questions and Methodology

This study is based on a single research question. How much money does a particular school system need to meet SOQ criteria? The study determines the number of staff and the amount of resources needed by a school, using the number of pupils as a key determining factor in the analysis. Staff members do not only include teaching staff, but also include principals, teachers, kindergarten and special education teachers (Rotz, FY2007).