Program Evaluation Integrate Data Collection Methods Into

Program Evaluation

Integrate data collection methods into the program evaluation plan.

The data collection method is seeking to integrate qualitative and quantitative research together. It is developing a program that is effective in helping to support smoking cessation efforts. Qualitative research is used to provide background on the study and proven smoking cessation initiatives. For instance, this portion of the research revealed that any effective program will integrate therapy, support groups and medication with each other. The basic idea is to address the chemical and psychological states a person will feel when they are quitting. However, changes in technology are leading to more people using online support. This gives them the flexibility to attend meetings and discuss their problems with counselors 24 hours a day. These shifts are indicating how there are variety of options and tools for having a successful smoking cessation efforts. (Yin, 2009)

The quantitative approach is taking a different perspective when it comes to quitting. It is interested in the way these trends compare with a sample of smokers. The main idea is to determine which programs are the most popular based upon age, gender and racial demographics. In this case, a total of 97 respondents were selected. They ranged from 29 to 79 and the majority of respondents (89.1%) were white males. This helps in determining which programs are the most effective, the rates of successes and corroborate the findings from the qualitative method. (Yin, 2009)

The way the data was collected is part of the evaluation by objectively looking at what is happening from a longer term perspective. This supports those smoking cessation programs that are the most effective and which one produces the best results. Once this takes place, is the point the research is used as a foundation to talk about these programs and the lasting impacts they are having on smokers. (Yin, 2009)

Evaluate the strengths and weaknesses of data collection sources for your program.

The strengths of the program are its ability to maintain objectivity and corroborate the findings with each other. This improves accuracy by confirming the underlying trends with each other. These conclusions will provide insights about how long-term cessation efforts can be established and which programs are the most effective. At the same time, it looks at what different demographics of people prefer and how they can be customized to meet their needs. This improves the success rates by knowing the individual and what is most important to them. Then, they can create a solution which can be customized to guarantee long-term success. (Yin, 2009)

The biggest weaknesses are with the sample that was collected. In this case, the majority of respondents were white males. This does not take into account other factors which could impact someone differently (i.e. racial, gender and economic). These variables can influence the kinds of solutions that are selected and which ones are the most effective. To make matters worse, the sample was very small. This is troubling, as it could skew the results and lead to unsubstantiated conclusions. If this were to occur, researchers could state these programs are effective. Yet, they failed to fully understand which ones could be customized to meet the needs of the individual. This is when the odds increase of them having long-term success with smoking cessation efforts. (Yin, 2009)

The strengths and weaknesses are showing how the research project can provide measurable results and new insights. However, the biggest drawback is the limitations in the size and quality of the sample. In this case, there was no diversity chosen to answer other factors which could affect someone differently. This can make some programs ineffective by failing to understand other characteristics. As a result, the study provides a good foundation for discussing these challenges and possible solutions. This can be used as an initial foundation for other research in determining how to establish smoking cessation efforts. (Yin, 2009)

Determine threats to the data collection process for your plan.

The biggest threat to the data collection process is the sample and subjectivity of researchers. The sample is problematic, as only a specific segment of society was chosen (i.e. white males). This ignores racial, gender and economic demographics. The combination of these factors is contributing to a sense of bias by ignoring areas that are important. This opens the data collection process up to accusations that the study did not provide any kind of measurable results. (Yin, 2009)

At the same time, researchers could be accused of showing favoritism and subjectivity. This is because they decided to use a sample that was exclusively focused on white males. If they wanted to expand their findings, the sample size and selection process would have focused on diversity. The fact that researchers ignored this is illustrating how they were bias and wanted to work with demographics they were most comfortable with. This ignores the reality that they needed to maintain objectivity in the process. Instead, researchers create a program which appeared to achieve some legitimate scientific interest. Yet, the sample only focused on one segment of society and was uninterested on expanding the study. In this aspect, one could argue that the researchers were bias from understanding and setting up the study in this way. (Yin, 2009)

Over the course of time, some could challenge the data collection process and the validity of the research. If this were to happen, there is a probability many peers will discount the research and look at conducting a broader study. This is the point they reduce the quality of the information and the findings. (Yin, 2009)

Recommend appropriate data analysis procedures for your plan.

The methodology will focus on using the mixed method approach. This is when both qualitative and quantitative sources are used to understand the problem. In this case, the qualitative research will identify key trends using previous studies and data were collected. This is used to establish different avenues of smoking cessation and the impact it is having on them. (Yin, 2009)

Quantitative research is used to comprehend the impact of smoking cessation programs on individuals in the 29 to 79-year-old demographic. Their responses will be compared with each other and categorized. Once this takes place, is the point the findings from the study will be compared with other information. (Yin, 2009)

During the process, comparative analysis will be utilized. This is when the information will be compared with each other to identify which programs are most effective. Anything that does not corroborate with each other is placed into a separate category as an anomaly. It will be researched further and listed as a possible limitation in the study. (Yin, 2009)

If this kind of approach is utilized, it will improve accuracy and validity. This is when the findings of the research will be more in depth. In the future, similar projects will take place in order to reach out to colleagues and demonstrate the importance of expanding upon these areas. Over the course of time, this could lead to customized programs that will look at the individual and try to understand how it is impacting them. (Yin, 2009)

A good example of this can be seen with insights from the National Institute of Health which said, "Mixed methods researchers use and often make explicit diverse philosophical positions. These positions often are referred to as dialectal stances that bridge post positivist and social constructivist worldviews, pragmatic perspectives, and transformative perspectives. For example, researchers who hold different philosophical positions may find mixed methods research to be challenging because of the tensions created by their different beliefs. However, mixed methods research also represents an opportunity to transform these tensions into new knowledge through a dialectical discovery. A pragmatic perspective draws on employing "what works," using diverse approaches, giving primacy to the importance of the research problem and question, and valuing both objective and subjective knowledge. A transformative perspective suggests an orienting framework for a mixed methods study based on creating a more just and democratic society that permeates the entire research process, from the problem to the conclusions, and the use of results. Optimally, all studies draw upon one or more theoretical frameworks from the social, behavioral, or biological sciences to inform all phases of the study. Mixed methods studies provide opportunities for the integration of a variety of theoretical perspectives (e.g., ecological theories, complexity theory, stress theory, critical theories, or others)." ("Best Practices," 2014) This is illustrating how the quality of the research will improve by embracing these concepts. Once this occurs, are when the study will be more focused and objective. This reduces arguments of bias by showing how the facts were corroborated with each other.

Distinguish between data of practical significance and data of statistical significance in your plan.

The practical significance is calculating the difference between the estimate and the actual outcome of the program. This helps in objectively analyzing, if it was successful and the way it answered key questions. A statistical significance will vary based upon the sample size. It is used to determine if it was large enough and looked at a number of other factors in the process. The basic idea is to enhance diversity when studying the problem. (Kirk, 1996)

According to Kirk (1996), these ideas are used to have a better comprehension of key trends with them saying, "Statistical significance is essentially scientific credibility. In many academic disciplines, research is considered statistically significant only if the results of the study would occur by mere chance less than five times out of 100. For many, this number has become a kind of gold standard, often determining which papers are published, where researchers find work, etc. As a consumer of numbers, this is important to you for two reasons. Practical significance is an arbitrary limit whereby an observed difference is of some practical use in the real world. Let's say you add Ingredient X to your car's oil that is supposed to improve fuel efficiency. You conduct a careful controlled experiment, measuring fuel efficiency before and after introducing the additive. You find that the difference before and after is statistically significantly better, and conclude that the additive does indeed improve fuel efficiency. However, Ingredient X costs $1,000 a bottle - effectively negating any fuel efficiency savings. You don't really drive your car that much so conclude that the difference is not practically significant." (Kirk, 1996) These insights are showing how both will be utilized to understand what is taking place.

In this case, practical significance is looking at the long-term effects of the programs. This is taking place by watching the cessation rates and the impacts it is having on stakeholders. While the statistical significance, is providing specific raw data that is collected. It is interested in supporting or refuting the hypothesis. The differences are the practical approach wants to understand key trends and use the data as a part of achieving these objectives. This means that the information they are providing is more reliable by taking a broader perspective. The data from the statistical approach, is examining a specific aspect of the problem. It is very limited and can only answer select aspects of the problem. (Kirk, 1996)

Determine the essential elements needed in your program evaluation report.

The most critical elements that will be needed in the program report include: beginning statistics, current figures and the scope of key changes. These areas will allow researchers to compare the information objectively. During this process, they will not be provided with information on the subject. Instead, they will only see an anonymous label. It will ensure that they are overly influenced in one way or another by specific respondents. (Yin, 2009)

In the case of beginning statistics, this information is necessary to understand what kind of smoker the person was and what kinds of programs they were involved with. This will improve accuracy by ensuing that each subject is carefully studied. The beginning figures will provide an initial foundation in reaching these objectives. (Yin, 2009)

The current figures will provide raw information about what is happening in real time with respondents. In this case, they will show how each cessation program is working and it if is having the desired effects. The data is useful in understand the transformations that are taking place and the long-term impact. (Yin, 2009)

The scope of the changes is when researchers are looking at what is happening. They want to understand which factors are influencing the subjects. This is when they can use these insights to establish key conditions and preferences. In the future, these views will be used to understand how this influencing the attitudes of those who are in these programs and those they prefer the most. (Yin, 2009)

According to a study conducted by the University of Washington (2014) this is critical to the long-term success of the research with them saying, "Program evaluation is the systematic assessment of the processes and/or outcomes of a program with the intent of furthering its development and improvement. As such, it is a collaborative process in which evaluators work closely with program staff to craft and implement an evaluation design that is responsive to the needs of the program. For example, during program implementation, evaluators can provide formative evaluation findings so that program staff can make immediate, data-based decisions about program implementation and delivery. In addition, evaluators can, towards the end of a program or upon its completion, provide cumulative and summative evaluation findings, often required by funding agencies and used to make decisions about program continuation or expansion. Evaluators use many of the same qualitative and quantitative methodologies used by researchers in other fields. Indeed, program evaluations are as rigorous and systematic in collecting data as traditional social research. That being said, the primary purpose of evaluation is to provide timely and constructive information for decision-making about particular programs, not to advance more wide-ranging knowledge or theory. Accordingly, evaluation is typically more client-focused than traditional research, in that evaluators work closely with program staff to create and carry-out an evaluation plan that attend to the particular needs of their program. The primary difference between evaluation and assessment lies in the focus of examination. Whereas evaluation serves to facilitate a program's development, implementation, and improvement by examining its processes and/or outcomes; the purpose of an assessment is to determine individuals or group's performances by measuring their skill level on a variable of interest (e.g., reading comprehension, math or social skills, to mention just a few). In line with this distinction -- and quite common in evaluating educational programs where the intended outcome is often some specified level of academic achievement -- assessment data may be used in determining program impact and success." ("What is Program Evaluation," 2014) These insights are showing how the program is effective understanding the problem. This is because it is looking at what is happening from a variety of perspectives.