Planning an Evaluation, There Are Several Steps

¶ … planning an evaluation, there are several steps an evaluator must take. One of the final steps in the planning process is to present a written proposal. What should the written proposal include, and why is it important to get the details of the evaluation in writing?

The steps that must be taken in the planning process for research or evaluation include the written proposal, which is comprised of specific sections. The first section of the research proposal is the introduction or background, which briefly provides information about the issue to be examined. The next section is titled the 'Research Objectives & Aims' which states the goals of the research that is being proposed. Following the objectives and aims of research is the 'Research Purpose' section, which sets out the reasons that the research is being conducted. The research proposal should contain a section entitled 'Significance of the Study', which relates the contributions that the study will make to the knowledge base existing and any new knowledge that might be provided. Next stated in the research proposal is the section entitled 'Justification or Rationale" which provides an explanation of why the research is important. The section that follows is the theoretical framework and conceptual framework. The theoretical framework examines existing or self-formulated theories relating to the objectives of the researcher while the conceptual framework is a set of variables, which a researcher will operationalize in the study in order to achieve the set objectives. The section entitled "Scope of the Research" provides "for the boundary of the research in terms of depth of investigation, content, and sample size, geographical, time frame and theoretical coverage. The section entitled "Literature Review" is such that examines literature in the area of research "with the objective of revealing contributions, weaknesses and gaps." (Makerere University, 2012, p.3 ) Finally, the proposal should contain a section summarizing the literature reviewed and its findings and state recommendations.

(2) According to the text, after evaluators and stakeholders have agreed on criteria that would indicate successful implementation and outcome, evaluators face the task of developing methods to measure those criteria. Measures should have both reliability and validity. Briefly describe the difference between reliability and validity and explain why they are important concepts when performing an evaluation.

Reliability and Validity

Reliability is defined as "the extent to which a questionnaire, test, observation or any measurement procedure produces the same results on repeated trials." (Miller, nd, p.1) Reliability is "the stability or consistency of scores over time or across raters." (Miller, nd, p.1) Reliability is reported to relate to scores rather than to people. Miller reports three aspects of reliability including those of: (1) equivalence; (2) stability; and (3) internal consistency. (nd, p.1) Equivalence is reported to refer to the "…amount of agreement between two or more instruments that are administered at nearly the same point in time. Equivalence is measured through a parallel forms procedure in which one administers alternative forms of the same measure to either the same group or different group of respondents. This administration of the various forms occurs at the same time or following some time delay. The higher the degree of correlation between the two forms, the more equivalent they are" (Miller, nd, p.5) Stability is such that occurs when the "same or similar scores re obtained with repeated testing with the same group of respondents." (Miller, nd, p.5) Internal consistency is related to the extent to "which items on the test or instrument are measuring the same thing." (Miller, nd, p.5)

Validity

Validity is reported to include the "entire experimental concept and establishes whether the results obtained meet all of the requirements of the scientific research method." (Experiment Resources, 2012, p.1) There is a requirement for randomization of the sample groups and appropriate care and diligence shown in the allocation of controls." (Experiment Resources, 2012, p.1 ) Internal validity is reported to dictate how an experimental design is structured and encompasses all of the steps of the scientific research method." (Experiment Resources, 2012, p.1) It is reported that control groups and randomization will reduce external validity problems but no method can be 100% successful.

(3) Briefly describe the difference between closed-ended and open-ended questions when conducting a qualitative interview.

Open-ended questions are reported to have the advantages of allowing respondents "to include more information, including feelings, attitudes and understanding of the subject. This allows researchers to better access the respondents' true feelings on an issue." (Answers, 2012) Open-ended questions cut down on two types of response error, respondents are not likely to forget the answers they have to choose from if they are given the chance to respond freely, and open-ended questions simply do not allow respondents to disregard reading the questions and just fill in the survey with all the same answers." (Answers, 2012) Open-ended questions enable the interviewer to obtain additional information from the respondent "such as demographic information and surveys using open ended questions are such that can be used in secondary analysis by other researchers than can surveys that do not provide contextual information about the survey population." (Answers, 2012) Open-ended questions allow respondents to use their own words making it different to compare meanings of the responses. Closed-ended questions can be analyzed more easily, they are more specific and more likely to communicate similar meanings than open-ended questions. In large-scale surveys, closed-ended questions are less time consuming for the interviewer and results in a survey method that is not as costly as those with open-ended questions.

(4) According to the text, an Institutional Review Board (IRB) is not necessarily required for a program evaluation. Please describe a situation where it would be appropriate to convene an IRB prior to conducting a program evaluation.

It is reported that the IRB must make provision of "substantive and meaningful review of research on a continuing basis at the interval established by the IRB at the prior review. IRB review must be performed by the convened IRB unless the research meets the criteria for expedited review." (Ohio State University, 2012, p.1) Research that is approved by the IRB must satisfy the requirements stated as follows:

(1) Risks to participants are minimized (but not necessarily eliminated) by using procedures that are consistent with sound research design and that do not unnecessarily expose participants to risks, whenever appropriate, risks to participants are minimized by using procedures already being performed for diagnostic or treatment purposes;

(2) Risks to participants are reasonable in relation to anticipated benefits (if any) and the importance of the knowledge that may reasonably be expected to result from the research. (Note: The IRB will consider risks and benefits that may result from the research, not risks and benefits of treatments or other activities the subject would undergo even if he or she were not participating in the research.);

(3) Selection of participants is equitable, taking into account the purposes of the research and the setting in which the research will be conducted.