Role in Evaluating the Impact

¶ … role in evaluating the impact of the study by providing information directly from the individual. This helps the researcher understand the processes being observed results and assesses the changes and modulations in people's perceptions. We will also use the data from Method A to design the questions for Method B, asking ourselves what particular items need to be accentuated and explained.

Open ended questions will be used which, admittedly, will have less structured protocols, but in combination with interviews (which are completely open-ended), will provide a more robust view of the individual's issues regarding linguistic development.

Guiding Questions -- The reason for the qualitative questionnaire is because Method A consists of individual research interviews. Method B. will allow us to hone and define the results so that we have a greater area of commonality with which to base our discussion and analysis.

Implementation -- The questionnaires will be distributed to research participants within two weeks of the completion of their individual interview. The questionnaire may be handed to them, mailed with SASE, or done online. There is no reason to expect collusion or inaccuracy of data since the process is very opinion oriented.

Data Analysis -- Because the questions will be standardized and ranked in order to research importance, we will be able to sort the responses into appropriate categories based on student, tutor, etc. In addition, the data will allow us to at least sort questions into Positive/Negative, and Neutral categories and compare the cross-tabs with differing groups.

Reporting Strategy -- The completed data will be graphed and cross-tabulated based on preassigned groupings (which will be defined within the discussion section). Once the data is collected and analyzed, the report will cover all three methods for analysis, each drawing upon the other for reference and veracity of argument. The point of using the three methods explained is to find better ways to triangulate the data, and ensure that we are reaching the level of robustness within the process that we expect. (For more information on this method, see: Questionnaire Design, 2010).

Method C -- Self Ratings

Purpose -- In Method A we interview stakeholders, in Method B. we use that data to refine our questions and gather commonality. In Method C. we further refine the individual participant's views with self-assessment (student) or assessment (tutor, etc.). These assessments will provide a greater understanding of the place the individual believes they are academically regarding their English language acumen and the place they may actual be test wise.

Process -- Based on the use of ESL criteria, participants will be asked to rate certain statements on a 1-5 scale, one being the lowest level, 5 being the highest. Questions will cover topics such as vocabulary development, cognition, fluency of speaking, fluency in writing; improvement in skills, skill levels, etc.

Guiding Questions -- Due to the subject matter, it is important that we understand more clearly how the individual stakeholders actually perceive the efficacy of ESL programs. Based on the interview data (unstructured opinions), and the qualitative questionnaire (structured opinions), we will add this tool of how they see themselves and others within the ELS paradigm.

Implementation -- The questionnaires will be distributed to research participants within two weeks of the completion of their individual interview, at the same time as the qualitative questionnaire. The questionnaire may be handed to them, mailed with SASE, or done online. There is no reason to expect collusion or inaccuracy of data since the process is very opinion oriented. This phase will be a simple numerical ranking and should take no more than 10 minutes to complete.

Data Analysis -- The data, while qualitative in nature, can be quantitatively analyzed because of its numeric structure. We will analyze this data and present in reference to both Method A and B, and within the context of the discussion and analysis.

Reporting Strategy -- The completed data will be graphed and cross-tabulated, graphed, and included in the final reporting structure as a part of the discussion. (For additional information on the use of this method, see: Self Test on Learning English, 2007).

Transferable Components - The use of testing and research really provides two cogent purposes: 1) to uncover data and prove or disprove the researcher's hypothesis and, 2) to allow the participants and stakeholders to undertake a learning experience in which, once the research is over, they are able to leave with a better understanding of the topic, and their skill level within that area or organization. Under this rubric, the stakeholders should finish the study with a better understanding of:

Knowledge -- because of the three methods used, stakeholders will understand in more detail expected knowledge ranges in ESL.

Skills -- Specific questions in the research will focus on particular types of skills necessary in order to be successful with ESL.

Attitudes -- Once the report is published, stakeholders will understand various attitudes and opinions from others, not just those having trouble with ESL, but tutors, administrators, etc.

Contributors to Transfer -- The basic transfer of learning will be in three phases: what the individual stakeholder gleans from the process; the moderator's comments and guidance in the interview process, and then finally, a discussion based on the results that may occur between researcher and stakeholders, stakeholder to stakeholder, or other informal channels.

Strategies and Techniques -- Within the structure of the assignment, the moderator will use data mining techniques to follow up on opinions in Method A, have the subject elaborate, and pull out as much information as possible. For other stakeholders, including participants and tutors, the types of questions and the validity of the report will be critical to ensure a robust discussion of the material.

Monitoring Process -- Within this research process the monitoring will be continuous and based upon the initial data collected from Method A. Method A provides the basic structure, methods B. And C. follow up on that structure, and are monitored for accuracy at each step, ensuring that the data sets outlined match. In addition, the correlation of the data as a process will be an important step in this process since it will link to the entire validity of the study. Additionally, data verification on all sets of materials will be established by coded information on each sheet, matching these up to individual respondents.

Program Structure -- English as a Second Language is an expected program in many American universities and colleges. Students are welcomed from other countries, and understandably are not initially expected to perform at the level of a native speaker. However, in order to participate in the academic environment of upper-level education, it is necessary to have a good command of spoken and written English.

Purpose -- The purpose of the study is to assess, in a variety of ways, the most effective means of ESL learning using a variety of stakeholders. What are the expectations of learning? Are these expectations reasonable? Do the expectations match with the level of resources given to the issue? Is success the goal of the program?