Mixed-Methods Analysis of College Teacher Effectiveness Perceptions

Introduction and Study Overview

The objective of this paper is to evaluate the methodology and data analysis of a research study titled "Perception of Students Characteristics of Effective College Teachers: Validity Study of a Teaching Evaluation Form Using a Mixed-Methods Analysis" (Onwuegbuzie, Witcher, Collins, et al., 2007). The study uses mixed methods to evaluate student perceptions of the effectiveness of their college teachers. It also examines the teaching evaluation form (TEF) as an indicator of accountability and as a decision-making tool in areas such as promotion, merit pay, and the selection of adjunct and full-time faculty members. To arrive at its findings, the study employs a specific methodology for data collection, which is discussed in the sections that follow.

Onwuegbuzie et al. (2007) employ mixed methods as their data collection technique. "Mixed methods research is the collection, analysis, and interpretation of both quantitative and qualitative data within a study" (Chow, Quine, & Li, 2010, p. 491). Mixed methods as an approach to data collection and analysis is increasingly common in research. The major objective for using mixed methods is to address the shortcomings of either qualitative or quantitative methods alone, while integrating the advantages of both in a single research inquiry. Fisher and Stenner (2011) argue that the division between qualitative and quantitative methods is unnecessary, since the history of research inquiry has largely involved a combination of the two approaches.

Methodology: Mixed-Methods Approach

Onwuegbuzie et al. (2007) therefore use mixed methods for data collection, data analysis, and the presentation of research findings, integrating the advantages of both qualitative and quantitative techniques. Qualitative research is a method of inquiry that employs naturalistic, ethnographic, field, or participant-observer approaches for data collection. It is premised on the belief that collecting data from natural settings allows a researcher to become immersed in the phenomena under study. The general approach in qualitative data collection is through semi-structured interviews that allow in-depth exploration of phenomena. Quantitative research, by contrast, employs empirical investigation, hypothesis testing, mathematical modeling, and statistical tools to provide unbiased findings and arrive at research conclusions.

Based on the complementary strengths of both approaches, Onwuegbuzie et al. (2007) combine them for their research inquiry. However, a critical evaluation of the study reveals that the authors make use of a combination of surveys and questionnaires to collect data, and the findings are largely presented in statistical form. While the authors claim to use mixed methods, a closer reading suggests they employ quantitative technique more extensively than qualitative technique in the data analysis.

Despite this shortcoming, the mixed-methods framework is nonetheless appropriate for the study's goals. The researchers use quantitative methods to sample the opinions of 912 college students — a scale that would be difficult to achieve with qualitative technique alone. They also use qualitative methods to capture personal experiences and perceptions that are difficult to quantify. By converting qualitative data into quantitative form, the researchers are able to enhance the overall understanding of students' perceptions toward their instructors (University course materials, 2011).

Data analysis is the process of organizing, inspecting, cleaning, transforming, and modeling raw data to extract useful information. Onwuegbuzie et al. (2007) employ several methods to collect and analyze their data. To examine student perceptions using the teaching evaluation form, they collected data through both quantitative and qualitative methods. Participants were 912 college students attending a public university in a mid-southern state.

"All 912 participants contributed to both the qualitative and quantitative phases of the study, and the qualitative phase preceded the quantitative phases; the mixed-methods sampling design used was a sequential design using identical samples... The sample size represented 10.66% of the student body at the university where the study took place" (Onwuegbuzie et al., 2007, p. 123).

Students were drawn from six colleges enrolled in 68 different degree programs. The authors used criterion sampling to select the sample population. The majority of the sample was female (74.3%). With respect to ethnicity, respondents comprised Caucasian-American (85.4%), African-American (11.0%), Asian-American (1.0%), Hispanic (0.4%), Native American (0.9%), and other (1.3%). Ages ranged from 18 to 58 years (M = 23.00, SD = 6.26). Among the sample, 77.04% were undergraduate students (Onwuegbuzie et al., 2007, p. 123). The sample population also included Caucasian-American (76%), African-American (16%), Asian-American (1%), Hispanic (0.9%), and Native American (0.86%) participants, with the ethnicity of the remaining 2.7% unknown.

Data Collection: Instruments and Sample

The researchers administered questionnaires as the primary instrument for data collection. The questionnaires specified the purpose of the study and provided instructions for completion. A consent form was distributed along with the questionnaires, in which participants were asked to identify and rank the features and characteristics of effective college instructors. Students were also asked to provide a description and definition of each characteristic, with lower rankings denoting the most effective traits. The instrument additionally captured personal information such as age, year of study, major, GPA, and number of credit hours taken.

Part of the research instrument was also the teaching evaluation form (TEF), which consisted of two parts. The first part contained ten 5-point rating scale items allowing students to rate their learning experiences, course outline, syllabus, difficulty level, and workload. The second part consisted of 5-point Likert-type items anchored on "strongly agree" and "strongly disagree," with 18 attributes allowing students to provide critique of their instructors.

Data collected through the qualitative technique involved the use of significant statements to describe students' responses toward their tutors. Approximately 2,991 significant statements were recorded, organized around the following nine themes: Student-Centered, Expert, Ethical, Professional, Director, Enthusiast, Responsive, Transmitter, and Connector. The endorsement rates for each theme are presented in Table 1 below.

Table 1: Percentages of Students Using Each Theme as Evaluation Criteria

Student-Centered: 58.88% | Expert: 44.08% | Ethical: 21.60% | Professional: 40.79% | Director: 21.82% | Enthusiast: 29.82% | Responsive: 5.04% | Transmitter: 23.46% | Connector: 23.25%

The study also employs statistical techniques to present quantitatively collected data using multivariate analysis, combining tables and illustrations. Despite the strategies employed, there is a notable imbalance in the data: more than 70% of the data collected is presented in statistical form, suggesting insufficient balance between qualitative and quantitative presentation. This raises questions about the reliability of the mixed-methods design as implemented.

Conclusion

This paper evaluates the research study titled "Perception of Students Characteristics of Effective College Teachers: Validity Study of a Teaching Evaluation Form Using a Mixed-Methods Analysis" (Onwuegbuzie, Witcher, Collins, et al., 2007). The paper examines the methods of data collection and data analysis employed by the study, and assesses the extent to which the research findings answer the stated research questions. The evaluation reveals that the study is more inclined toward quantitative methods than qualitative techniques in both its data collection and the presentation of its findings. While the mixed-methods framework is conceptually appropriate and yields useful results, the imbalance between qualitative and quantitative components limits the study's reliability as a true mixed-methods investigation.