Computers in teaching: applications and outcomes

Sociology Research

Organizational Research

Article Evaluation

The purposes of the research study completed and reported by Poirier, Christopher, R., & Feldman, Robert S. (2004) centered on the effectiveness of in-class vs. online learning environments, as measured both by students' performance academically and their levels of satisfaction with each course structure and delivery method. In measuring student satisfaction with the course, four scales were used. These four scales consisted of overall course rating, interaction rating, feedback rating, and amount of time spent on the course. The authors define the methodology as using a waiting-list experimental design where variations in students' interest between in-class and online instruction is minimized as a source of potential variation in the study results.

Setting/Location

As the purpose of the research effort is to measure the effectiveness of in-class vs. online teaching techniques, both from a learning effectiveness and also from a student satisfaction standpoint, classrooms at a large state university were used for one set of respondents, and the respondents in the second set worked from home, remotely via the Internet. As Poirier, Christopher, R., & Feldman, Robert S. (2004) are from the University of Massachusetts, Amherst it is assumed that this is the state university mentioned.

Methods

Participants/Subjects. The participants in the study are twenty-three undergraduate students enrolled in a large state university. These participants have enrolled in an introductory psychology course, indicating they were interested in taking either in a classroom environment or online. A waiting list of students was created, and then a simple random sampling technique was used to assign 11 students to the very large in-class course, which had 477 total students in the course. The remaining 12 students were assigned to the online course. Two of the 11 enrolled in the large, traditional course dropped within two weeks are not included in any subsequent analysis. Poirier, Christopher, R., & Feldman, Robert S. (2004) relied on a waiting-list experimental design to significantly minimize the variation in results based on in-sample variation alone.

Variables. Status/categorical variables included the students' registration and opt-in for either course format while the manipulated variables include 21 specific evaluation criteria distributed across the four scales of instructor evaluation, overall course rating, interaction/feedback ratings, and amount of time spent on the course. Independent variables are the characteristics of the participants, the content and learning materials presented both in the classroom and online, and the teaching approaches, strategies and techniques used by instructors. Dependent variables include the 28 multiple choice questions that comprise the examinations given to both in-class and online students, 23 of which are both administered to both groups of students. 21 total evaluation variables comprise the of instructor evaluation, overall course rating, interaction/feedback ratings, and amount of time spent on the course.

Instruments. Dependent variables were measured in three specific areas, the first being examination scores, the second being papers authored by students, and the third being course evaluations. As the first two of these are common means of measuring student performance (exams and papers) they are not discussed in detail here. The third instrument for measuring course evaluations was completed in class on forms produced by the psychology department, as is commonly used to evaluate both course and instructor effectiveness from the perspective of students. For online students a Web-based form was used for completing the data collection. The reliability and validity of examinations and papers are basically sound as both groups of students completed their papers outside of class, and both groups were tested in a classroom. The validity and reliability of course evaluations is strengthened by the anonymity of responses both in class and online, and also supported by the results of the 21 factors being tested between each group.

Design. The experimental research uses a causal comparative model as the basis for testing the variations in student performance between in-class and online academic performance and student satisfaction. The diagram is shown below:

Internal Validity Threats.

History - Yes, this could have been a potential threat to the research if the Internet connections of online respondents were impacted by an outage, or if the courseware deviated between audiences. Unforeseen technology-related events could have impacted the study.

Maturation - No, this would not have been a potential threat as the class is short in relative time period. This was controlled through semester scheduling.

Testing - Yes, this could have had a major impact on the research, and needed to be addressed through the definition of methodologies and statistical test definitions.

Instrumentation - Yes, this could have had an impact on the research, yet was minimized by the standardization of tests, the inclusion of online students in classrooms for proctored exams, and the use of a standard evaluation form.

Mortality - Yes, this was a major risk and is illustrated in the 2 students who dropped in-class participation. This was controlled through the selection of the respondent base.

Regression (Statistical Regression), Location - No, did not pose a risk to the research as the respondent base did not include previous scores.

Selection threat - Yes, this could have been a risk to the research, and is exemplified in the 2 who dropped the in-class session. False sampling could have lead to this threat occurring.

Social interaction threats a. diffusion of treatment - Yes, this could have specifically been a threat, and it was mitigated through the confidentiality of the test.

b. compensatory rivalry - Yes, this could have been a threat yet due to the confidentiality of the research this did not occur.

c. resentful demoralization - Yes, this could have been a threat yet it was also managed through confidentiality.

d. compensatory equalization of treatment - Yes, this could have been a threat yet was managed through confidentiality of the study.

Results/Discussion

What was the research hypothesis? The null hypothesis was that there was no significant difference in student academic performance or satisfaction based on their learning through in-class or online. The alternate hypothesis is that online learning students are significantly different in terms of academic performance and satisfaction.

What is the name of the inferential statistic used to test the hypothesis e.g., t-test, Chi-square, one-way ANOVA, two-way ANOVA, one-way

ANCOVA, MANOVA, MANCOVA)? A mixed design ANOVA was used, please see Poirier, Christopher, R., & Feldman, Robert S. (2004) (60 -61) for specifics of the design.

Label the calculated value, the df and the significance. Please see Poirier, Christopher, R., & Feldman, Robert S. (2004) for the degrees of freedom and statistical significance of the ANOVAs completed.