This paper presents a detailed methodological critique of Fisher and Meyer's study examining the effects of inclusive versus self-contained educational programs on children with significant disabilities. Using the SIB and ASC assessments as dependent measures, the original study attempted to evaluate child development and social competence outcomes. This critique systematically evaluates the study's research question formulation, hypothesis testing, research design, participant sampling, measurement instrument validity, and statistical tool selection. The paper identifies significant errors across each component and concludes that the study's findings cannot be meaningfully interpreted. Specific corrective recommendations are provided to guide future, more rigorous investigations into inclusive educational programming.
The paper demonstrates systematic research critique, a core graduate-level skill in which a reviewer evaluates each component of a study (design, sampling, instrumentation, statistics) against established standards. The author correctly distinguishes between research questions designed to test for differences versus effects, and explains why this distinction dictates statistical tool selection — a nuanced methodological point that anchors the central argument.
The paper opens by defining the study's variables, then moves through the original study's components in logical order: research question and null hypothesis, research design and internal validity, participant sampling, measurement instruments and statistical methodology, and finally results and implications. Each section identifies a specific flaw and, where appropriate, explains what the correct approach should have been. The conclusion consolidates all critiques into an ordered list of recommended revisions.
Scientific investigation includes both independent and dependent variables. The independent variable is the cause (antecedent) of the dependent variable, the presumed effect (consequence). For the present study there are two independent variables: inclusive and self-contained educational programs. The receiving, or dependent, variables are child development and social competence as measured by the SIB and ASC tests. Although not included in the present investigation, additional independent variables could have been age, gender, and eligibility category.
According to the authors, the study was designed to examine the effects of two different types of educational programs — inclusive versus self-contained — for students with significant disabilities, with respect to gains and rate of improvement in levels of development and social competence as measured by the SIB and ASC on a pre- and post-test basis. The authors' research question was stated somewhat inappropriately: "What are the effects of attending inclusive vs. self-contained programs on children with significant disabilities, and what types of gains and rates of improvement can be anticipated?" When stating a research question, the investigator is not at liberty to assert directional consequences. The authors did this by using the word "what," which implies that differences are to be expected. The researchers should instead have stated the research question more neutrally: "Are there any effects…?" In this manner, the researchers remain open to results or no results — which is the prudent way to formulate a research question.
When a research question has been presented, the natural flow leads to the statement of a testable null hypothesis or hypotheses. Null hypotheses state that there will not exist any effect of the independent variable on the dependent variable as measured by a pre-selected assessment instrument. The Fisher and Meyer study failed to state any testable null hypothesis, and as a result, the test data is susceptible to erroneous and flawed interpretation. More appropriately, the authors should have constructed the following testable major null hypothesis: "There exists no statistically significant difference at α = .05 in pre- and post-test SIB and ASC scores for severely learning disabled students and social competence with respect to inclusive and self-contained educational programming." As the study contains various levels of social competence and skill development, the appropriate null hypotheses should also have been stated with respect to interaction effects. Nested variable consideration should have been given with respect to nested interaction effects and nested variables such as age, gender, and eligibility levels.
Written testable null hypotheses are required for making appropriate statistical tool selections. When a researcher chooses to study differences, a certain statistic is called for. When a researcher is looking to study relationships, a different type of statistical tool is employed. Without a null hypothesis from the outset that clearly states whether the study examines differences, relationships, or effects, the researchers have no basis for choosing the statistical tool they opted to use, nor for interpreting the results.
With respect to the study's research design, the authors only lightly presented the type of study they conducted. The design section must be extremely organized and specific. All researchers are required to state precisely what type of research investigation is being presented for evaluation and conclusion, and all sub-components of the selected design are to be clearly stated as well. The Fisher and Meyer study again failed in its commitment to regulatory research compliance. Although a longitudinal approach was mentioned early in the study, there was no explanation of how the study was to be kept free of internal, external, or extraneous error. A researcher must always keep in mind that any selected design carries with it certain limitations as well as unique sources of error. In order to counter these effects, certain manipulations or controls must be applied to both the design and the statistical tool.
Anything that can affect the controls of a study threatens its internal validity. With respect to internal validity contamination brought about by lack of control, Fisher and Meyer failed to report sampling procedures, test administration controls, procedures and interpretations, scoring mechanisms, SIB and ASC reliability factors, and a host of possible nested variables — including standardization procedures in test administration, location of testing, conditions of the testing environment, and subjects' level of test-taking acceptance. Without sufficient information regarding the entire testing situation and structure, the chosen sampling method, and the standardization qualities of the selected measurement instruments, the results and outcomes of the study are highly suspect with regard to meaningful interpretation and result application.
As noted above, no information was presented regarding the reliability and validity of the two chosen instruments. Whether or not the selected measurement instruments meet the requirements of standardization is unknown. Furthermore, the review of literature section did not discuss the worth of the two instruments whatsoever — yet it is within this section that a reader looks for information as to the importance of using the selected measurement instruments. Additionally, Fisher and Meyer failed to present any previous research information regarding the selected measurements' comparability to, or compatibility with, other similar assessment devices. Should no similar measurement instruments exist, the authors were obligated to state that lack. All of this is required for the reader to be able to objectively evaluate the "goodness" of the research and the data being produced.
Not only are results interpreted on the basis of the alignment between the research question and the null hypothesis, but the very manner in which a research question is phrased dictates the type of statistic that will be used to analyze the collected data. For example, should a researcher wish to test for the relationship between intelligence and school achievement, a correlation coefficient is used to analyze the data. Fisher and Meyer stated that their research interest lay in testing the "effects" of the independent variable on the dependent variable with respect to two sets of measurement scores in a pre- and post-test situation. Choosing an independent-samples t-test does not, in any way, demonstrate effects. A t-test is employed to test for differences between two independent groups that have been measured in one or more ways. By using a t-test, the Fisher and Meyer research question should have read: "Are there statistically significant differences in the SIB and ASC scores of learning disabled students on a pre- and post-test basis who have been in either an inclusion program or a self-contained program?" Under no circumstance is the t-test suited to testing for effects — only differences. As a result, the analyzed information is inappropriately applied back to the investigators' original research question. Either the question must be revised, or another statistical tool — one suited to testing for effect — must be selected. The recommended tool in this example would be an analysis of variance with orthogonal modifications. Even though the authors used a univariate analysis of variance to show simple differences in gain scores for each group, this particular type of ANOVA shows differences that result from "effect," not differences between mean scores. Again, the wrong statistical choice was made.
Assuming the t-test were the appropriate tool, the authors failed again by not pre-selecting their level of statistical acceptance. When there is no stated testable null hypothesis, there is no reason to pre-set an alpha or probability level for accepting or rejecting the null hypothesis. In cases like this, researchers are effectively free to explain any result they choose, unbound by controls established at the outset of the study. This, as most research scholars would agree, constitutes sloppy research.
Always verify citation format against your institution’s current style guide requirements.