Analysis of Null Hypothesis Significance Testing

¶ … Nhst

Compare and Contrast Null Hypothesis Significance Testing (NHST)

The most commonly used statistical technique for testing the impact of the factor being discussed on observations is Null Hypothesis Significance Testing (NHST). Consequently, NHST is the famous approach to inferential statistics, especially when conducting quantitative research. Despite being the dominant approach, NHST has also become increasingly controversial given the belief by a considerable number of people that it is a flawed statistical method. The controversy and consideration of Null Hypothesis Significance Testing as a flawed statistical approach has contributed to the development of alternatives whose proponents consider more beneficial or advantageous unlike NHST. However, an understanding of Null Hypothesis Significance Testing requires correct interpretation of p values.

Meaning of p = .05

P value is commonly used across statistical approaches including regression analysis and t-tests because it determines the statistical importance or significance in testing a hypothesis. According to Frost (2014), p values are usually used to determine the statistics to be published as well as projects that require funding. Regardless of its importance in determining the statistical significance in a hypothesis test, p value is usually a slippery concept that is incorrectly interpreted and understood. An example of incorrect interpretation of p values is the meaning of p = .05, which has been characterized by some misconceptions and wrong interpretations. Some of the common misconceptions of p= .05 include belief that the null hypothesis has a 5% chance to be true, there is a 5% chance of a Type I error, there are no variations between groups, and there is a 95% chance of similar results if the study is repeated. These misconceptions are wrong because p values are not the likelihood of making mistakes through rejecting a true null hypothesis.

Generally, p values examine how well the sample data support the claim that the null hypothesis is true (Frost, 2014). Therefore, p = .05 means that the sample data is unlikely with a true null hypothesis because the p value is low. In essence, a higher p value implies that the data is likely with a true null hypothesis and vice versa. All research should comply with the p = .05 standard for significance because it evaluates the compatibility of the sample data with the null hypothesis. In essence, the standard postulates that the sample offer sufficient evidence that the null hypothesis can be rejected for the whole population. The other reason for all research to adhere to p = .05 standard is because researchers use the value to determine whether or not to reject the truth of the null hypothesis (Carver, 1978).

Effect Size and Statistical Significance

Effect size can be described as the measure of the magnitude or extent of variations between groups, which are standardized through controlling for differences within groups. In contrast, statistical significance basically means statistical rareness, which implies that results are regarded as important from a statistical perspective since they occur rarely in random sampling based on null hypothesis conditions (Carver, 1978). The similarity between the concepts of effect size and statistical significance is that they both rely on the p = .05 standard to determine important aspects of the study from a statistical perspective. However, these concepts differ in the sense that effect size depends on variations between groups while statistical significance seemingly depends on sample size. Moreover, statistical significance implies variations between research groups at the level of 0.05 unlike effect sizes.

Statistically Significant Result v. Clinically Significant Result

A statistically significant result differs considerably from a clinically or real world significant result. The major difference between these two concepts is that a statistically significant result considers results to be vital from a statistical perspective based on the conditions of the null hypothesis whereas a clinically or real world significant result considers results on the basis of their practicality rather than hypothetically. As a result, statistically significant results may not necessarily be clinically or real world significant in many studies. For instance a statistically significant result can be the mean variation between two research groups or samples. On the contrary, an example of a clinically or real world significant result is the meaningfulness of the outcome of variations between examined samples or research groups.

Null Hypothesis Significance Testing

Null Hypothesis Significance Testing (NHST) can be described as a statistical technique that is employed to test the impact of factors being examined on the observations made by researchers conducting the study. Consequently, this statistical method has been regarded as the single solution to inductive inference when conducting a study. Moreover, Null Hypothesis Significance Testing (NHST) allegedly offers social scientists with a means for differentiating probabilistically true findings from those linked to simple chance difference (Levine et al., 2008, p.175). NHST is based on several assumptions including belief that p value indicates the likelihood of obtaining a value of test statistic, rejection of the null hypothesis if the p value equals or is less than the selected alpha, and consideration of the null hypothesis to be false if data is adequately improbable if the null hypothesis is true.

Criticisms of NHST

Despite being regarded as the most popular inferential statistics method in quantitative communication studies, Null Hypothesis Significance Testing has been characterized by increased controversies. These controversies mainly emerge from the criticisms associated with this method, which has led to the development of more advantageous alternatives. One of the most common criticisms of this statistical approach is that it is increasingly sensitive to sample size, which is a major limitation. In this case, when the sample size is small, important and strong impacts can be non-significant and in situations where the sample size is large, trivial effects can generate relatively impressive p values (Gliner, Leech & Morgan, 2002, p.84). Secondly, NHST is criticized for the fact that the point or nil-null hypothesis is usually literally false regardless of the verisimilitude of the significant hypothesis. Therefore, it is unimpressive and uninformative to disprove the null hypothesis if it's always false. The third criticism of NHST is that it comprises power and error rates, which are reflected in highly unacceptable Type II error rates.