Why Controlling for Variables Matters in Studies

Appraising Findings of a Quantitative Study

1. Were the primary hypotheses addressed in the introduction?

No hypotheses were stated for the article by Sigala et al. (2022). Instead, the introduction reviewed relevant literature and stated its purpose, which was “to provide the foundation for rigorous evaluations of the impact of restaurant menu added-sugar warning labels on consumer behavior” (Sigala et al., 2022, p. 2). Specifically, the authors aimed to better understand the “relative performance of multiple added-sugar warning label designs while establishing whether restaurant menu added-sugar warning labels could change consumer perception and knowledge outcomes on the causal pathway between warning-label exposure and behavior change” (Sigala et al., 2022, p. 2). With this research intention, the authors proceeded in explaining the methodology of the study, leaving no indication of how they expected the results to play out.

2. How were participants selected and assigned to groups and were groups similar at the start (before the intervention)?

Participants were selected and assigned to groups through a process of recruitment. The researchers recruited 1327 U.S. adults matching 2018 American Community Survey (ACS) 5-year estimates (2013–2018 American Community Survey, 2021) for age (18–34, 35–54, ?55 years), gender, race and ethnicity (Hispanic [any race], non-Hispanic White, non-Hispanic Black, and non-Hispanic Asian), and education (?highschool diploma/GED, some college, ?bachelor\'s degree) from an online sample provided by Dynata. Participants gave their informed consent and participants were assigned to view restaurant menu excerpts with 1 of 25 labels. A simple allocation ratio (via Qualtrics Randomizer) was used for assignment.

3. Did the researcher discuss how the size of the sample was determined?

The researchers did not discuss how the size of the sample was determined, but they did identify it as one of the study’s strengths that the sample reflected national distribution of key demographics. When determining sample size for a study, there are a number of factors to consider. The first is the type of data that will be collected. For example, if the data is dichotomous (i.e. it can only take two values), then a smaller sample size may be sufficient. However, if the data is continuous (i.e. it can take any value), then a larger sample size will be needed. The second factor to consider is the level of precision required. For example, if the study is looking at a small effect size, then a larger sample size will be needed in order to detect this effect. Finally, the type of statistical analysis that will be used should also be taken into account. For example, parametric tests usually require larger sample sizes than non-parametric tests. In general, therefore, there is no one-size-fits-all answer to the question of how sample size should be determined. Rather, it depends on the specific characteristics of the study in question (Adcock, 1997).

4. Were the interventions well defined and consistently delivered (fidelity to treatment)?

The experiment was well defined in terms of explaining how the study was done with participants interacting with the visual stimuli. Consistency is assumed throughout. No intervention was delivered in the sense of a treatment being given, but rather various groups were given different stimuli in accordance with the details provided in the write-up. Control groups were said to be given menus with a QR code for scanning information, while experiment groups were given visual stimuli related to sugar warnings.

5. Were study groups treated equally other than the difference in the intervention?

It appears to be the case that study groups were treated equally to the control group other than the difference in the intervention. Ideally, the same number of people are assigned to each group, and the groups are given equal attention and resources. Only by ensuring that the groups are as indistinguishable as possible can researchers be confident that any differences in outcome are due to the intervention itself, and not to other factors. However, in some cases, one group may be more motivated to stick with the program. If there are differences of this nature it can bias the results of the study, making it difficult to draw accurate conclusions. The researchers did not identify any issues of this sort in their study.

6. Were important extraneous variables and bias controlled? If yes, how?

The researchers made no mention of extraneous variables or bias, although doing so would have made the study stronger. That is because it is important to assess whether the extraneous variables and bias are controlled. By definition, an extraneous variable is anything that influences the dependent variable other than the independent variable. There are many extraneous variables that could potentially influence a study, so it is vital for researchers to identify and control as many of these variables as possible. Otherwise, the results of the study could be invalid. Additionally, bias is another factor that can potentially distort results. Researchers must be aware of their own personal biases and take steps to control for these biases in their studies. Only by carefully assessing all of these factors can we ensure that the results of a study are valid and reliable. The researchers did look at many different demographics to try to control for different extraneous variables but no discussion was given to the matter in terms of the design of the study. However, the researchers did discuss one form of bias and how they addressed it by stating that “although social desirability bias could have influenced PME and warning label support, this is unlikely given participant anonymity” (Sigala et al., 2022, p. 7).

7. If a difference was found, are you confident that it was due to differences caused by the intervention?

I believe that the differences found between the control and experiment group were based on differences caused by the intervention. As the researchers noted, “added-sugar warning labels for restaurant menus are perceived as effective and may educate consumers on menu items\' added-sugar content are consistent with evidence from a small body of online experiments testing restaurant warning and traffic-light labels for other nutrients” (Sigala et al., 2022, p. 6). Still, there are some limitations of the study that need to be considered, and the researchers are good about going over them, as they may have influenced outcomes as well—such as the fact that participants were primed in advance, which “could have led to larger effect sizes than would be observed for ordering outcomes without priming” (Sigala et al., 2022, p. 7).

8. Do the findings demonstrate statistical significance? Clinical significance? How would you use the study findings in practice?

The researchers did not test for clinical significance but they did test for statistical significance and state clearly that “to examine the statistical significance of potential moderators, we ran linear models unstratified with the addition of indicators for each level of a moderator and interaction terms between level of a moderator and warning label group” (Sigala et al., 2022, p. 4). In the report they provided stratified effects on PME and p-values for interaction terms. The researchers also added that “for the primary outcomes of PME and knowledge of items\' added-sugar content, we additionally calculated Cohen\'s d and examined statistical significance after using the Holm-Bonferroni procedure (not pre-registered but added based on peer-reviewer feedback) to correct for multiple comparisons within each family of outcomes and comparisons (e.g., comparison of PME among the 3 main label groups, comparisons of PME among the 6 icon-only labels)” (Sigala et al., 2022, p. 5). In terms of demographics among the groups, the researchers reported no statistically significant differences.