Survey Questionnaire Design: Critique and Recommendations

Survey Length and Early Placement of Critical Questions

Because the modal time for dropouts in live surveys — whether short or long — is at the halfway point, the researcher needs to place the most crucial information early in the survey (Cape, 2010). Which information qualifies as crucial depends on the focus of the researcher (demographic information is addressed separately below). Additionally, the questionnaire is somewhat cumbersome and will require considerable training before interviewers can use it effectively, particularly in sections such as Section F. It would be better to add clearer guidelines, simplify some content, and make the instrument more user-friendly. The fewer errors an interviewer makes, the lower the bias and the greater the reliability — and hence validity — of the measure (Bradburn, Sudman, & Wansink, 2004; Kalton & Schuman, 1982). Specific issues with individual sections are discussed below.

Measurement Consistency and Recall Period Issues

Question A4 asks about reading within the past week, whereas other questions in the same section ask about more typical behavior patterns over time. For the sake of validity, it is best to keep questions measuring similar parameters consistent with respect to the time frame they reference (Bradburn et al., 2004). Furthermore, the optimal recall length that participants have for ingrained behaviors — such as reading the newspaper or watching the news — is longer and more reliable than it is for other behaviors, so questions of this nature can be framed more generally (Clarke, Fiebig, & Gerdtham, 2008).

Dichotomous Scales and Level of Measurement

In Section B, the researcher uses a dichotomous opinion variable where a higher level of measurement could yield more information (Bradburn et al., 2004). This can be problematic for later analysis alongside other variables that appear at the ordinal or interval level. Although classic findings suggest that no significant changes to the internal consistency of a scale necessarily result from this type of change (e.g., Matell & Jacoby, 1971), using a higher-level scale does yield more information, and it is best to remain as consistent as possible across items (Bradburn et al., 2004). If the variable will ultimately be used as a dichotomous independent variable, it is preferable to measure it on an ordinal or interval scale for consistency and then dichotomize it later during analysis (Bradburn et al., 2004).

Section D asks whether the respondent or anyone in the household belongs to a particular group. As worded, it is impossible to determine who specifically belongs to the group in question. Specifying who belongs to the group would be useful for later analysis (Bradburn et al., 2004).

Section L contains a number of confusing questions and is quite cumbersome (e.g., L2A). These items need to be simplified. In addition, several questions again use dichotomous answers to measure dimensional attitudes. For purposes of comparison and later analysis, these variables could be transformed to an ordinal or interval level — for example, L4 could ask "How strongly do you favor…" with a Likert-scale response. This change would also streamline the questionnaire so that the interviewer would not need to jump between questions but could simply ask one question about whether the participant opposes or favors a particular policy.

Double-Barreled Questions and Ambiguous Items

Section E asks a number of double-barreled questions. If the researcher wants more specific details about a particular behavior, it would be prudent to add a follow-up query after each question to capture that specificity (Bradburn et al., 2004).

Section P may be problematic in a live interview format because it risks putting respondents on edge by making them feel uninformed. This can prompt a defensive reaction, and as currently written, the researcher does not yet have any demographic information about the participant when these questions are asked. If this information is crucial to the aims of the research, the questions should be reworded to resemble opinion items rather than a quiz — for example, "Who do you believe has the final responsibility…?" and, if the respondent does not know, asking whether they would care to guess.

Section Q also contains double-barreled questions. For instance, specifically identifying a few minority groups and then lumping all remaining groups together as "minorities" may elicit a different response than asking about minorities in general. It might be advantageous to change Q3 simply to "minorities" or a similarly general term. There are other double-barreled questions and artificially created dichotomous answers in this section that would be more useful if restructured. The same critique applies to Section R. Section X asks the same question twice and should be corrected accordingly.