Survey vs. Reality: Testing Employer Hiring Discrimination

Introduction: The Problem with Self-Reported Survey Data

The research article "Walk the Talk? What Employers Say vs. What They Do" by Devah Pager and Lincoln Quillian aims to develop a method of testing employers' self-reported intentions to hire applicants based on race and/or criminal history. Sociologists have long been aware of a significant gap between reported behavioral intentions and actual actions, yet Pager and Quillian argue that the field continues to rely heavily on surveys as a valid source of information, "with little effort to validate this assumption" (355). This near-universal acceptance of survey and self-report results — particularly with regard to difficult-to-research social phenomena — is what the authors identify as a serious discrepancy in how sociological understandings of race and discrimination are developed.

The article opens with a brief account of a landmark study conducted in 1930 by a sociologist who twice traveled across the country with a Chinese couple to test establishment reactions to their presence. The trio was refused service by only one establishment. The researcher then contacted each establishment by survey, asking whether they would serve people of Chinese heritage — 90% reported that they would not, even though nearly all had done so in person (355). The intentions of that early study closely mirror those of Pager and Quillian: to test the validity of survey results against real behavior on a socially charged issue.

Race, Criminal History, and Labor Market Disparities

Pager and Quillian effectively establish that race and criminal history are particularly problematic topics for survey research, because people's stated positions on these issues rarely match their actions when confronted with a real situation. Because of the relative difficulty of gathering truthful data in a natural setting regarding discriminatory attitudes or potential behaviors, many sociologists continue to rely almost exclusively on surveys. The authors provide several examples of survey types and standards that have been reported as accurate despite well-known weaknesses in the method (357).

The authors first illustrate the potential gap between reported intentions and actual behavior by drawing on research demonstrating pervasive social and economic disparities among those who have a criminal record, are Black, or both. White or Black men with criminal records are far more likely to be unemployed or working in jobs with limited reward and limited mobility. White men with criminal records tend to fare slightly better than Black men with criminal records; nevertheless, both groups face clear discrimination in the labor market and in many other areas of life. Pager and Quillian use raw employment and wage data, as well as data on the disproportionately high incarceration rates among Black men, to anchor the significance of their research questions (355–362). The article also highlights the limited number of direct studies that have tested the correlation between survey responses and actual behavior (360).

The Dual-Method Research Design

The work contends that because openly discriminatory statements carry significant social stigma, information gathered through self-report surveys on such topics is likely to be flawed and inaccurate. To explore this further, the authors designed a study that tests real employer behavior against reported employer behavior using a dual-method audit design. One component tested actual job application experiences; the other surveyed the same set of employers about their hypothetical hiring intentions. The two datasets were then compared to identify any statistically significant disparities between what employers said they would do and what they actually did.

Works Cited

Pager, Devah, and Lincoln Quillian. "Walking the Talk? What Employers Say vs. What They Do." American Sociological Review 70 (2005): 355–380.

Gray, Paul S., John B. Williamson, David A. Karp, and John R. Dalphin. The Research Imagination: An Introduction to Qualitative and Quantitative Methods. Cambridge, MA: Cambridge University Press, 2007.