bias reinforcement through survey questionnaires

Today I play media theorist and examine how survey questionnaires reinforce survey designers’ biases:

The knowledge that biases emit from survey questionnaires is nothing new. The extreme case, “push-polling”, intentionally guides the questionnaire reader toward a viewpoint, without real interest in their prior opinion. Any survey writer willing to push-poll already understands my concerns about bias (because they are propagandists).

It is the unintended or “honest” biases that concern me here.



Consider for example the common belief that individuals can be categorized as a member of one out of four or five distinct racial groups, a belief reflected in many survey questionnaires that ask respondents to indicate which race they belong to. This is an example of what I call an “honestly” projected bias; the survey writer likely has limited awareness that there is even a problem, and does not expect their respondents to question the belief. In these cases, the bias enters the survey questionnaire through the questionnaire writers’ phrasing and provided options, and is confirmed when each respondent chooses one of the options.

Stepping back, we observe “bias in, bias out” where the belief itself gains strength across the survey process. It strengthens among the respondents as they accept the belief when answering the questions, and strengthens in the mind of the survey creator when they see tacit acceptance of the bias in the responses. At each step, neural pathways supporting the belief become stronger due to exercise.

I’ve mapped this process below, illustrating the cumulative bias amplification by degree of red in the arrows’ color:

While we cannot completely escape projecting our biases through our measurement instruments, I call on questionnaire writers to step back and consider what we might be propagating. We may have to become more creative to limit the damage. (For one example of a creative approach, see my post “a better way to ask about gender in survey questionnaires” for an idea on how to avoid propagating the binary sex/gender bias through survey questionnaires).

a better way to ask about gender in survey questionnaires

Survey questionnaires regularly ask respondents’ sex or gender, and mostly offer only the binary options:

When presented with such a survey on paper, I typically add and then select a third option: “Fuck you”. (Similarly, I do the same with race/ethnicity questions when asked to choose one out of four or five options).

However, we increasingly answer surveys online, making this write-in approach unavailable. Furthermore, scrawling profanities onto survey forms fails to positively address the very serious problem underlying my anger: that the binary sex/gender classification erases folks who, for a variety of reasons, do not fit within it.

In what I perceive as an honest attempt at inclusion—and I sincerely appreciate the effort—Google offers an “Other” option in its Google+ profile form:

But simply adding an “other” option still emphasizes the binary classification system; it reminds respondents that they either fit in, or don’t. Very few of those who don’t fit in enjoy that interjection when it involves something as fundamental as gender identity.



I recommend the following alternative for collecting gender and sex data from survey respondents:

Here the use of sliders reflects the continuous natures of sex and gender, while the division of the query into separate, orthogonal dimensions accounts for the distinctness of biology (sex) from social artifact (gender).

Certainly this scheme fails to capture all the nuances of gender identity, particularly its flux within individuals, but it reaches for a more honest and inclusive world.