<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=1745228055785029&amp;ev=PageView&amp;noscript=1">
BOOK DEMO

The Horror of Survey Bias and 4 Ways to Avoid It

by Michelle Kratzer
|

Halloween is right around the corner, invading our minds with ghouls and ghosts and our movie screens with horror films. But, perhaps the most frightening of all is that thing slithering across our gray matter, influencing our thoughts and impacting our conduct. You know what I’m talking about -- survey bias. I’m not saying I’m terrorized by nightmares about bias lurking about, plotting to infiltrate my survey questions. That would be weird. Still, let’s review how to protect our questions from bias and our data from contamination. After all, the data you gather from a survey is only as good as the questions you ask on it.

Take it from Dr. David Michaelson from Teneo Strategy: “The most important advice I can give about spotting bad research is to assess if the questions are self-serving and biased.” We couldn’t agree more! 

Well designed survey questions are the key to usable data and great analytics.

That’s why we’re sharing 4 tips on avoiding bias in your survey questions:

 1. Don't use Leading Language

If your survey questions are worded so that they steer respondents toward a certain answer, your data will be predictable garbage. As we learned in Freshman Statistics class, there are all kinds of ways to manipulate a respondent into saying what you want to hear, but if you already know what you want to hear, why are you conducting a survey in the first place?

Survey response data should rock your world. If you aren’t hearing things you didn’t know before, you either aren’t asking the right questions, or you aren’t asking them the right way. Write questions in neutral language so that the only one steering the answer is the one taking the survey. Be wary of leading adjectives or words that carry extra connotations. Instead of “Do you oppose this EXPENSIVE new method?”, just ask “Do you oppose this method?”

Instead of “How much BETTER do you feel today?” ask “How do you feel today?” A neutral, plain-Jane question keeps you from influencing customer responses and, therefore, it allows for answers that give you the juicy information you need to know.

Follow me

 

2. Make it Anonymous

Oh to be loved... or at least liked... or at least belong…

Human beings have a hard time with honesty. Honest answers and opinions may make us feel like we’ve put our love and belonging in jeopardy. But, behind the magic veil of anonymity, we become bolder to speak our truth. Anonymity is crucial in surveys, where, according to Pew Research, respondents understate everything from alcohol and drug use to tax evasion and racial bias.

Bindu Reddy, who runs her own large social-media marketing company called MyLikes, and has managed apps and services for large companies like Google, says that her inspiration for her best selling app Candid was her friends’ aversion to stating their opinions openly on social media - “it’s hard to say anything opinionated or even remotely controversial without facing a huge backlash.”

She created Candid so that users would feel like they had a safe space to be honest in the shadow of the app’s anonymity.

Be invisible

  

3. Consider the Order effect

You can’t just throw ingredients together in a bowl and expect a perfect cake to come out. Ingredients must be added in a certain order to make the magic happen. Similarly, survey questions must be ordered in a certain way, or the answers they render may be skewed, flawing your data (cf. Schuman & Presser, 1981).

For example, if you are doing a survey which requires a question on marital status, and the respondent must choose between single, married, divorced or separated, all kinds of emotions get stirred up which could bias the answers for the rest of the survey. Put anything difficult near the end to leave the other questions unpolluted -- this includes personal questions, or anything that stirs up strong emotions.

Remember, getting people to think about unpleasant things and events can put them in the wrong state of mind which can cast a negative shadow on subsequent questions. 

 

4. Avoid General Confusion

Nothing about ambiguity renders useful data. Take note of the following ways that confusing questions can compromise your data:

  • Using technical jargon, acronyms or slang may confuse the user, rendering flawed responses. Instead, use everyday language that all respondents will understand and interpret in the same way.

  • Avoid double barreled questions that try to measure 2 things at once. You can’t ask “Do you eat fruits and vegetables?” because some may eat only fruit, or, only vegetables. Stretch this out into 2 questions.

  • Not including a time or date on your survey increases its ambiguity. Include a time or date, if possible, to ensure that your respondents are all talking about the same time period. A question like “What is your GPA” becomes clearer when a time-frame is given: “what is your GPA for all of the classes you took in Spring semester of 2018?”

  • As often as possible, avoid negatively worded questions. Research suggests that these questions take longer to process, resulting in respondents getting frustrated with decoding the questions and the extra time it takes to answer them.

Ask only answerable questions, and include all possible answers options. If you don’t, a respondent may be forced to select an answer that is not totally true or does not accurately reflect their feelings. This is called forced choice bias. If you cannot be sure you’re including all answers, switch to an open ended question so that the respondent can answer as they like.


Creating a great survey is a worthy endeavor which will pay for itself in both money and peace of mind; but it takes time to make sure that none of that nasty bias is hiding in your questions. Now that you know how to recognize it -- don’t be horrified if it appears, just use these helpful tips to kick it to the curb and get the feedback you need and want.

If you need any help creating effective customer surveys, call SurveyMe. Our CX team will slap on their white gloves and remove that ugly bias for you ASAP!

White Glove Service

Click on the bitmoji above to contact us!

Get our blog right in your email!