The Seven Deadly Sins of Web Survey Design

Guest post by Sue Chambers

Not so long ago, administering a survey to get feedback from your clients was a time-consuming and expensive venture. These days, conducting a survey of your clients (I’m using the term loosely and expanding it your blog readers) has become ridiculously simple and cost effective with the advent of web-based survey tools. All you have to do is find a user friendly survey tool, throw together a few questions and send them out any time you want to do a bit of marketing research. Perhaps the process has become a bit too quick and easy.

Why do I say that? Because while it’s easy to write out a few questions you think will answer your questions for little or no cost, constructing good survey questions is an art and a skill. Designing a good survey isn’t something you can or should just “crank out in a hurry.” Commit any of the following survey design “sins” often enough and you will likely end up:

(1) irritating your survey respondents to the point of tossing out your survey;

(2) having to throw out data because you can’t use it (Please don’t use bad data to guide your decisions or planning!), and;

(3) wasting both your and your clients’ precious time.

If you want to find out how to design excellent survey questions that will keep your clients happy and give you a great set of data to use for planning and decision making, here’s how to make amends for committing survey design sins.

1. Drop the cloaking device.

Be up front and clear about what you’re asking for. If you want feedback from your readers about a catchy title for a workshop, then ask them to rate a variety of different titles, don’t ask them about the types of workshops they’d take. Your survey respondents will catch on pretty quickly that their options are really just variations on the same theme, and they might not appreciate feeling manipulated or played for a fool.

2. Nothing personal, but…

Do you really need to know the age, sex, ethnicity and exact annual income of your survey respondents? You might be surprised at how sensitive individuals can be about sharing personal details–especially if they have any concerns about how you might use that information. If you’re really convinced that it’s vitally important to have some demographics about your clients, use age or income ranges rather than asking for exact numbers, and place the demographic questions at the end of your survey. That way, even if your survey respondents refuse to answer any more questions, you’ll still have their answers for the really important questions.

3. Stop pointing that double-barreled question at me.

Have you ever had someone ask you about two different issues in one question? Did you wonder how on earth the person would figure out which part of the question you were responding to? Here’s how a double-barreled question might appear in a questionnaire:

Do you think that politicians should be allowed to accept gifts from lobbyists and should be recalled if they don’t follow their constituents’ wishes? Yes No

Not only would a question like this perplex your survey participants, it would perplex you when it came time to analyze the data. How would you know whether the responses referred to the issue of accepting gifts from lobbyists or the question of whether politicians should be recalled? A better way to frame your question would be as follows:

For each of the following items, please tell us whether you agree or disagree

Politicians should be allowed to accept gifts… Agree Disagree

Politicians should be recalled if … Agree Disagree

4. Leading questions are not star material.

If you are trying to nudge your respondents’ answers in a particular direction, then you have written a leading question. If you catch yourself writing questions that start with phrases like “Don’t you think” or “Wouldn’t you like…”, reword those questions immediately.

Leading questions feel manipulative. Survey participants might grudgingly give you the answer you obviously want, or they might teach you a lesson by deliberately selecting the opposite of what they think you want. Either way, it puts the validity of your responses in doubt and it would be unwise to use the data as a basis for making decisions. Just for the fun of it, here’s a painfully obvious example of a leading question I threw together for this article:

Don’t you think this is the most helpful blog post you’ve ever read? Yes No

By the way, if I were actually going to ask you what you thought of this article I’d phrase it like this:

On a scale of 1 to 5 where 1=Not at all helpful and 5=Very helpful, tell us how helpful the article was to you. 1 2 3 4 5

5. Unload those value-laden words in your questions.

Stay away from using descriptive words with a lot of emotion or judgment attached to them. If you attach a negative label to a behaviour you’re asking about, you’re not likely to get truthful answers to your questions. In turn, this will affect the quality of your data. Think about it. How likely are you to give an honest reply to the following question:

How much time per day do you waste on ridiculous social media activities instead of working on important projects?

I’m betting you’d be more willing to report the actual amount of time per day you spend on social media activities if the question were phrased neutrally.

While I’m thinking of it, your survey participants will greatly appreciate it if you banish vague terms like “once in a while”, “sometimes”, “frequently”, etc, from your response categories. These descriptors are interpreted subjectively so you don’t know what you’re really measuring. Use specific quantities and units of time; e.g., once a week, less than 2 hours a day, twice a month, etc.

6. Appreciate the joys of mutual exclusivity and exhaustive categories.

Make life simple for your survey respondents and avoid overlapping response categories. What’s a 25 year old person to do when she is confronted with something like this: “Are you 20-25 years of age, 25 -30 years, of age, etc”. The other side of this coin is the failure to include enough response categories to include all potential respondents. Suppose a 55 year old male wants to complete your survey but there are no response categories for people over 50. What do you think he’ll conclude about your ability to be inclusive of all clients or readers?

Your response categories should be mutually exclusive but you need to ensure that your survey is as inclusive as possible. Obviously you can’t list every possibility but you want to capture as much information as you can in your survey. You can do this by including categories with no specific end points (e.g., 55 years and over, more than $100,000) and by adding a category and a space to describe “other” responses you either hadn’t thought of or missed.

7. Your clients have the right to be uncertain, to not know, and to not care one way or the other.

I don’t know about you, but I get really annoyed if I’m forced to answer a survey question when none of the choices accurately describe my experience or the survey item is not applicable to me. To my way of thinking (based on my experience with survey research), forcing respondents to select a response when they don’t know the answer or the question is not applicable to them is not helpful from a data perspective nor is it respectful of survey participants’ choices.

Likewise, I’m a proponent of having a neutral point for survey questions with rating scales. I’ d rather risk having a few individuals take the path of least resistance and select all of the middle (neutral) values on a series of scaled items than artificially inflate the overall number of positive responses. (If you are interested in learning more about scaled items in a survey, you might like to read this article.)

Of course, there are a few other cardinal rules and principles to consider when you’re designing a survey, but learning how to identify and fix the seven sins covered here should help to get you out of survey purgatory and on the way to writing good survey questions.

Sue Chambers likes to use her editing, research and writing skills to help others make a positive difference in the world. If you’re looking for tips on writing or creating social change, you might enjoy reading her blog Sagewit.

Photo credit: NeoGaboX

Author: Annabel Candy