The good news: Online survey tools have made it easier
than ever to conduct member-needs surveys quickly and
inexpensively.
The bad news: These technologies also make it easier than
ever to conduct badly designed surveys.
The problems aren’t limited to online tools. Poor design
can undermine the value of any survey, including the
traditional mail and phone kinds. However, the popularity
of online survey tools that practically anyone can use to
disseminate Web-based surveys makes it more likely that
untrained staff will neglect the importance of writing
questions that will produce reliable results.
Whenever you inadvertently conduct a badly designed
survey, the best-case scenario is that you realize your
information is useless before you take action based on it.
The worst case occurs when you divert your association’s
resources to act on what you think is good information—and
the information turns out to be either useless or actually
misleading.
Even a simple member-needs survey has subtle but important
question-design points you must keep in mind. Most
question-design problems arise from ignoring one or both
of the two most basic question-design principles: You must
1) start with the end in mind and 2) eliminate all
ambiguity. Here bad and good examples of these principles
at work.
1. Start with the end in mind
When you
design a survey question, start by thinking about what you
want to do with the information you’re trying to gather.
Consider the following question:
Please indicate your company’s annual equipment
expenditures for the most recent fiscal year:
a. Less than $100,000
b. $100,000 to $500,000
c. $500,001 to $1 million
d. More than $1 million
As
written, this question may be fine as a basis for running
a segment report based upon annual equipment expenditures.
However, it doesn’t work if you want to be able to provide
average annual equipment expenditures as a selling point
for advertising or exhibit sales or as the basis for a
calculation (such as equipment expenditures per employee).
In that case, you need to ask for an actual expenditure
number in an open-ended question:
Please indicate your company’s annual equipment
expenditures for the most recent fiscal year:
$_______________
2. Eliminate all
ambiguity
To get a
sense of how unclear even a simple question can be,
consider these examples of two different ways to ask the
“same” question.
Version 1:
Please rank the value of the following five XYZ
Association products and services using a scale of 1 to 5:
____ Item 1
____ Item 2
____ Item 3
____ Item 4
____ Item 5
This question has
several problems, including:
a.
Does a ranking of 1 or 5 mean this is the most valuable?
b.
Assuming that a ranking of 1 means the item is the most
valuable:
b1. Does that mean that the item ranked 1 is
highly valuable? Not necessarily.
b2. Is there a large or small difference
between the item ranked 1 and the item ranked 2? There is
no way to know.
b3. Is there a large
or small difference between the item ranked 1 and the item
ranked 5? There is no way to know.
c.
Does low value mean that the item is not important—or that
it’s important but not well delivered?
d.
Is this question asking the value of the item to the
individual, the individual’s company, the industry, or the
profession? Again, not clear.
e.
Ranking questions are unnecessarily difficult to answer.
After reading down all items in the list, respondents must
then go back and compare each item to each of the others
rather than just reviewing the
value/importance/satisfaction of each item on its own.
This is especially hard with long lists of items that are
similarly valued.
The meaning of this question’s results will be ambiguous
and so essentially useless. Because you’ll come away with
no information on the importance of each item or how well
your association delivers it, the question gives you
nothing to act on.
Now consider an alternative.
Version 2:
Please rate the importance you place on the
following five XYZ Association products and services and
your satisfaction with them:
|
Product or Service |
Importance Rating:
NA –
Not Aware
1 –
Not Important at All
2 –
Somewhat Unimportant
3 –
Neutral
4 –
Somewhat Important
5 –
Critically Important |
Satisfaction Rating:
NA –
Not Aware
1 –
Completely Unsatisfied
2 –
Somewhat Unsatisfied
3 –
Neutral
4 –
Somewhat Satisfied
5 –
Extremely Satisfied |
|
Not
Important Critically
At All Important |
Completely Extremely
Unsatisfied Satisfied |
|
Item
1 |
NA 1 2 3 4 5 |
NA 1 2 3 4 5 |
|
Item
2 |
NA 1 2 3 4 5 |
NA 1 2 3 4 5 |
|
Item
3 |
NA 1 2 3 4 5 |
NA 1 2 3 4 5 |
|
Item
4 |
NA 1 2 3 4 5 |
NA 1 2 3 4 5 |
|
Item
5 |
NA 1 2 3 4 5 |
NA 1 2 3 4 5 |
Look back through the problems that come with Version 1
and you’ll see that this question avoids every issue it
raises.
Here is another example of an ambiguous question:
How do you find out about industry events?
a. Web
b. news media
c. advertisement
This question is ambiguous in several respects:
a.
The responses are not mutually exclusive, because both
news media and advertisements may be found on the Web. The
overlap creates ambiguity for your respondent and means
your aggregated results will also be unclear.
b.
The responses are not at all exhaustive. Other choices
might include word of mouth and e-mail.
c.
Respondents have no way to tell you if they find out
through multiple sources (which is especially important if
they’re interested in more than one type of industry
event).
d.
If it’s possible that respondents are part of more than
one industry, the question should define the industry
being referenced.
The following version of the same question will provide
much more actionable information:
How do you prefer to find out about events in the ABC
industry?
a. association Web site
b. event Web site
c. Web search engine/index
d. e-mail advertisement
e. newsletter
f. industry magazine
g. word of mouth
