This is the second of two posts.
Richard Kirsch, Director
Our Story – The Hub for American Narratives
This is the second of two posts on how to be a good consumer of polling. The first post talked about audience theory. This posts help you understand more about why the messages that poll the highest may not actually be the best.
5 – Small differences are often just that – small and insignificant
How often have you seen responses to different messages listed by rank order, with pollsters emphasizing the messages that poll the highest. A lot of the time, the difference is just one or two percentage points. But statistically, there is often little or no difference.
Most polls have a margin of error of between plus or minus 3% to 5%. Which means that if one question gets 65% and another 62%, the poll cannot say that there is a reliable difference between the preference of the general population. (Actually, since it’s plus or minus 3%, a difference of 6% points is not reliable.)
And when you get to smaller groups within a poll – for instance base voters or demographic groups – the point spread that is reliable is much bigger.
So when a pollster advises you to select one message because it polls higher – be sure it’s a real difference.
6 – Changes in wording but no changes in meaning
It is common to get different poll results by changing one word. For instance, one poll found that “improving wages” polled a little higher than “raising wages.” But they both polled well and the difference was not large. And as with any message, you should always check which groups it polled better with.
When it actually comes to choosing a word in your messaging, it may be awkward to use one term, instead of another. Unless you’ve got strong evidence that one is better than the other, you can use a number of words than convey the meeting in the actual messages you are using.
7 – Changes in wording with real changes in meaning.
On the other hand, there are times that word changes or phrases are meaningful or can help in change attitudes.
For example, when we describe inequality as a “gap” between the rich and poor, people are less likely to believe government can make a difference. But when we talk about “the economy being out of balance” between the rich and poor, people are more likely to think government can take effective action. The underlying reason is that a gap is natural while an imbalance can be fixed.
We actually don’t need poll results to inform us about many word choices. If we understand what a work means, including the images it creates, we can choose words that more closely convey our worldview.
Remember, one of the most important elements in making an idea popular is familiarity through repetition. So we should choose words that convey our wordview and repeat them over and over again; they will become more popular.
8 – “And” is often better than “or.”
Pollsters often creates false binaries. My favorite example is testing whether “working families” or “middle-class” is a more popular term. Both are popular but they convey different images and ideas. While a pollster is likely recommend using the one that polls higher, your most powerful messages should use both. Here’s an example: “Working families and the middle class are having a tougher time than ever making ends meet.”
Another way that pollsters create false binaries is in testing alternative ideas or frames behind a message. For example, there are two basic progressive arguments for raising the minimum wage. One is the moral argument – working people should earn enough to meet the basics. The other is the economic argument – working people will spend their higher wages in their communities.
When pollsters measure which message tests higher, it’s always the moral argument. But the moral argument doesn’t address the main opposition argument, that raising the minimum wage kills jobs.
Through an innovative polling project that I worked on with Topos Partnership, we found that the strongest message combined the moral and economic: When we raise the minimum wage so working people can meet the basics, their spending boosts the economy. That message was particularly important in the face of opposition attacks.
It’s almost always possible to easily combine message ideas in one more powerful message, rather than having to choose between two or more effective messages.
9 – Taking false comfort from positive polling
Most polls test your message against the opposition message. And in many cases, your message will come out on top. But don’t take too much comfort from that. In a real campaign, the size of the megaphone is more important than the message.
To find the most effective messages, you should stress test them, by simulating what happens if the opposition gets its message out early and often. Your goal should be to identify messages that do the best job under assault.
10 – Test and Tell Your Story – Not the Opposition’s
I’m ending with my golden rule for messages – don’t use a message that includes the opposition’s values and worldview. Making change requires changing worldview, winning the contest of values and ideas. And while you may pick up a few points in a message, we shoot ourselves in the foot when we repeat the opposition’s message.
Pollsters will often recommend using messages that include the opposition’s arguments as a way to respond to their messages. One theory is that by acknowledging the validity of opposition arguments you can more easily move people in the middle to support your counter position.
But you don’t have to do this to win. First, as we explained in point 2, repeating the opposition argument risks alienating your base.
Second, it’s always possible to write a popular message that makes your case based on your values and worldview. It may take some creativity and effort, but you can do it. If you’re having trouble, I’m glad to help!
It is possible that the message you come up with won’t test quite as well overall as another message. But if you’ve got this far in this blog, you’ll know why that might not really be true (point 5) and isn’t what matters anyway (points 2-4).
In conclusion, polling and other public opinion research can be a valuable tool for making change. But pollsters have their own biases and the field of polling has its institutional biases. The goal of polling should not be to find out what’s most popular. It should be to find out how to make our values and worldview popular. If you would like help navigating your work with pollsters, please let me know.