Yesterday, BDN blogger Ethan Strimling released some results from a poll of potential Democratic primary voters in the Second Congressional District. I have a number of questions about the poll and the reporting, but the biggest problem by far is that Strimling has refused to say which candidate sponsored the poll.
The sponsorship of a poll is the most basic and probably most important thing to disclose when writing about a public opinion survey. It’s the first item on the list of minimum disclosure standards for both the American Association of Public Opinion Researchers (of which I’m a member) and the National Council on Public Polls.
The reason why is simple. Even if we set aside conscious or unconscious bias from a hired pollster towards their candidate, (and this poll was conducted by PPP, a Democratic pollster with a sterling reputation) polls can vary quite a bit just from statistical deviation. In this case, based on the margins of error, the top three candidates could easily be off by around 3% each. It could almost as easily have shown Mills ahead and Baldacci in third. Additionally, these numbers are for the 95% confidence level, which means that even the best pollster using perfect methodology has an additional 5% chance of being even farther off than the statistical margins.
Campaigns usually only release polls when they fit a narrative that they’re trying to encourage and may do so even for polls that they know are outliers. An analysis by Nate Silver found that internal polls (which are usually done by reputable pollsters) overstate the standing of the candidate they’re affiliated with by an average of six points, likely mostly because of this kind of selection bias.
In his post and in follow-up conversations on Facebook and Twitter, Strimling claims that he has examined the poll and has determined that it is accurate. This is both incorrect and missing the point. Unless he has personally and exactly surveyed every Democratic voter in the Second District, he doesn’t have the ability to judge a poll’s real accuracy, only its methods and likely basic statistical variability. That isn’t enough. Without knowing the sponsor, people who read his post can’t fully evaluate the poll for themselves and take into account the potential effect of selection bias in favor of the candidate who paid for it.
Even if basic disclosure practices had been followed, there would be other, major problems with the way this poll was released.
This early, in this kind of election, the horse race results aren’t very useful. Most people aren’t even aware there’s an election coming up and aren’t familiar with the candidates. The field will also likely change dramatically before the vote, as potential candidates choose whether or not to run. That’s why the most important numbers at this point are the favorability ratings and name recognition scores. If, for instance, Troy Jackson had a lower unfavorable rating and lower name recognition than Janet Mills, then he could have a much greater ability to expand his support and win the actual election than she does, despite the horse race results. Strimling apparently had access to the entire poll but chose not to share these important numbers.
Additionally, in his post, Strimling explains that the top three candidates are in a statistical dead heat in a “lead pack,” but he doesn’t use the same caution in his headline, implying there that Baldacci is ahead, Cain is in second and Mills is in third. This is misleading, and especially so for the many people who will see the headline on another website or on social media and not click through and read the entire post.
This isn’t the first time that Strimling has released a poll without following basic disclosure practices, nor the first time that I’ve taken issue with it. If he wants to be taken seriously on issues of public opinion, he should immediately disclose this poll’s sponsor, release the full results and change the misleading headline.
I’m also disappointed that the Bangor Daily News chose to summarize and promote Strimling’s post on their Facebook page without disclosure of the sponsor of the poll.
For more on this issue, and Strimling’s responses to my concerns, see our conversations on Facebook and Twitter.