Lies, damn lies and statistics

Last night I was invited to examine the methodology of surveys conducted by the John Locke Foundation. Having been in the market research business for more than 20 years, I consider myself at least partially qualified for the task.

To those not familiar with the industry, there is as much art as science involved. For example, knowing how to ask questions to get unbiased responses is one aspect of the art – assuming of course that you actually want unbiased responses. This is especially challenging for researchers who come to their projects with ideological biases and agendas. The tobacco industry, for example, is widely known for its abuse and misuse of all kinds of research.

The science half of the equation is equally fraught with challenges. For example, in today’s busy world, it’s getting harder and much more expensive to do research well. Telephone response rates have plummeted in recent years as fewer people are willing to even pick up the phone, let alone sit through a survey that can take 30 or 40 minutes. As a result, many research firms simply don’t do the number of call-backs necessary to obtain representative samples. Nor do they build internal reliability into their surveys by getting at the same question in several different ways.

Which brings us to the survey research company that does polling for the John Locke Foundation. The company is called Tel Opinion Research.

In 1997, a loose-knit group of professionals agreed to serve as members of the Board of Directors of an organization called "The National Association of Republican Campaign Professionals." This organization allows us to share ideas about fundraising, survey research, media and other vital parts of campaigns. And we can come together in a social setting with people who share one common goal of electing more Republicans at all levels.

And then there’s the president of the company, who has this to say about himself:

Bill Lee has been professionally involved in political efforts and campaigns for more than twenty-five years in more than half of the United States, Central America and Africa. He is an acknowledged expert in campaign planning and strategy and has taught these and other subjects in virtually every national conservative or Republican campaign school, for a variety of professional associations and as a guest lecturer at the Harvard Institute of Politics and American University. Bill Lee co-founded and serves as President of Tel Opinion Research, a political and commercial survey research firm. Lee has faced Bill Clinton in three gubernatorial contests, winning one, and is the only consultant to have ever defeated him.

The John Locke Foundation and Tel Opinion Research have every right to do whatever they want, of course. That’s the great thing about America. But any policymaker who comes across this kind of polling should take it for what it is. Survey research done on the cheap by proudly partisan hacks.

A

PS Just to be clear, I am not saying this research does not reflect the real opinions of those surveyed. I'm just saying we have no way of knowing whether that's the case - and good reason to suspect ideological bias.

Comments

How to lie with statistics

The title of my posting is nothing I invented. It's a well-used phrase in the market research industry, with whole books written about it. You can find out more here.

That's why I try to give the statistics when

I'm giving my interpretation. Like when I quoted the decrease in small contributions for Robin Hayes. There could be many explanations and meanings for that data. I was just giving my interpretation.

This morning the Char Obs..made a blanket statement that polls were down for Republicans but some polls indicated that people weren't ready for Democrats to lead. What polls? That isn't what I've seen from Pew and other polls I've read online. Are they talking NC polls? Is this just one poll. Did they actually see the poll? I think making a blanket statement like that without referencing the poll to give others the opportunity to look it up themselves is irresponsible.



***************************
Vote Democratic! The ass you save may be your own.

They would have gotten an automatic 'F'

in J-school, right? Send those lazy editors an LTE . . . or give me a link and I'll do it.

Now I can't find the durned article

I'm pretty sure it was this morning. They have a habit of renaming articles as the day goes on. I'll let you know when I find it. We are leaving for Raleigh in about an hour for the state history fair, so I'm in a bit of a rush...still have clothes to iron and pack.



***************************
Vote Democratic! The ass you save may be your own.

AHA! I found it!

I just had to remember what I was reading. It was the article about Howard Dean coming to raise money for the MCDP tonight. He did a literature drop and this was in response to that.

"Howard Dean can drop all the literature he wants, but it cannot make up for the fact that his leftist Democrat beliefs in gay marriage, abortion and taxes don't sit well with North Carolina voters," said Kevin Howell, a spokesman for the N.C. Republican Party.

Polls have shown that while some voters are dissatisfied with Republican leadership in Washington, they're not necessarily ready to give the reins to Democrats. Republicans have continued to make gains at the federal level, especially in the South

The article link can be found here.



***************************
Vote Democratic! The ass you save may be your own.

LTE sent.

Feel free to join the party, folks. Just hit the "Take Action" link on the left side of your screen and send a little note to the Charlotte observer.

Here' something to get you started:

Dear Editor:

Your story on Dr. Dean's arrival in Charlotte included this paragraph.

"Polls have shown that while some voters are dissatisfied with Republican leadership in Washington, they're not necessarily ready to give the reins to Democrats. Republicans have continued to make gains at the federal level, especially in the South."

And just which polls might those be? This kind of sloppy journalism would gain you an automatic F in any decent J-school in the country.

Go get 'em.