Matt Yglesias’s noticed quick turn around polls showed big bounces for both Kerry and Bush during their respective parties’ conventions. However, later polls showed significant erosions. He highlights the most recent eroding bounce:
Matt wonders about people’s psychology, but the phenomenon may have nothing to do with psychology. I suspect the quick turn around bounce may be due to slightly greater numbers of Democrats staying home during the DNC convention and answering the phone, resulting in a pro-Kerry sampling bias. Later on, Republicans stay home to watch the RNC convention resulting in a pro-Bush bias. On the weekend, phone answering rates go back to normal, resulting in the ordinary “people who answer the phone” bias. This erases the effect of the sampling bias that would occur during a convention.
All three would be examples of the type of sampling bias that is not accounted for in the sampling error reported for polls in news reports.
The issue is also discussed at Pandagon.
I’ve always been curious about August/Labor Day polls in general. Are there party differences between who goes away on vacation when? Where I work, at least, the higher-ups, who are most wealthy and most likely to vote Republican, are always on vacation the last two weeks of August. The lower-downs, who are more likely to be Democrats, take their vacations earlier in the month. I don’t know who’s around the week of Labor Day.
It seems to me that a lot of the weekly ebb and flow is probably somewhere correlated to when people take vacations.
I also think it’s probably strongly correlated to who’s got “Called ID” and “Call Blocking” on their phones. I know that I will never be part of a telephone poll, because I never answer unless it’s a person I know on my Caller-ID. That’s one family Democratic voters (and donors) who will be excluded from all extrapolations.
This is precisely my hypothesis for the weird poll results we’ve seen during and immediately after each of the conventions. The ‘likely voter’ screens appear to make this sampling volatility even worse.
Most commenters on the net seem to be taking the polls at face value, and I think that is a mistake. There is no way the public is swinging by 10 points in a matter of days without any major events interceding.
I’m curious as to how cell phone possession affects phone polls. While it seems nearly everyone has a cell phone, what are the demographics of those who don’t? My land line is constantly tied into my computer, resulting in a busy signal for callers. My cell phone is the number I give for all purposes otherwise. It is my “home number”. Many people I know are WiFi with no land line at all. Do pollsters take into account the degree to which, if any, this skews their results?
Generally, between 3 and 8 % of households are “cell-only” (no landline). Is generally assumed to be on the low side (often reported as 3-4, but perhaps as high as 8 percent).
That number is highly skewed by age, however. 10-14% of 18-45 year olds are are “cell only” and half of all college-aged students are “cell only.”
Assumedly, a poll would have to control for age in order to avoid this distortion.
The bounce isn’t eroding, it was a skewed poll.
“You saw the TIME and the Newsweek polls, I’m sure, that showed Bush with double-digit leads. We now know, folks – this is not speculation – those two surveys were over-sampled with Republicans. Those two surveys were over-sampled with Republicans. I think 7% to 8% larger sample of Republicans. Now, the pollsters for TIME and Newsweek are saying, “Well, we didn’t do it on purpose. It’s just who was home when we called. They were home because they were all excited about the Republican convention and so forth.” So there’s been some reworking of the numbers, and I think the TIME poll, if you properly weighed it, would show Bush with a six-point lead. The Newsweek poll would show Bush with a three-point lead. Rasmussen has a one point lead in his poll, but he’s weighting it in a different way. He’s throwing out a Saturday to give Bush a four-point lead. The Bushies think they have a four-point lead.”
Source:
Rush Limbaugh
Great site!
New Zealand Mountainbike
http://www.vorb.org.nz
Ahh! So, Rush gives the same reason and has data on who was home.
Dick Morris has an interesting take about the recent Time and Newsweek polls.
“How big is Bush’s lead? Don’t believe the surveys that show it in the 5- to 7-point range. Believe the surveys of Time and Newsweek, which show a lead in excess of 10 points.
“The difference is because pollsters disagree about whether or not to weight their results to keep constant the ratio of Republicans, Democrats and Independents in their sample. Some polling firms treat party affiliation as a demographic constant and, when they find that their sample has too many Republicans, they weight down each Republican interview and assign an extra weight to each Democratic response.
But other polling firms — and I — disagree. We feel that political party is not a demographic, like gender or race or age. If the survey finds more Republicans than usual, we think it’s because the country has become more Republican, so we treat the result as a indicator of national mood, not of statistical error. ”
Source
Any thoughts?
I think you have to be very, very cautious whenever “”correct” a poll result based on demographic or other factors. Even though biased sampling causes errors, that doesn’t mean you can easily correct for them!
First, if you correct, you should report the results before weighting, and then explain why you corrected and how you corrected for this biased sample.
Clearly, if 90% or phone answerers were retirees, we could all understand weighting to account for age and more or less figure out how to do it based on the US census. (Then, when you report the sampling error, you should calculate it using the real method instead the the simplified one all news reports stick at the bottom of their poll even when it makes no sense!)
Weighting for Republicans or Democrats is much more difficult. Even if Zell Miller thinks you are born with party affiliation –like birthmark– the fact is people change affiliations. And the fraction of Dems and Reps can change during the election cycle. Unless you have some really accurate method of knowing the fraction of Republicans and Democrats in the population as a whole, correcting can make things worse!
There is another problem with correcting. Let’s look at the case where 90% of phone answers turn out to be retired. Wouldn’t you sort of wonder whether there was some problem with your sampling method? Would you be confident the 10% non-retirees who answered the phone were typical of most non-retirees?