Thursday, 6 December 2012

Campaigns can be to blame

Excellent piece here by Nate Silver, pointing out that when internal polls are wrong, it's often the fault of the campaign itself:
Pollsters can expect to take their share of blame when their campaigns lose, and this year has been no exception. Not long after Barack Obama and Democrats had a strong night on Nov. 6, Republicans began to complain publicly that the polls conducted by their campaigns and by affiliated groups implied considerably more optimistic outcomes for them than actually occurred.
Perhaps these Republicans shouldn’t have been so surprised. When public polls conducted by independent organizations clash with the internal polls released by campaigns, the public polls usually prove more reliable.
Take, for example, the gubernatorial recall election in Wisconsin earlier this year. Independent polls had the Republican incumbent, Scott Walker, favored to retain his office by about six percentage points. A series of polls conducted for Democratic groups showed a roughly tied race instead.
Mr. Walker in fact won by seven points: the independent polls called the outcome almost exactly, while the internal polls were far from the mark.
Cases like these are fairly typical. My database of campaign polls released to the public in United States House races found that they were about six points more favorable to their candidate than independent surveys on average — and that they were typically less accurate in the end.
 It goes further to say:
But sometimes, internal polls make their way to the public through leaks that are not authorized by the campaigns. Or reporters and analysts may see the “real” numbers, or reasonably explicit characterizations of them, on background, on the condition that they not report them. (I saw some of the Obama campaign’s internal polling in 2008, along with that of some other Republican and Democratic campaigns in a few instances over the years.)

Perhaps these internal polls are more accurate?

My experience has been that these polls can also exaggerate the standing of their candidate, if perhaps not by quite as large a margin as those that are authorized for an on-the-record release. An interesting example of this comes from Noam Scheiber of The New Republic, who says he received data on Mr. Romney’s internal polling in six states from an aide to Mr. Romney’s campaign. In addition, Mr. Romney’s chief pollster, Neil Newhouse, disclosed the campaign’s polling to Mr. Scheiber in a seventh state, Ohio.

On average, the polls had Mr. Obama ahead by just one point between the seven states. They had Mr. Obama trailing in Colorado and New Hampshire and tied with Mr. Romney in Iowa.
The most interesting part though is about the mismatches which organisations generate willfully:
Our self-perceptions are very often more optimistic than the reality; 80 percent of people think they are above-average drivers, for example.

These problems can be worse when we join together to form businesses or organizations. Honest self-assessment is a challenge for any business, and it is one reason that management consultants are sometimes engaged at considerable expense to provide a supposedly more objective and unbiased take on the state of the organization’s operations. (Much of Mr. Romney’s success in business, of course, came precisely because he was able to identify companies whose organizational cultures prevented them from functioning efficiently.)

A pollster working within a campaign may face a variety of perverse incentives that compete with his ability to produce the most accurate possible results to his candidate. He may worry about harming the morale of the candidate or the campaign if he delivers bad news. Or he may be worried that the campaign will no longer be interested in his services if the candidate feels the race is hopeless.
Groupthink and confirmation bias are also risks in any organization, particularly under the stress of the end stages of a political campaign.
Allowing mismatches to exist without analysis is fundamentally unacceptable, particularly when the stakes are so high.

1 comment:

  1. This is a well-known bias in research - the tendency to confirm your already held beliefs by skewing the data in your favor or explaining it in some bafflingly positive way.