Thursday, 9 February 2012

Numbers, and why they matter.

Hat tip to Starbuck_WOI for this

I work in market research, pretty much what I do day to day is find stories in data and try and articulate that to the client. A surprising amount of what I end up saying to the client doesnt involve numbers, although I always have numbers in the back pocket in case they're needed. A mixture of trust in us as consultants and general lack of desire to see lots of numbers mean it's better to avoid too much data on a slide. That means when I read this:
Poll finds broad support for Obama’s counterterrorism policies
I want to know what data is actually saying that, because I'm looking at story, not the reality of the data.

So, are the public agreeing that Obama's counter terrorism policies are "right" or not?

Well, turns out that, from a sample of 1,000 (statistically robust with a margin of error of +/- 3% approx):
  • 42% strongly approve of the decision to keep Guantanmo open
  • 59% strongly approve of the use of drones to kill suspected terrorists
  • 56% strongly approve the draw down of troops in Afghanistan
Now, those numbers are strong, very strong, and here's why I believe thats happening. Its because people think, when the Government says 'suspected terrorists' what they are actually saying is 'they are terrorists, but we can't tell you why, because then we'd have to kill you'. And thats the language continued in 2 out of three of these questions is saying. So people are basically agreeing they're find with killing terrorists and putting them in jail, they're also absolutely A-OK with getting troops out of Afghanistan. Well, that makes more sense to me.

If you re-worded the question to something that reflected reality it might read a bit more like this:
Do you support the use of drones to kill civilians, who may at some point commit acts of violence against American troops or civilians
you might get a different answer. If you asked an emotive question (as the WP has done) it might read like this:
Do you support the use of drones to kill civilians, including first responders and those attending funerals, who may at some point commit acts of violence against American troops or civilians
Then you might be playing a whole different game.

Due to the low base size this statement is factually not correct:
Support for drone strikes against suspected terrorists stays high, dropping only somewhat when respondents are asked specifically about targeting American citizens living overseas, as was the case with Anwar al- Awlaki, the Yemeni American killed in September in a drone strike in northern Yemen.
The variance is 2%, meaning it falls well within the margin of error on the data. Its not totally wrong, its just directional. If you had another month or two's worth of data using the same question, you could assert it was rising or falling if the trend continues.

Interestingly the article cites audiences which I can't see in their sample, for example:
But fully 77 percent of liberal Democrats endorse the use of drones, meaning that Obama is unlikely to suffer any political consequences as a result of his policy in this election year.
So, 77% of liberal Democrats, who likely trust this Government highly, are in favour of killing people the Government has suggested are terrorists. That doesnt surprise me, thats normal.

I've done my share of so called "PR-able" research. It tends to be massaged at the very least. Journalists think in terms of a story, and have the story to some extent lined up, they then want data which demonstrates that story. Its always possible to then find that data, if you're willing to ask the right question. In this case, some very specific questions have been asked, which will elicit a very obvious set of responses, those have then been packaged up and presented as raw fact.

Be afraid of data, it is not your friend, particularly in the hands of a journalist.

No comments:

Post a Comment