Lying About Statistics

statistics-1

Surveys on National Policy Issues—Asking Bad Questions Gives you Bad Information

AMERICANS ARE EVENLY SPLIT ON “GUN CONTROL.”

Quinnipiac University. June 24-30, 2014. N=1,446 registered voters nationwide. Margin of error ± 2.6.
“Do you support or oppose stricter gun control laws in the United States?”
Support Oppose Unsure
% % %
6/24-30/14 50 47 3
CNN/ORC Poll. Nov. 18-20, 2013. N=843 adults nationwide. Margin of error ± 3.5.
“Do you favor or oppose stricter gun control laws?”
Favor Oppose Unsure
% % %
ALL 49 50 1
Fox News Poll conducted by Anderson Robbins Research (D) and Shaw & Company Research (R). April 20-22, 2013. N=1,009 registered voters nationwide. Margin of error ± 3.
“Which is more important: protecting the constitutional right of citizens to own guns or protecting citizens from gun violence?” Options rotated
Protecting
right to
own guns
Protecting
citizens
Unsure
4/20-22/13 53 42 5
1/15-17/13 51 40 9

So, Americans are evenly split on the issue of more gun control.  A clear majority think that it is more important to protect the constitutional right to own guns than to protect citizens from gun violence. That is evidenced in the polls.  But, these questions are the wrong questions.  See what happens when you ask policy relevant questions rather than generic “philosophical” questions.

ABC News/Washington Post Poll. May 16-19, 2013. N=1,001 adults nationwide. Margin of error ± 3.5.
“As you may know, the Senate recently rejected a proposal to require background checks on people buying guns at gun shows and online. Do you think the Senate did the right thing or the wrong thing in rejecting expanded background checks?”
Right
thing
Wrong
thing
Unsure
5/16-19/13 29 67 4
Quinnipiac University. April 25-29, 2013. N=1,471 registered voters nationwide. Margin of error ± 2.6.
“Would you support or oppose a law requiring background checks on people buying guns at gun shows or online?”
Support Oppose Unsure
4/25-29/13 83 13 3

THE SAME HOLDS FOR PUBLIC OPINION ON THE NRA.

CBS News/New York Times Poll. April 24-28, 2013. N=965 adults nationwide. Margin of error ± 3.
“Is your opinion of the National Rifle Association favorable, not favorable, undecided, or haven’t you heard enough about the National Rifle Association yet to have an opinion?”
Favorable Not
favorable
Undecided Haven’t
heard
enough
Don’t know/
No answer
% % % % %
4/24-28/13 34 26 16 21 3
1/11-15/13 38 29 14 17 2

Why not ask?

How does knowing that the NRA opposes background checks for people buying guns at gun show or  online make you feel about the NRA—much more favorable, more favorable, less favorable, much less favorable, no change

 Pew Research Center. May 1-5, 2013. N=1,504 adults nationwide. Margin of error ± 2.9.

“Do you think the National Rifle Association has too much influence, too little influence or the right amount of influence over gun control laws in this country?”
Too much Too little Right
amount
Unsure/
Refused
5/1-5/13 39 18 35 9

Why not ask?

The NRA’s actions played a major role in the passage of concealed handgun legislation in Kansas that allows anyone to carry a concealed handgun without a license or any training.  Would you say that means the too much, too little, just the right amount of influence, or are you unsure about the amount of influence the NRA has on legislation?

Sure, you conflate attitudes about the NRA and the Kansas legislation. But that is exactly what you want to do.  You want to provide concrete examples of what an organization does, then ask about its influence.

AMERICANS ARE EVENLY SPLIT ON PRO-LIFE/PRO-CHOICE ISSUE

Gallup Poll. May 2-7, 2013. N=1,535 adults nationwide. Margin of error ± 3.
“With respect to the abortion issue, would you consider yourself to be pro-choice or pro-life?”
Pro-choice Pro-life Neither/
Mixed (vol.)
Don’t know
what terms
mean (vol.)
Unsure
% % % % %
5/2-7/13 45 48 3 3 2
12/27-30/12 48 44 3 4 1
CNN/ORC Poll. Aug. 22-23, 2012. N=1,055 adults nationwide. Margin of error ± 3.
“Now I am going to read some specific situations under which an abortion might be considered. For each one, please say whether you think abortion should be legal in that situation, or illegal. . . .”
Legal Illegal Depends
(vol.)
Unsure
% % % %
“When the woman’s life is endangered”
8/22-23/12 88 9 1 2
“When the woman’s physical health is endangered”
8/22-23/12 83 12 2 2
“When the pregnancy was caused by rape or incest”
ALL 83 14 1 3

LESSONS LEARNED:  

The general rule is that the broader the question becomes, the less meaningful you can consider the answer.  Pollsters, of course, know this.  They often follow up these philosophical or “hot-button” style questions with more nuanced queries.  But, commentators, of whatever political persuasion, choose those results that support their position.

Remember the old book, Lying With Statistics?  The important part of that title is the second word—With.  Maybe a better title would have been—Lying About Statistics.  The statistics are rarely wrong.  It is what one says about what those statistics mean that is always the question.

statics-2

Go to http://www.pollingreport.com to see polling results from a wide range of polls on a wide range of issues.  The poll data reported here comes from that site.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s