With Research This Bad, Who Needs Opponents?
[ Patrick Tuohey ]TonysKansasCity brings readers the latest salvo in the Kansas City mayoral race in the form of a survey conducted by Cambridge Consultants, LLC. Unfortunately, the study appears seriously flawed in execution, analysis and presentation.
In the opening paragraph, the study proclaims, "The results are confidential, and for the exclusive use of subscribers to the survey. Results will not be released to the news media." And with good reason. If Cambridge Consultants released the results, they would have been required, ethically, to release information about all the questions asked and the names of the people or groups who sponsored the study. Both these items are considered the standard for minimal disclosure according to the American Association of Public Opinion Researcher's (AAPOR) code of professional ethics.
Releasing the survey publicly would have also invited unwelcome scrutiny of the analysis. The errors in this study begin in the second paragraph of the first page when the report says, "The margin of error in this survey is 3.5%." That is wrong. The proper margin of error (assuming the standard confidence interval of 95%), for 407 completed surveys is +/- 4.85%--almost 1.5% off! I would not recommend putting much credence into a survey analysis prepared by someone who cannot properly perform the most basic research function.
This is a shame because the rest of the analysis requires faith in the analytical talents of the researcher as no raw data is provided. In addition, even the table at the bottom of page two is suspect. (See below.) Labeled "Funkhouser Tracking Question," the way it is presented suggests that data points are equally spaced over time. But examination of the timeline across the bottom shows this not the case. Three of the dates are from 2008, only one is from 2009. As a result, casual readers might judge the space between the first two data points--one month--as the same as those for the last two points--which is one year. Even if the data is reliable, the presentation is misleading.
A corrected version of this same chart appears below and suggests a subtle but important difference: the mayor suffered in 2007 and had a bad 2008, but his favorability/unfavorability numbers remain largely unchanged since January 2009. This could mean that he has been reduced to his hardcore supporter base or that he has had an uneventful year. What is clear is that is numbers are not as steadily on the decline as the above chart suggests.
Another frustration with this report is that no effort has been made to conduct an analysis based on crosstabulation. In other words, the author merely restates the topline results without breaking them into subgroups. For example, on page four the report states, "In geo-political terms, voters north of the river are likely to be divided in the primary, with Burke, Hermann and Funkhouser splitting a high percentage of the vote in that area." [Emphasis added.]
Yet the study measured primary vote and collected data by city region. The author shouldn't need to speculate about a primary vote north of the river--they have the answer in their dataset!
Pages seven through nine purport to show the questions asked in the survey, but it is incomplete because it does not include all the questions to which the previous pages refer.
While question ten above could be considered "message testing," it is so long and argumentative that the results are meaningless. No voter is ever presented information in this way. It is an example of trying to do too much in a single question.
Question 14 suffers the same flaw. It mixes "stop the nonsense" and "get our city back to work." These are two different messages. If this was an attempt at message testing, we don't know which of the two messages is responsible for the 74% who strongly agree.
Question 14 also is further evidence, I hope, that the entire survey questionnaire is not presented. If the study's purpose was to test several possible anti-city hall slogans, then the questionnaire included several such statements. Perhaps it also asked, "Do you agree or disagree that the city council needs to get serious about governing?" or, "Do you agree or disagree that the city council needs to focus on the basics?" Then the researchers would compare the answers and recommend the message ("stop the nonsense," "get serious about governing," or "focus on the basics") that got them the response they wanted.
The last piece of evidence that these questions were part of a larger message-testing document is that report itself tells us on page six, "Asking voters whether they approve or disapprove of the city council is like asking them if they approve of Congress." If they were not testing the "stop the nonsense" language, why did Cambridge Consultants take someone's money to collect data (question 14) that it admits was uninformative?
I hope for the sake of this study's client that the report published on TonysKansasCity was prepared with the intention of back-door public release. This would make it a political document--albeit a clumsy one--rather than a research document. If what these few pages provide is the extent of a research report presented to a client, they are owed a refund and an apology.
2/19/2010 1:47:15 PM
Submit a Comment - Sign in if you have a login or create an account to start.
- Recent News
- Table of Contents
- Overriding Priorities
The Missouri General Assembly ought to meet its Constitutional obligation to oversee federal money given to the state.
- Ethics Reform: A Return To Justice
Our legislatures must return to their primary function and our people must insist our legislatures protect the liberty of all individuals.
- Repealing the 17th Amendment
Since the founding of the nation, the states have been the consistent advocate of liberty within this nation. Their authority must be respected.
- Malfeasance as a Justification for Preemption
The federal government ought not use its inability to secure the border as a reason to keep the states from doing their part to enforce immigration laws.
- Missouri's Stake In The Energy Debate
Missouri will be affected the most by energy regulation because currently we have some of the cheapest electricity in the country.
- The Negative Consequences of Autism Insurance
Mandated autism coverage increases insurance costs for those with diseased not singled out by government.
- The Rise of the Patriot
A brief summary of human development and the rise of the American patriot.
- Ned Ryun: The Fight for the Founders
The fight for a more free America will take courage and will suffer losses, but the battle is worthwhile.
- The Coming Political Dissonance
A change is coming to Washington. From Missouri and elsewhere, a new class of politicians will be expected to dismantle the federal city and send its power back to the states.
- Ike Skelton: Do Ask, Do Tell
Why does Rep. Ike Skelton resist something that military leadership and even some conservatives have come to embrace?
- Need a Bridge?
Free-spending Republicans like Kit Bond are worse than Democrats, because they understand the principles of limited government and intentionally ignore them.
