The International Social Science Journal published a volume on Opinion Surveys in Developing Countries as far back as 1963. In it, Emily Jones wrote an elementary paper/article titled Courtesy Bias in South East Asian Surveys. Sadly, that still hasn’t been outdone much in the Indian context despite the time that’s elapsed. The basic problem can be paraphrased as, “surveying the underlings of a feudal society results in them telling the interviewer what they think the interviewer wants to hear.”
The response bias in this case is not unique to feudal societies, though the context is. An example that a well written paper on the subject considers is drug use, welfare receipts etc in a western society. However, the model suggested in it simply cannot be imported to the problem at hand.
Let’s explore the problem at hand.
Many surveys in India, including CSDS, use college students for the field work. As it happens, colleges across India happen to be over represented with Upper Caste students. The outcome of this isn’t easy to guess: most likely, an Upper Caste, male student is going end up asking many lower caste people whom they will be voting for. More likely, a UP Brahmin is likely to go to a Dalit in that state and ask a question on voting preference. The probability that the Dalit either doesn’t answer or lies outright appears non-negligible.
These conjectures are actually borne out by the following piece of evidence: every single Opinion Poll in Uttar Pradesh in the past 10 years has consistently under estimated the BSP. In 2004, a simple average of the BSP’s Opinion Poll results were 9 percentage points lower than their actual result. In 2009, it was much closer to the actual result. One wonders if this was because Ms Mayawati was the Chief Minister then and the fear/courtesy transformed into pride. But now, we are back to 2004 in terms of who’s in power in Lucknow. A piece of data that stands out in this regard from the CNN-IBN/CSDS Poll is only 6% of UP Muslims supporting the BSP while 13% claim to support the BJP.
The only reasonable solution to the problem is more and more polling data and a detailed study of this phenomenon to model the bias well enough. Perhaps some clever student can even come up with an adjustment factor. Until then, it can only mean polling data in India is to only be bench-marked against its own past results. And only trends mean anything and absolute numbers can safely be discounted.