Recent OFCOM Research highlighted that 71% of the UK receive 9 nuisance calls a month, and that telephone research is the #4 culprit (1). So has telephone research had its day? At the same time online grows apace. We’ve looked closely at the merits of telephone, online and face-to-face (ftf). So if you commission quantitative research surveys, this article summarises some insights and ideas to help you make the most of your research investment.
Costs of quantitative research surveys
Quantitative research costs are sensitive to sample size, ease of reaching an audience or ‘incidence’, the length of survey, mode and complexity of fieldwork and analysis. Compared with online (index =100) fieldwork costs are typically higher for face-to-face (index 350-450) than telephone (index 250-300) due to the greater human time involved. Other costs such as coding for online research, computer aided telephone interviewing (CATI) and computer assisted personal interviewing (CAPI) are similar.
Coverage or reach
97% of the UK are online though many online surveys use panels which cover just 5% of the population. There are some geographic gaps and respondents are more ‘Internet experienced’. Thus some sample bias is possible. Nearly all homes have access to at least one phone though telephone databases cover just 60% UK homes (though even fewer will have agreed to take part in research!). Fixed line telephone reaches 79% homes (and proportionately more of the elderly) while mobiles reach 96% (and proportionately more of the young) (1). Face-to-face can reach most places (though with extra travel costs).
Online response depends on the nature of the panel, and how responsive and interested respondents are. Expect between 5-30%. Response from links on websites or emails will similarly depend on the nature of the source. Telephone responses have fallen over the last decade and responses are now around 10-15%. Face-to-face response is also around 15-20%.
Avidy bias (Sample bias)
The self-selection nature of online panels means there is a greater risk of respondents only participating in surveys that interest them. So-called avidy bias. Typically online respondents are younger, more familiar with the online world and spend more time on it. They are also more informed, more opinionated and more politically activist. (2) Panels also contain more early technology adopters though it remains possible to discern other types on the ‘diffusion of innovation’ spectrum.
Social desirability bias
Telephone research respondents present more socially desirable responses more often than face-to-face (3,4). This is particularly the case with those with lower intellectual ability or fewer years of formal education (i.e. C2DEs). Research has also shown that respondents are more comfortable discussing sensitive subjects face-to-face as they can see, and thus have greater trust in, the interviewer. Conversely, face-to-face interviews conducted in the respondent’s home minimises anonymity, making socially desirable responses more pronounced. Overall however, interpersonal trust between the interviewer and the respondent has a greater influence resulting in more honest responses. Face-to-face shows similar results to online (where there is no interviewer effect) though some research (5) has observed higher valuation responses to some ‘willingness to pay’ questions (for example, when there is a perceived ‘civic virtue’ in being seen to add to a common good).
Satisficing (a combination of the words satisfy and sacrifice) involves short-cutting the response process, settling on a solution that is ‘good enough’ but could be ‘optimised’.
Telephone poses an increased cognitive burden. The increased difficulty to comprehend questions, reduces the effort to cooperate, search the memory and process information. Perceived time pressure also fatigues and demotivates. This results in questions being less considered, giving rise to higher acquiescence (answering affirmatively regardless of the question), having no opinions, choosing mid-points or only extremes in rating scales, easier to defend answers and reduced disclosure. Again this is more clear with those with lower intellectual ability. Research (3,4,5) suggests face-to-face researchers are better able to judge confusion, waning motivation, distraction (via watching a tv, eating etc.) and be able motivate and make it easier for the respondent to understand the questionnaire and improve cooperation on complex tasks. With online respondents go at their own pace.
1. Useful quantitative research surveys start with a clear market research brief. Decide your objectives, target market, what you need to know and any guidance. Beyond feasibility and answers to questions, what’s the relative importance of cost, speed, ‘reliability’ etc? Be as clear as you can about expected sample sizes.
2. Understand the pitfalls in conducting quantitative research. Larger samples give greater reliability, for example a sample in excess of 1000 will give more reliability than a sample of 500, i.e. if repeating a survey 100 times, in 95 instances a confidence interval i.e. variance of responses will be within +/- 1%. Prefer shorter surveys to cut the risk of satisficing.
3. Make sure samples are not biased and give reliable findings. Nationally representative samples are key to measure awareness, usage and market share. Anything else builds in bias and risks misleading. Ensure your sample eliminates any demographic, subject affinity, usage or other bias.
4. There are even more pitfalls if you would like to repeat a quantitative research survey or set up a brand tracker. Take extra care to make sure the pool of respondents will deliver a sufficient and matched sample for each survey wave so findings are comparable.
5. Beware spurious analysis. Remember the Whiskas advert that famously told us that ‘8 out of 10 cats prefer Whiskas’. This eventually changed to ‘8 out of 10 owners that expressed a preference said their cats preferred Whiskas’. What we still don’t know is how many said ‘don’t know’, how many expressed a preference, and the sample size. Whatever quantitative research surveys you plan, be clear about the sample size, what is statistically significant or merely directional to make the context clear. Ensuring clear and fair analysis gives more useful insights and ensures better decision-making!
1. OFCOM Telephone Nuisance Research (2014).
2. Duffy Bobby, Smith Kate, Terhanian George, Bremer John. Comparing Data from Online and Face-to-face Surveys. International Journal of Market Research Vol 47 Issue 6. (2005)
3. Holbrook Allyson L, Green Melanie C, Krosnick Jon A. Telephone versus Face-to-face interviewing of National Probability Samples with Long Questionnaires. Public Opinion Quarterly, Volume 67:79–125 (2003).
4. Szolnoki G, Hoffman D. Online, face-to-face and telephone surveys – Comparing different sampling methods in wine consumer research. Wine Economics and Policy 2 (2013) 57-66.
5. Lindhjema Henrik, Navrudb Ståle. Are Internet surveys an alternative to face-to-face interviews in contingent valuation? Ecological Economics 70(9): 1628-1637 (2011).