Recent OFCOM Research highlighted that 71% of the UK receive 9 nuisance calls a month, and that telephone research is the #4 culprit (1). So has telephone research had its day? At the same time online grows apace. We’ve looked closely at the merits of telephone, online and face-to-face (ftf). So if you commission quantitative research surveys, this article summarises some insights and ideas to help you make the most of your research investment.
Costs of quantitative research surveys
Quantitative research costs are sensitive to sample size, ease of reaching an audience or ‘incidence’, the length of survey, mode and complexity of fieldwork and analysis. Compared with online (index =100) fieldwork costs are typically higher for face-to-face (index 350-450) than telephone (index 250-300) due to the greater human time involved. Other costs such as coding for online research, computer aided telephone interviewing (CATI) and computer assisted personal interviewing (CAPI) are similar.
Coverage or reach
97% of the UK are online though many online surveys use panels which cover just 5% of the population. There are some geographic gaps and respondents are more ‘Internet experienced’. Thus some sample bias is possible. Nearly all homes have access to at least one phone though telephone databases cover just 60% UK homes (though even fewer will have agreed to take part in research!). Fixed line telephone reaches 79% homes (and proportionately more of the elderly) while mobiles reach 96% (and proportionately more of the young) (1). Face-to-face can reach most places (though with extra travel costs).
Online response depends on the nature of the panel, and how responsive and interested respondents are. Expect between 5-30%. Response from links on websites or emails will similarly depend on the nature of the source. Telephone responses have fallen over the last decade and responses are now around 10-15%. Face-to-face response is also around 15-20%.
Avidy bias (Sample bias)
The self-selection nature of online panels means there is a greater risk of respondents only participating in surveys that interest them. So-called avidy bias. Typically online respondents are younger, more familiar with the online world and spend more time on it. They are also more informed, more opinionated and more politically activist. (2) Panels also contain more early technology adopters though it remains possible to discern other types on the ‘diffusion of innovation’ spectrum.
Social desirability bias
Telephone research respondents present more socially desirable responses more often than face-to-face (3,4). This is particularly the case with those with lower intellectual ability or fewer years of formal education (i.e. C2DEs). Research has also shown that respondents are more comfortable discussing sensitive subjects face-to-face as they can see, and thus have greater trust in, the interviewer. Conversely, face-to-face interviews conducted in the respondent’s home minimises anonymity, making socially desirable responses more pronounced. Overall however, interpersonal trust between the interviewer and the respondent has a greater influence resulting in more honest responses. Face-to-face shows similar results to online (where there is no interviewer effect). However, some research (5) has observed higher valuation responses to some ‘willingness to pay’ questions (e.g. when there is a perceived ‘civic virtue’ in being seen to add to a common good).
Satisficing (a combination of the words satisfy and sacrifice) involves short-cutting the response process, settling on a solution that is ‘good enough’ but could be ‘optimised’.
Telephone poses an increased cognitive burden. It increases difficulty to comprehend questions, thus reducing the effort to cooperate, search the memory and process information. Perceived time pressure also fatigues and demotivates. This results in questions being less considered, giving rise to higher acquiescence (answering affirmatively regardless of the question), and reducing disclosure. Also choosing mid-points or only extremes in rating scales, easier to defend answers and having no opinions. This is most evident with those with lower intellectual ability. Research (3,4,5) suggests face-to-face researchers are better able to judge confusion, waning motivation, distraction (via watching a tv, eating etc.) and thus motivate and make it easier for the respondent to understand the questionnaire and improve cooperation on complex tasks. While online respondents go at their own pace.
1. Useful quantitative research surveys start with a clear market research brief. So decide your objectives, target market, what you need to know and any guidelines. Also beyond feasibility and answers to questions, what’s the relative importance of cost, speed, ‘reliability’ etc? Be clear too about expected sample sizes. All helps recommend the best market research service for your needs.
2. Understand the pitfalls in conducting quantitative research. Larger samples give greater reliability. Thus a sample over 1000 gives more reliability than a sample of 500, i.e. if repeating a survey 100 times, in 95 instances a confidence interval i.e. variance of responses will be within +/- 1%. So prefer shorter surveys to cut the risk of satisficing.
3. Make sure samples are not biased. Nationally representative samples are key to measure awareness, usage and market share. Anything else builds in bias and risks misleading. Thus ensure your sample removes any demographic, subject affinity, usage or other bias.
4. There are even more pitfalls in repeating a quantitative research survey or a brand tracker. So take extra care to make sure the pool of respondents delivers a sufficient and matched sample for each survey wave. Then findings will be comparable.
5. Beware spurious analysis. Remember the Whiskas advert that told us that ‘8 out of 10 cats prefer Whiskas’. This eventually changed to ‘8 out of 10 owners that expressed a preference said their cats preferred Whiskas’. However, what we still don’t know is how many said ‘don’t know’, how many expressed a preference, and the sample size. So in all quantitative research surveys, be clear about the sample size. Also what is statistically significant or merely directional to make the context clear. Ensuring clear and fair analysis gives more useful insights and thus leads to better decision-making!
1. OFCOM Telephone Nuisance Research (2014).
2. Duffy Bobby, Smith Kate, Terhanian George, Bremer John. Comparing Data from Online and Face-to-face Surveys. International Journal of Market Research Vol 47 Issue 6. (2005)
3. Holbrook Allyson L, Green Melanie C, Krosnick Jon A. Telephone versus Face-to-face interviewing of National Probability Samples with Long Questionnaires. Public Opinion Quarterly, Volume 67:79–125 (2003).
4. Szolnoki G, Hoffman D. Online, face-to-face and telephone surveys – Comparing different sampling methods in wine consumer research. Wine Economics and Policy 2 (2013) 57-66.
5. Lindhjema Henrik, Navrudb Ståle. Are Internet surveys an alternative to face-to-face interviews in contingent valuation? Ecological Economics 70(9): 1628-1637 (2011).