Comparing CATI vs. IVR-to-Web Results

Advanis has been conducting a self-sponsored COVID-19 survey among Canadians since the end of March 2020 using an IVR-to-Web approach. As part of our efforts to continually validate the reliability of the approaches we use to collect data that inform the decisions made by our clients, we conducted a research-on-research study where the same questions were administered via phone (CATI) and via the IVR-to-Web approach.

The surveys were conducted among adult Canadians between July 16 and 22, 2020. 819 completed surveys were collected via CATI (margin of error of ±3.4%, 19 times out of 20) and 4,677 completed surveys were collected via IVR-to-Web (margin of error of ±1.4%, 19 times out of 20). Results for both surveys were weighted by province, age, and gender to ensure that the final results were representative of Canada’s adult population.

Results for both methodologies were similar when it comes to most behavioural questions (e.g. activities undertaken in the past week and mask wearing). Differences we observed between the two methodologies were consistent with previous research when comparing data collected via phone vs. online (e.g., Greene, Speizer, and Wiitala 2008, Chang and Krosnick 2009):

1. Respondents being more likely to agree with statements or indicate that statements describe them well on CATI compared to Web, potentially due to acquiescence bias being more prevalent when administering a survey with a live interviewer compared to online. Examples include:

• Respondents are more likely to indicate that the following statements describe them well (% top 3 box on a 7-point scale):
  • I appreciate what the crisis has given me (like quiet, time with friends and family): 49% for CATI vs 43% for Web
  • If we social distance for a few weeks everything will get back to normal: 34% for CATI vs 22% for Web
  • This virus won’t affect me in any way: 27% for CATI vs 16% for Web
• Respondents are more likely to indicate that their household was affected by COVID-19 (% top 3 box on a 7-point scale): 57% for CATI vs 51% for Web

2. Social desirability bias is more prevalent among those completing the survey via phone compared to those completing online, leading to an overstatement of socially desirable behaviors and attitudes and an understatement of opinions and behaviors they fear could elicit disapproval from the telephone interviewer, especially for topics of a sensitive nature. Examples include:  

• Respondents are more likely to indicate that they are practicing social distancing (% top 3 box on a 7-point scale) for CATI at 87% vs 82% for Web
• Respondents are more likely to somewhat or strongly support mandatory mask wearing legislation for CATI at 77% vs 73% for Web
• Respondents are more likely to indicate that their mental health is good, very good, or excellent for CATI at 82% vs 78% for Web.
• Respondents are less likely to indicate that the COVID-19 pandemic has impacted the mental health of their household (% top 3 box on a 7-point scale) for CATI at 28% vs 40% for Web.
• Respondents are less likely to indicate that they have experienced several negative emotional effects for CATI compared to Web:
  • Change in the quality or the duration of your sleep: 32% for CATI vs 37% for Web
  • Feeling more irritable or having moments of anger that you did not have before: 32% for CATI vs 37% for Web
  • Loss of interest and pleasure in your day activities: 32% for CATI vs 37% for Web
  • Moments of great anxiety or panic: 26% for CATI vs 32% for Web

In conclusion, these results, consistent with studies conducted previously, indicate that there may be certain advantages to administering surveys online, in particular when the survey covers topics that are of a sensitive nature or otherwise more likely to be subject to social desirability. That’s not to say that online surveys will always provide advantages over telephone surveys. E.g. telephone surveys have been found to provide better access to certain segments of the population than probability-based Web surveys (Keeter 2015).

References

• Chang, Linchiat and Jon A. Krosnick (2009): “National Surveys via RDD Telephone Interviewing Versus the Internet” Public Opinion Quarterly 73: 641–678.
• Greene, Jessica, Howard Speizer, and Wyndy Wiitala (2008): “Telephone and Web: Mixed‐Mode Challenge” Health Services Research 43: 230–248.
• Keeter, Scott (2015): “Methods Can Matter: Where Web Surveys Produce Different Results than Phone Interviews” Pew Research Center,  www.pewresearch.org/fact-tank/2015/05/14/where-web-surveys-produce-diffe..., accessed August 7, 2020.