Impact of Mobile Optimized Surveys: Young Adult Audiences

Mobile surveys matter. Over half of adults in the US and Canada own a smartphone, with nearly that many owning a tablet. Smartphone usage has increased exponentially in the past several years, to as high as 92% among college students. It is integral that survey experiences match the mobile habits of potential respondents to maximize response rates and overall data quality.

Survey fatigue is detrimental to survey data quality and response rates. So how do you ensure that valuable data can still be obtained in longer surveys? Promising results are emerging from studies of survey optimization for smartphones.

Optimization has shown to improve results even in longer surveys. Advanis partnered with clients to provide “proof in the pudding” to demonstrate how optimized surveys net a positive outcome. Here we present a few examples, specifically focusing on the student.

National Survey of Student Engagement (NSSE)

In 2015, the National Survey of Student Engagement (NSSE), Advanis, and the Indiana University Center for Survey Research worked together to examine the impact of smartphone optimization for long surveys. Comparisons were made between those students using surveys optimized for smartphones, unoptimized surveys, and desktop computer participation. The survey was 20 minutes in length. In total, 7,347 responses analyzed.

The study assessed the following indicators and respondent characteristics that affect data quality:

• Early Abandonment – Only 5% of first year students abandoned the optimized mobile survey compared to 12% of desktop respondents and 26% of unoptimized mobile survey respondents. Similarly, senior students had a 4% abandonment rate for optimized, 10% for desktop, and 22% for unoptimized.
• Missing Data – Optimized surveys reduced missing data for both seniors and first years. This optimization was even more meaningful to first year students.
• Duration – There was an 18% decrease in duration for optimized surveys compared to both the unoptimized and desktop users groups.
• Straight-lining – The practice of answering in such a way that conserves time and energy, yet produces a response that seems “good enough” was reduced in the optimized survey compared to both the unoptimized and desktop surveys.
• Subjective Evaluations – The optimized survey received the highest scores for both ease of use and visual design.

BC Stats Student Outcome Survey

Similar results of increased completion rates and decreased survey times were noted in the BC Stats Student Outcome Survey, which compared data between 2014 (unoptimized) and 2015 (optimized).

In two separate surveys of recent graduates conducted in 2015, smartphone participation rose 6.1% and 13.6% respectively and completion rates increased 4.6% and 10.9% for optimized mobile users. Completion time for mobile users also decreased between 1 and 2 minutes.

Design Matters

So… which survey design elements are conducive to increased response quality and quantity?

• Scrolling verses Paging – Although there are no significant differences for non-response, scrolling decreases survey duration. The result is an easier, more engaging survey experience for smartphone users. A slight auto-advance feature also improves survey experience and shortens overall survey length.
• Survey Legibility – Text size, ability to see all responses, navigation without zooming, and a general improvement in survey legibility delivers positive data quality results.
• Horizontal versus Vertical Answer Scales – Although there is no significant difference in missing items, primacy and recency effects are noticed when the responses were not all in view at the same time. Freezing the root question above the responses delivers positive results.
• Shortened Surveys – Although Ideal for reducing survey abandonment, shorter surveys are not always an option when collecting important information. In the case of the NSSE study, universities and colleges require a vast amount of survey data to properly evaluate and improve student experience.

These studies demonstrate that mobile optimization (in survey design and survey technology) and not just mobile ‘sizing’ is critical to survey participation and engagement.

Anu Bhalla is the Senior Vice President of Business Development with Advanis. Anu has spent the past 20 years developing and managing key accounts and client partnerships. Anu has worked across all sectors including Financial Services, Telecommunications, QSR, Retail, Energy and Mining, Technology, Government departments and agencies, as well as non-profit.