Get Rid Of That Grid!

Advanis is at the forefront of 'mobile first' web surveys.

We automatically offer every participant in our research surveys an app-like survey experience that is designed to work on devices of all sizes.

Many of our features, such as auto-advance, right swipe, and large answer buttons, enhance the participant experience [87% of participants say our surveys are better than others they have taken, in a survey among SSI participants in 2015].

But one of these enhancements has met with some trepidation among researchers - Advanis has removed grid questions from its standard survey format.

As shown below, we have replaced traditional grid questions with a single item fly-in style question.  Fellow researchers have asked us if this method has resulted in different/skewed results, or more straight lining behaviour, compared to when the question was asked using a traditional grid.   Using our syndicated Net Promoter Score Index, we tested for these concerns, by comparing the results of participants seeing additudinal survey questions using either a grid or a fly-in approach.

Between November 2015 to March 2016 593 respondents completed the survey on desktop computers. Of those, 301 saw the new ‘mobile’ fly-in format while 292 saw a more traditional grid format. The groups were similar in other aspects (demographics, for example).

The data for 8 statements were analyzed to determine if question presentation impacted responses:

• For 6 of the 8 statements tested, there were no significant differences in the mean response.
• For 7 of 8 statements there was no difference in the top box or net top 3 box responses.
• There was no difference between the styles in the proportion of respondents flagged for straight lining behaviour.

What about my 18 item grid question?

Along with technology, mobile friendly survey design also requires us, as survey authors, to be mindful of the experience that we design for our participants. In many cases, a mindful review of the survey can shorten or eliminate large grid questions.

Recently, one of our clients asked us to review historical data for two classic grid questions (with 12 and 10 items in the grid). We were able to show that it is possible to ask only 5 or 6 items, while still maintaining consistent segmentation classifications. 

What's the bottom line? You can be confident that Advanis' mobile responsive survey experience offers a better participant experience, with high quality and consistent data results.

Lori Reiser is a Principal Consultant at Advanis. Her consulting engagements focus on ensuring that the consumer is heard at the strategic table - reducing risk in marketing decision making. 

Sarah Harmer is a Research Manager at Advanis, currently working on projects in the telecommunications and technology industries. Her responsibilities include survey programming and testing, data analysis, and report writing. Prior to joining Advanis, Sarah worked in research positions with Correctional Services Canada, and the Carleton University Survey Centre. .