Margin Of Error

Business questions are typically phrased like “Who would be interested in our new product?”, “How satisfied are our customers with our service?”, or “Who is likely to recommend our brand?” Survey research is often done to answer these questions and is typically reported out as showing the percent of people who fit into a particular category. Unless a census is completed, there will be some error in the results obtained. One very important question is, how large is this error?

There are four things that affect the margin of error:

• the level of confidence,
• the sample size,
• the score obtained, and
• the population size.

The level of confidence is correlated with the margin of error: the stronger the confidence you select, the larger the margin of error has to be. For example, if a survey determines that a metric’s score is 85% with a margin of error of 3% at 90% confidence, that means that if the survey were repeated a total of 100 times, 90 of the surveys would predict a value between 82% and 88% (the confidence interval) and 10 surveys would predict a value outside of this range. If we want to be 95% confident in our score rather than 90%, the margin of error has to increase so that only 5 surveys would have scores outside of the confidence interval.

Next, sample size affects the margin of error; the more people surveyed, the more reliable the results (i.e. the smaller the margin of error). To visualize this, consider the cases where 99 or 999 people have been surveyed and we survey 1 more person. When we go from 99 to 100 respondents, that last respondent has the potential to move the metric by up to 1%. However, the respondent who takes the sample size from 999 to 1,000 has the potential to move the metric by only 0.1%. This is why larger sample sizes yield more precise results.

You may be surprised to learn that the score itself will affect the margin of error. However, this isn’t all that unintuitive. If an agreement score is 95% (for example) and you survey one more person, there is a very good chance (95% in fact) that they will also agree. However, if the agreement score is only 50%, then the chance that the next person will agree is a toss-up. So, the further a score is from 50%, the higher our confidence in the value and the smaller the margin of error. Since it isn’t feasible to quote a margin of error on every number in a report, the standard practice is to state the maximum standard of error which would occur for metrics scoring 50%.

Finally, population size can have an impact as well. Often the assumption is made that the population size is infinite. While this is never actually the case, this assumption allows for simple statistical calculations but leads to otherwise larger than actual margin of errors. With an infinite population, you could never know the exact score of a metric so there must always be some margin of error no matter how many people you survey; the next person surveyed always has the potential to change your estimate. However, with a finite population, once you have surveyed everyone (i.e. done a census), you will know the exact value and the margin of error must therefore be 0%. So, how small is small? As the table below shows, if your total population is less than 10,000, you may want to consider asking your research provider to apply small population corrections to any stat testing.

To recap, in survey research a margin of error always exists unless a complete census is taken. The size of the error depends on the level of confidence, the sample size, the score obtained, and the population size. If your population size is less than 10,000, you should consider asking that small population size statistical corrections be applied to all statistics.

Patrick Kyba

is a Consultant with Advanis. Patrick plays a lead research role on projects having responsibility for survey development, survey programming and testing, development of analysis plans, data cleaning, weighting, and report writing and production. He strives to find the best reporting solution for his clients, from simple tables and graphs to complex automated online reporting or detailed custom reports.