Methods and metrics

by Tom Ehrbar

Part 2 of a series on the survey process

First of all, when doing a survey we need to consider the “universe” of respondents whose views we’re interested in capturing. A full-blown survey of all relevant respondents would be a “census”; given the impracticality of that for most research, we use a representative “target population” instead. The challenge for survey work is narrowing the target population to a reasonable and statistically significant sample size. In order to do this, the sample selection must be (1) representative of the population; (2) selected randomly; and (3) large enough to produce accurate (non-biased) results. These are understood as follow:

(1) Representative= selected from the target population and no other; a sample that is too narrow will be biased.

(2) Random= anyone within the target population has an equal chance of being selected.

(3) Given 1 and 2, a larger sample size= more accurate.

When designing a survey, the first thing to consider is the confidence leveland margin of errorwe’d be happy with. A confidence level tells us how sure we can be that our results are accurate; the margin of error (MoE) shows the range these results would fall between if our confidence level held. Most statisticians use 95% as a standard confidence level, though different scientific domains may have higher or lower standards in this area.The standard acceptable margin of error is 3%–5%.

Let’s say we want to measure the satisfaction level among 1,200 subscribers to an email newsletter we publish weekly. By using this calculating table, we find that a sample size of 292 will give us a confidence level of 95% with a 5% margin of error. Our 95% confidence level means that 19 out of 20 times we choose to sample a randomly selected 292 subscribers our results would land within our margin of error. The MoE score means this sample is an accurate representation of our 1,200 subscribers to within 5 percentage points. If we asked our sample readers to rate their satisfaction with our newsletter on a 0–10 scale and their mean score was 8.5, we can be 95% certain that if we had surveyed all 1,200 readers our mean score would be between 8.0 and 9.0. If we wanted a narrower MoE, say ±3 percentage points, we’d need to sample at least 566 subscribers. Alternatively—and this is the most common way we use these calculations—we could access the calculator to find how big our sample size needs to be if we want to construct a random, representative survey of a known population size, say the US population of 330 million. We don’t have the time or money to conduct a true census, of course, but if our sample were truly representative and constructed randomly (see above), we would only need to gauge the responses of about 1,070 people altogether to be 95% confident of their responses at a ±3-point margin of error.This is what makes political polling reliable (when it produces bad results, it’s often because the sample was too narrow, as when pollsters were excluding cellphone users because they weren’t in their databases).

In order to get the best participation rates (which helps support an “unbiased” sample), our surveys are both anonymous and confidential, though we can ask respondents to voluntarily “opt in” with identifying details if they’re willing.

In the next post, I’ll go through this with an example from the real world—or at least our real world of fielding business surveys.

Tom Ehrbar is Senior Editor with the Thought Leadership group and manages much of its survey research.