More People ≠
Better Data
Quotas can't fix who shows up to take a survey.
Lumaris Research surveys use a probability-based methodology. This produces significantly more accurate results than opt-in sampling, which has grown in popularity due to its speed and cheaper prices.
This convenience comes at a cost. Explore what happens when this non-probability sampling method turns a known government statistic into a flawed answer.
Visual demo summary: an opt-in survey sample clusters among adults with more free time and more time online, and estimates that 43% to 47% of adults worked for pay in the previous week — versus the 56% government benchmark. A probability-based sample spreads across all groups and estimates 51% to 55%.
Meet the population
Start with a number we know. In mid-2021, 56% of U.S. adults had worked for pay in the previous week — a benchmark from the government's Current Population Survey (CPS).
In the grid, each dot is a group of adults — 100 dots, one for every 1% of the population. They're sorted along two dimensions that matter for who ends up in a survey.
Reading the grid
Left to right: free time.
Top to bottom: time spent online.
The upper left contains people with the most free time and the most internet use. The lower right contains those with the least of both.
Opt-in sampling pulls
from the same crowd
An opt-in survey recruits people who signed up to take surveys — through ads, loyalty programs, or panel websites. It's fast and inexpensive, and fine for some purposes. Representing the whole population isn't one of them.
In a 2023 report, Pew Research Center put this method to the test: three opt-in samples and three probability-based panels, all asked the same questions, all checked against government benchmarks.
Who shows up when a sample selects itself? People with more free time who spend more of it online. This can include retirees, people between jobs, and frequent survey-takers.
Notice how the highlighted dots cluster in the upper-left quadrant.
A skewed answer
Ask this sample, "Did you work for pay last week?" and here's what comes back:
Quotas were
already applied
These opt-in samples already included standard demographic corrections, meeting quotas for age, gender, race and ethnicity, and education.
The 12-point miss happened after meeting quotas, not before.
Quotas can only reshuffle the people who showed up. They can't fix the ways opt-in respondents systematically differ from everyone else.
Quotas can't fix
bad data
There's a second problem no amount of weighting can solve.
The study found that roughly 8% of opt-in respondents were what researchers called “bogus respondents” — answering inattentively or dishonestly, often saying yes to everything. In the grid, they're the hollow dots.
On the probability-based panels: 1–2%.
More on this: AI can now fake survey responses.
The lesson: you can reweight a sample to match the population. You can't reweight your way out of made-up answers.
Probability sampling
selects carefully
In a probability-based panel, households are randomly selected by mail — every adult has a known chance of being invited, whether or not they spend time online.
The result: A sample that mirrors what your population actually looks like.
Notice the selected dots — spread across all four quadrants. That's the difference.
A more accurate answer
Now ask this sample, "Did you work for pay last week?" and you get:
A 12-point employment bias
doesn't stop at employment.
If a survey undercounts working adults, it doesn't just get employment wrong. It distorts anything correlated with employment — which is much of what surveys measure.
Economic policy & sentiment
Workers and non-workers experience the economy differently and hold different views on minimum wage, paid leave, childcare, and tax policy.
A skewed sample misrepresents how constituents feel about the issues legislators are voting on.
Financial behavior & consumer research
Working adults have different spending patterns, savings rates, insurance coverage, and housing situations.
If you're sizing a market or planning a program, a sample tilted 12 points toward non-workers distorts the picture of household finances.
Health & well-being
Employment is tightly linked to insurance status, access to care, stress, and mental health outcomes.
Research informing healthcare policy or program design needs to accurately reflect the working population, not oversample people with more free time.
And employment is just one benchmark.
On every opt-in sample tested, 10 of the study's 28 benchmarks came back with consistently large errors — and error topped 5 points on as many as 17. This is because people who sign up for online surveys differ from everyone else in overlapping ways that no quota list can fully anticipate.
Average error across the three opt-in samples, in percentage points, versus the government benchmark.
You'd need quotas for every one of these dimensions — and their interactions — to eliminate the bias.
A probability sample addresses all of them at once.
Probability panels have a documented weakness too. In this same study, all three overestimated turnout in the 2020 election by 8 to 9 points — people who join survey panels are more civically engaged than average — and turnout was the one benchmark where the opt-in samples performed well. Election polling has also improved since then: in 2024, pre-election polls were more accurate than in 2016 or 2020, and no single sampling method guaranteed accuracy. Rigor doesn't mean perfection. It means knowing exactly where your errors are.
The accuracy gap is real and measurable
Across 28 benchmarked variables, probability-based panels were consistently more accurate than opt-in samples. On average, they were twice as accurate.
Need survey data you can stake decisions on? Lumaris Research operates Minnesota's only publicly available probability-based survey panel.
Talk to us about your research
Comments ()