> ## Content Index > Fetch the complete content index at: https://www.lumaris.com/llms.txt > Use this file to discover other available public pages before exploring further. # Survey methods interactive: More People ≠ Better Data - URL: https://www.lumaris.com/interactives/more-people-not-better-data/ - Published: 2026-08-12T16:26:03.000Z - Updated: 2026-08-19T20:09:57.000Z - Description: Opt-in surveys missed a government benchmark by 12 points. Probability-based sampling came within 3. An interactive explainer from Lumaris Research. - Author: Lumaris Research - Tags: Interactive, polling Understanding Survey Methodology # More People ≠ Better Data Quotas can't fix who shows up to take a survey. Lumaris Research surveys use a probability-based methodology. This produces significantly more accurate results than opt-in sampling, which has grown in popularity due to its speed and cheaper prices. This convenience comes at a cost. Explore what happens when this non-probability sampling method turns a known government statistic into a flawed answer. Scroll Visual demo summary: an opt-in survey sample clusters among adults with more free time and more time online, and estimates that 43% to 47% of adults worked for pay in the previous week — versus the 56% government benchmark. A probability-based sample spreads across all groups and estimates 51% to 55%. More free time Less free time More time online Less time online Dot graphic is illustrative. Shows how self-selection can skew a sample. Not based on survey estimates. Estimated % who worked for pay last week The real number: 56% Opt-in43–47% Source: Current Population Survey (BLS); Pew Research Center. The Setup ## Meet the population Start with a number we know. In mid-2021, **56% of U.S. adults had worked for pay in the previous week** — a benchmark from the government's Current Population Survey (CPS). In the grid, each dot is a group of adults — 100 dots, one for every 1% of the population. They're sorted along **two dimensions** that matter for who ends up in a survey. The Setup ## Reading the grid **Left to right:** free time. **Top to bottom:** time spent online. The **upper left** contains people with the most free time and the most internet use. The **lower right** contains those with the least of both. The Shortcut ## Opt-in sampling pulls from the same crowd An opt-in survey recruits people who *signed up to take surveys* — through ads, loyalty programs, or panel websites. It's fast and inexpensive, and fine for some purposes. Representing the whole population isn't one of them. In a 2023 report, Pew Research Center put this method to the test: three opt-in samples and three probability-based panels, all asked the same questions, all checked against government benchmarks. Who shows up when a sample selects itself? People with **more free time who spend more of it online**. This can include retirees, people between jobs, and frequent survey-takers. Notice how the highlighted dots **cluster in the upper-left quadrant**. The Consequence ## A skewed answer Ask this sample, *"Did you work for pay last week?"* and here's what comes back: 43–47% workedThe real number is 56%. All three opt-in samples came in low — off by 12 points on average. The sample overrepresents people who aren't working. The Catch ## Quotas were already applied These opt-in samples already included **standard demographic corrections**, meeting quotas for age, gender, race and ethnicity, and education. **The 12-point miss happened after meeting quotas, not before.** Quotas can only reshuffle the people who showed up. They can't fix the ways opt-in respondents **systematically differ** from everyone else. The Quality Problem ## Quotas can't fix bad data There's a second problem no amount of weighting can solve. The study found that roughly **8% of opt-in respondents** were what researchers called **“bogus respondents”** — answering inattentively or dishonestly, often saying yes to everything. In the grid, they're the **hollow dots**. On the probability-based panels: **1–2%**. More on this: [AI can now fake survey responses](https://www.lumaris.com/ai-fake-survey-responses/). **The lesson:** you can reweight a sample to match the population. You can't reweight your way out of made-up answers. The Method ## Probability sampling selects carefully In a probability-based panel, households are *randomly selected by mail* — every adult has a known chance of being invited, whether or not they spend time online. **The result:** A sample that mirrors what your population actually looks like. Notice the selected dots — **spread across all four quadrants**. That's the difference. The Payoff ## A more accurate answer Now ask this sample, *"Did you work for pay last week?"* and you get: 51–55% workedEvery probability panel in the study landed closer to the benchmark than any opt-in sample — about 3 points off, on average. The surveys used standard weighting and no quotas for employment, free time, or internet use. Random selection covered them all, within a known margin of error. Why This Counts ## A 12-point employment bias doesn't stop at employment. If a survey undercounts working adults, it doesn't just get employment wrong. It distorts anything correlated with employment — which is much of what surveys measure. ### Economic policy & sentiment Workers and non-workers experience the economy differently and hold different views on minimum wage, paid leave, childcare, and tax policy. A skewed sample misrepresents how constituents feel about the issues legislators are voting on. ### Financial behavior & consumer research Working adults have different spending patterns, savings rates, insurance coverage, and housing situations. If you're sizing a market or planning a program, a sample tilted 12 points toward non-workers distorts the picture of household finances. ### Health & well-being Employment is tightly linked to insurance status, access to care, stress, and mental health outcomes. Research informing healthcare policy or program design needs to accurately reflect the working population, not oversample people with more free time. The Pattern ## And employment is just one benchmark. On every opt-in sample tested, 10 of the study's 28 benchmarks came back with consistently large errors — and error topped 5 points on as many as 17\. This is because people who sign up for online surveys differ from everyone else in overlapping ways that no quota list can fully anticipate. Average error across the three opt-in samples, in percentage points, versus the government benchmark. You'd need quotas for every one of these dimensions, and their interactions, to eliminate the bias. **A probability sample addresses all of them at once.** The Caveat Probability panels have a documented weakness too. In this same study, all three overestimated turnout in the 2020 election by 8 to 9 points — people who join survey panels are more civically engaged than average — and turnout was the one benchmark where the opt-in samples performed well. Election polling has also improved since then: in 2024, pre-election polls were more accurate than in 2016 or 2020, and no single sampling method guaranteed accuracy. **Rigor doesn't mean perfection. It means knowing exactly where your errors are.** ## The accuracy gap is real and measurable Across 28 benchmarked variables, probability-based panels were consistently more accurate than opt-in samples. On average, they were twice as accurate. \~2.6 points Avg. error, probability panels \~5.8 points Avg. error, opt-in samples Need survey data you can stake decisions on? Lumaris Research operates Minnesota's only publicly available probability-based survey panel. [Talk to us about your research](https://lumar.is/meet?ref=lumaris.com) Or get Minnesota opinion data in your inbox — [subscribe to The Briefing](https://www.lumaris.com/#/portal/signup) Lumaris ResearchLumaris Research horizontal lockup — ember convergence mark and wordmark, for dark backgrounds. Employment benchmark from the Current Population Survey (BLS). Accuracy comparison, bogus-respondent shares, benchmark errors, and turnout finding from [Pew Research Center, “Comparing Two Types of Online Survey Samples,” Sept. 2023](https://www.pewresearch.org/methods/2023/09/07/comparing-two-types-of-online-survey-samples/?ref=lumaris.com)(surveys fielded June–July 2021). Election polling accuracy from the [AAPOR Task Force on 2024 Pre-Election Polling](https://aapor.org/wp-content/uploads/2025/10/AAPOR-Task-Force-on-2024-Pre-Election-Polling%5FReport.pdf?ref=lumaris.com) (2025).