Elena, a researcher at an app company, sends a client the cleaned data from her latest study with speeders, straight-liners and failed attention checks removed, and still wonders how many of the respondents who stayed were real. Is data that passed those checks clean, or does it only look clean? Often it only looks clean, because those checks were built to catch careless people, while fake responses from careful automated respondents pass them. In a 2025 PNAS study, an AI agent passed 99.8% of standard attention checks, so clean data now needs checks before, during and after the survey.
This article explains that one step at a time. We look at what survey fraud research reports about how much online survey traffic is suspect, why attention checks catch careless people but not careful respondents that aren't human, and why survey data quality therefore needs several layers instead of one. We finish with three things you can change before your next fieldwork and an honest account of what no single signal can do, including the one we build.
How much survey data is fraudulent?
Nobody can tell Elena the share for her own study, but the published figures are high enough to take seriously. Rep Data, a survey-quality vendor, reports that it flagged between 21% and 38% of respondents from major panel sources as suspicious or fraudulent in its 2025 data. In a separate study of 1,928 respondents announced in June 2025, the same company found that 31% of responses were fraudulent.
Those numbers deserve a caveat, and the top of that range is best read as a vendor's upper bound rather than a typical rate. They come from a company that sells fraud prevention, they depend on how that company defines "suspicious", and they describe panel traffic rather than, say, a customer list you invited yourself. What they do show is that a meaningful share of what arrives through open online sampling may not be what it seems, which is reason enough to ask how your own data was screened.
Why don't attention checks catch fake responses?
If a share of that traffic is suspect, you'd hope Elena's attention checks caught it, but attention checks were designed to catch careless humans, which is exactly why they struggle against careful respondents that aren't human. A typical check hides an instruction inside a question, such as "select 'strongly disagree' for this row", and a person skimming to finish fast misses it. The check works because carelessness is the thing it measures.
An AI agent that reads every instruction doesn't miss it. In a study published in PNAS in November 2025, Westwood reports that an AI agent passed 99.8% of 6,000 standard attention checks. So the same check that removes a distracted student rushing through a survey for course credit keeps an automated respondent that followed every instruction perfectly. The check still catches carelessness, as it always did, but carelessness is no longer the whole risk.
Why does passing your checks prove less than it used to?
So passing your quality checks proves less than it once did, because the checks look for carelessness while the newer problem is answers that are careful and not human. A careful automated respondent finishes in a believable time, never straight-lines, gives consistent answers to related questions and writes open answers that fit the question. A cleaning checklist rewards every one of those traits, so this kind of response passes your checks and can even look like your best data.
That is why quality needs more than one layer, with each layer looking at something different. Checks before anyone answers decide who can get in, such as one response per person, a CAPTCHA, or a sample you recruited yourself. Checks during the survey look at how answers were given, such as how long each answer took and whether a choice had to be traded against another. Checks after fieldwork look at what was said, such as open answers that don't fit their question or the same pattern repeated across respondents. Each layer catches something the others miss, and none of them catches everything.
| Layer | What it looks at | Examples |
|---|---|---|
| Before anyone answers | Who can get in, and how often | One response per person, a CAPTCHA, a sample you recruited yourself |
| During the survey | How each answer was given | Time per answer, choices that trade one option against another |
| After fieldwork | What was said | Open answers that don't fit their question, patterns repeated across respondents |
What should you change before your next fieldwork?
Since no single check settles the question, the three changes below spread your checks across the whole survey instead of piling them up at the end.
- Layer checks before and during the survey, not only after it. Decide who can answer and how often before fieldwork starts, through duplicate rules, a CAPTCHA or an invited sample, and include some questions that record how they were answered. Cleaning afterwards can only remove what the earlier layers let in.
- Look at how long each answer took, not only the total time. A total duration can look reasonable even when one section was rushed and another was left open in a tab. If your tool records time per page or per question, compare it with the length of each question, and if it doesn't, split the survey into more pages so you get that detail.
- Keep the data you removed and note why. Store removed responses in a separate sheet with the rule that removed each one. When a client or reviewer asks how you cleaned the data, you can show the rules and the counts, and you can rerun the analysis if a rule turns out to be too strict.
What can't a single quality signal do?
Even with layers in place, every quality signal measures one thing, which is why none of them can vouch for a dataset on its own. A CAPTCHA tells you a challenge was passed, a duplicate rule tells you the same browser or email didn't answer twice, and an attention check tells you an instruction was followed.
The same limit applies to what we build. PingPolls times each this-or-that pick and turns that time into a confidence signal called Certiscore™, so a quick, sure pick counts for more than one made after a long pause. That tells you how sure an answer looked, and it can't tell you whether a person or a program gave it, so we treat it as one layer next to your own checks and never as a bot detector (we're the new name next to tools researchers have trusted for years, so we'd rather say that here than in a footnote).
Frequently asked questions
How much survey data is fraudulent?
It depends on where respondents come from and how fraud is defined. Rep Data, a survey-quality vendor, flagged between 21% and 38% of respondents from major panel sources as suspicious or fraudulent in 2025. Read the top of that range as a vendor's upper bound, treat open panel traffic with care and measure your own sample rather than assuming a figure.
Can AI bots take online surveys?
Yes. A PNAS study published in November 2025 reports that an AI agent passed 99.8% of 6,000 standard attention checks, which means an automated respondent can follow instructions as well as an attentive person does.
Do attention checks still work?
They still catch careless people, which is what they were designed to do. They are much weaker against automated respondents that read every instruction, so use them as one layer next to entry checks, timing and a review of open answers.
What is the difference between careless and fraudulent responses?
A careless response comes from a real person who rushed, skimmed or straight-lined a grid, while a fraudulent one comes from someone or something answering under false pretenses, such as a duplicate account or an automated agent. Attention checks and speed checks target the first kind, so the second kind needs entry checks and other layers.
