How it works
AI detector accuracy, explained: what 95% really means
Balanced accuracy, false alarms, misses and base rates in plain English, with our own measured numbers as examples, so you can judge any AI detector's claims.
Video Verifier team · · 5 min read
When an AI detector says it's "95% accurate", that doesn't mean your result has a 95% chance of being right. It's an average over a test set, and what it leaves out matters more than what it shows: which mistakes the detector makes, on which kinds of video, and how common fakes are in whatever you're checking. Here's how to read the number, with our own results as the worked example.
There are two ways to be wrong
A detector can make two quite different mistakes:
- A false alarm: a real video called AI. This hurts creators, witnesses and anyone whose genuine footage gets dismissed as fake.
- A miss: an AI video called real. This is the dangerous one for viewers, because a fake gets a stamp of approval.
A single "accuracy" figure mixes the two together. You need to know both.
Why plain accuracy can mean nothing
Imagine a test set of 1,000 videos in which 950 are real and 50 are AI. A "detector" that answers "real" to everything, without looking at a single frame, scores 95% accuracy. It hasn't caught one fake.
That's why careful testers report balanced accuracy: the average of the share of AI videos caught and the share of real videos passed. The lazy detector above scores 100% on real and 0% on AI, so its balanced accuracy is 50%, which is what it deserves. Our video model's balanced accuracy is 95.3% on 11,952 held-out test videos: 2,988 clips it never trained on, each tested as published, as a WhatsApp-quality copy, at 360p and heavily compressed.
Base rates: the part most people miss
Even a good detector's "AI" flags mean different things depending on how many fakes are in the pile. Here's an example with made-up round numbers (not our results), for a detector that catches 95 of every 100 AI videos and wrongly flags 5 of every 100 real ones:
- An ordinary feed where 1 video in 20 is AI. Out of 1,000 videos, 50 are AI and 950 are real. The detector flags about 48 of the AI videos, and about 48 real ones by mistake. Roughly half of its "AI" flags are wrong.
- A pile of suspicious viral clips where half are AI. Out of 100 videos, it flags about 48 of the 50 AI ones and only 2 or 3 real ones. Nearly every flag is right.
Same detector, very different meaning. Two lessons follow. A detector's answer is strongest when it agrees with other evidence about the clip, such as who posted it first and whether anyone can confirm it. And when most of what you check is real, a low false-alarm rate matters as much as catching fakes.
Confidence matters as much as the label
Most detectors produce a score, and a score near the middle means "unsure". An honest result says so. Ours answers "Most likely real" only when our video model is confident; otherwise you'll see "Leaning AI, not conclusive", "Leaning real, not conclusive" or "Too close to call".
That's why every AI tool on our accuracy page gets two numbers:
- Flagged as AI: how often the model leaned AI on that tool's clips.
- Called real by mistake: how often it was confident enough to back a "Most likely real" answer, the miss that matters most.
For OpenAI Sora, that's 97.5% flagged and 0.8% called real. For PixVerse, 98.1% and 0.2%.
Why per-tool numbers matter
An overall figure can hide a weak spot, and ours range widely:
- Kling: 96.8% flagged, 0.5% called real.
- Google Veo (Veo, 2, 3 and 3.1): 88.9% flagged, 2.3% called real.
- Luma Ray 2: 80.0% flagged, 6.2% called real.
If the clip in front of you came from Ray 2, the headline figure overstates how sure you should be. The same goes for real videos: everyday real footage was read as real 96.5% of the time, but polished stock footage only 55.4%.
Below: two correct results from our held-out test. In the first card, a real video of a river with a bridge in the distance, filmed into the sun and shared on Wikimedia Commons, read as 99% real. In the second, a kangaroo in a park made with Kling 1.5, scored 99% AI. An accuracy figure is thousands of results like these added together. It can't tell you which kind your clip is, or how hard it is.
Video is likely
Real
Video is likely
AI-made
Lab numbers and the real world
Test sets are tidy. Viral clips are edited, re-encoded and mixed with real footage. On AI videos we found "in the wild" on Wikimedia Commons, our model flagged only 58.3% and called 10.0% real. That's a small sample, but a fair warning.
Tests on tools a model has never seen matter for the same reason. In an independent test with two AI models outside our training data, 97% of 601 AI videos were flagged and about 1% (0.67%) were wrongly called real, while 96% of 84 real videos were read as real. Those numbers are a better guide to next month's new generator than any score on familiar tools.
Seven questions to ask about any accuracy claim
- Was it tested on videos the detector never trained on?
- Are misses and false alarms reported separately?
- Is the figure balanced, or could a lopsided test set flatter it?
- Are there results per AI tool?
- Was it tested on compressed copies, like WhatsApp forwards?
- Does it say where it's weak?
- Does it say "not sure" when it isn't sure?
If the answer to most of these is "we don't say", treat the headline number as marketing. Our guide to choosing a free AI video detector has more on this, and how we test our detector sets out our method in full.
See a result with its evidence
Numbers make more sense once you've seen a real result. Paste a link on the home page and look at which checks agreed and which didn't. For clips from the tool in the example above, see our Kling page.
Quick answers
Is 95% accuracy good for an AI detector?
It depends on how it was measured. 95% balanced accuracy on unseen, compressed videos is strong. 95% plain accuracy on a test set that's mostly real videos can mean very little.
What's worse, a false alarm or a miss?
Both cause harm. For someone deciding whether to believe a video, a confident miss is usually worse, which is why we count "called real by mistake" separately.