How it works
How we test our AI video detector
Held-out clips, four copies of every video, a test on AI tools the model never saw, and published weak spots: exactly how we measure our detector, and its limits.
Video Verifier team · · 6 min read
We test our AI video detector on videos it has never seen, in the worse quality people actually check, and on AI tools it was never trained on. Then we publish every result, including the ones that don't flatter us. Here is the method, step by step, with the numbers from our October 2026 tests.
Rule 1: lock the test videos away before training starts
A detector tested on clips it trained on will look brilliant and then fail in the real world, because it has memorised the answers. So before our video model learned anything, we set aside 2,988 clips, real and AI, and never used them to train or tune it.
There's a subtler trap too. Many research collections contain a real video and an AI video made from the same description, say "a dog running on a beach". If one sits in training and the other in the test, a model can pass by recognising the scene instead of spotting AI. We keep such pairs on the same side of the split, so the only way to score well is to learn what AI video looks like over time.
Rule 2: test every clip four times
Nobody checks a pristine studio file. People check a forward that has passed through three family groups. So every held-out clip is tested in four versions:
- As published.
- As a WhatsApp-quality copy.
- Shrunk to 360p.
- Heavily compressed.
That makes 11,952 test videos. Across all of them, the model's balanced accuracy is 95.3%. Balanced accuracy is the average of how well it does on AI videos and on real ones, so a test pile with more of one kind can't flatter the score. Our explainer on what detector accuracy really means goes into why that matters.
Below: two clips from the held-out set. In the first card, a real aerial video of Niagara Falls from Wikimedia Commons, which our video model read as 99% real. In the second, a school of fish swirling underwater, made with OpenAI Sora, which it scored 99% AI. The model never saw either clip while it was being trained.
Video is likely
Real
Video is likely
AI-made
Rule 3: test on AI tools it has never met
A held-out test still uses the AI tools the model trained on, and new generators arrive every few months. So we also run an independent test with AI models that were outside the training data:
- NVIDIA Cosmos-Predict 2.5, a family the model never saw: 96.2% flagged as AI.
- Wan 2.2, a newer version of a tool whose earlier versions were in training: 98.2% flagged.
Across 601 AI videos from these two models, 97% were flagged, and about 1% (0.67%) were wrongly called real. We also tested 84 real videos from people whose uploads the model had never seen: 96% were read as real. Each clip was checked as published and as a WhatsApp-quality copy.
Rule 4: count the dangerous mistake separately
Every AI tool on our accuracy page gets two numbers, not one:
- Flagged as AI: how often the model leaned AI on that tool's clips.
- Called real by mistake: how often it was confident enough to back a "Most likely real" answer on an AI clip.
The second number matters most. A fake that comes back "not sure" still leaves you cautious. A fake that comes back "real" gives a false sense of safety. For OpenAI Sora, the figures are 97.5% flagged and 0.8% called real; for Kling, 96.8% and 0.5%.
The same thinking shapes the product. The site only answers "Most likely real" when our video model is confident, and a check that couldn't run is shown as unavailable instead of being quietly scored.
Rule 5: publish the weak spots
Leaving out the bad rows would be easy. We don't, because you can't judge a result without knowing where the tool struggles:
- Polished stock footage: only 55.4% of real stock clips (from Pexels) were read as real, against 96.5% of everyday real videos from Wikimedia Commons. Smooth, perfect footage looks AI-like to the model.
- The newest generators: Luma Ray 2 (80.0% flagged, 6.2% called real), Hailuo 02 (84.8% and 5.4%), Moonvalley Marey (87.9% and 6.8%) and Google Veo (88.9% and 2.3%).
- AI videos found "in the wild": edited, re-encoded clips on Wikimedia Commons, often mixed with other footage: 58.3% flagged and 10.0% called real. It's a small sample, and it's the hardest case for any detector.
Rule 6: use test data that's open to inspection
Every test clip comes from openly licensed sources: research collections such as VideoGen-Eval, DeepAction and Rapidata's public text-to-video sets, plus real videos shared on Wikimedia Commons. The independent test's AI clips come from another openly licensed research collection. That keeps the tests repeatable, and it lets us show you real frames from them, like the two above, with credit.
What these numbers don't tell you
Be wary of anyone, us included, who turns a test score into a promise. The honest limits:
- The figures are for our video model alone. A full scan also weighs an AI-pixel frame check, a voice check, visible watermarks, Content Credentials, metadata and, for links, the post's text and published fact-checks. A language model, Jev, weighs it all, with fixed rules that stop it calling anything real without solid evidence.
- Test videos can't cover every phone, every lighting condition or every kind of scene. Footage from your street may look different from anything in a public archive.
- Tomorrow's generators aren't in today's test. Until we measure a new tool, our numbers for it are unknown.
So a result is strong evidence, not a verdict. It reads "Most likely AI-generated", "Most likely real" or, when the evidence is split, "not conclusive", and it shows the checks behind it.
What we do with the misses
Every mistake in testing is a lead. Misses on new tools shape the next round of training, and confirmed fakes that keep circulating go into a known-fakes list, so re-uploads of them are recognised. If you want to know what the model actually measures, read how our motion model works; the other checks are covered in how AI video detectors work.
Run your own test
The best test is a video you care about. Paste a link or upload a clip on the home page: a Quick Scan is free to try, needs no sign-up and answers in under a minute for most videos. Or play the Real or AI quiz and see how your eye compares.
Quick answers
Does 95.3% mean my result is 95.3% likely to be right?
No. It's an average over thousands of test videos. How sure a single result can be depends on the clip: its quality, its length and which tool made it. That's why the report shows its evidence, not just a number.
Do you test on WhatsApp copies?
Yes. Every clip in both tests was checked as a WhatsApp-quality copy as well as the original, and the held-out test adds 360p and heavily compressed versions.
Why publish the weak results?
Because they change how you should read a result. A lean towards AI on a glossy stock-style clip deserves more caution than the same lean on a shaky phone video.