VGI-Bench: A Holistic Benchmark for Video General Intelligence

We introduce VGI-Bench (Video General Intelligence Bench), a holistic multimodal benchmark probing performance across twelve distinct visual and audio-visual skills. VGI-Bench comprises 550 human-curated questions designed to mitigate the common mistakes of today’s video benchmarks and to expose pragmatic challenges for modern state-of-the-art models. We evaluate a range of multimodal vision-language models and find that the best-performing model reaches 64.73% accuracy under our default protocol, against a 27% random-chance baseline, while humans reach 85.4%.

Vision-language models (VLMs) are today’s de-facto standard for visual reasoning tasks. They are widely applied in embodied systems such as humanoid robots[1] and self-driving cars,[2] and they serve as the visual backbones of computer-use[3] and design agents.[4] This renders the need for robust, holistic and, in particular, diagnostic benchmarks that let model developers see exactly where a model can be trusted and what remains difficult.

Where video benchmarks fall short

While existing benchmarks are available, most of them suffer from at least one of the following shortcomings:

• Language priors Questions do not require an actual understanding of the video or audio and can be solved with strong language priors alone.[5]
• Single-frame shortcuts Questions do not require understanding many frames in order; a single frame, or an unordered bag of frames, may suffice.[6]
• Training-data contamination The questions may have been included in the training data, inflating the results.[7]
• Setup artifacts The model’s design or input settings make it impossible for the model to answer, deflating the results in a non-informative way.[8]
• Unclear real-world meaning The interpretation of the results in terms of real-world applications remains unclear.[9]

The performance

We present results for a range of widely used vision-language models. We call the models with their default settings where possible, and where a long video does not fit the context window we pass a downsampled version of it that retains as many of the relevant frames as possible. All models receive the same neutral system prompt and answer in a multiple-choice format in which the number of distractors may vary. The baseline random-chance accuracy is 27%. As a human baseline, we report 85.4%, where each question is answered by at least two distinct human reviewers and we take the average of their answers.

The live leaderboard is on Hugging Face, and the evaluation harness we used to produce these numbers is on GitHub.

Small context windows create a measurement challenge, because passing the whole video at a high frame rate is impossible, while naively downsampling the video to a uniform FPS is likely to miss the relevant information needed to answer the question. We do two things to mitigate this:

• Short videos for dense skills Long videos do not require dense frame understanding (such as motion direction or rapidly changing details). Videos that test those skills are all shorter than 60 seconds, and every long-video question can be answered from discrete frames in sequence.
• A 256-frame sampling floor We ensure that at a floor of 256 frames sampled uniformly across the video, each section relevant to the question is seen in at least one sampled frame.

Examples

The dataset

Our dataset consists of 398 public videos and programmatically generated videos, the latter to avoid models having been exposed to the test data at train time, spanning just under 100 hours.

Where the camera sits matters just as much as the length, because it decides which kind of system the footage resembles. We therefore classified every video by viewpoint.

All videos have been annotated by human experts and we apply strict filtering to mitigate the issues discussed above. Every question in our final dataset must pass these two gates, in order:

• Gate 1 pass^3 < 100% for a blind, text-only model.
• Gate 2 pass^3 < 100% for an older baseline VLM. We chose Gemini 2.5 Flash-Lite.

We chose to pose questions in a hybrid format. We use a combination of public multiple-choice and hidden open-ended questions that can be submitted to prevent training data contamination.

Questions in our benchmark fall into one or more of the following skill categories:

Peeks into the future

We also ran models inside coding agents, which can write code and call tools, instead of prompting them directly. For Gemini 3.6 Flash this helped: driven by the Antigravity CLI it answers 77.36% of the benchmark correctly, against 63.45% as a bare call. In some smaller experiments we did see Coding Agents use OpenCV and other standard image libraries in sophisticated ways to answer questions which bare Vision Language Models could not solve.[11] The question of whether the harness might be the bottleneck for visual reasoning tasks, and the natural next question about which failure modes are harness dependent, are subject of future work.

Infinibench-v0

Some of the questions in VGI-Bench have been created by a continuously exploring agent that is rewarded for identifying new failure modes in VLMs. This agent creates the videos itself, using standard Python libraries like PyGame. The outcome is quite interesting, and our team will continue researching how to automate evaluation in this way. We find that a continuously exploring agent can create very interesting, non-trivial problems which serve as challenging out-of-distribution eval data and diagnose failure modes that were previously unknown.

Conclusion

We constructed VGI-Bench to serve as a North Star guiding a direction for the improvement of vision models. It shows that, despite impressive progress over the last years, VLMs still lack fundamental skills needed to understand and reason about everyday situations. Consequently, pushing the frontier of embodied AI requires a map of failure modes for the strongest VLMs that is continuously expanded and updated. VGI-Bench serves as the beginning of this effort. If this research is interesting to you, or you want to use our dataset for evaluations, please reach out here.