From Minutes to Days: Agents Beat Frontier VLMs in Video QA

At Seldon, we believe multimodal AI is the only way to achieve real-world intelligence. Moving forward in this domain will require the next generation of systems to reason across vision, sound, time, and tools, which needs both more scalable labeled datasets and the tools to search over and understand them.

In this light, we introduce MULE. MULE performs at both extremes that frontier models still struggle with: handling long-horizon inputs like a week of egocentric video, where evidence is sparse and scattered across hours of footage, as well as shorter inputs that demand deep, multi-step reasoning, where a single forward pass leaves accuracy on the table. The toolset behind this goes well beyond what most readers picture when they hear "video labeling".

We hit 66% accuracy on EgoLifeQA and 55.3% Non-Lin Score on Video-MME-v2, an 8.5-point lead on EgoLifeQA and a 5.9-point lead on Video-MME-v2 by replacing the most naive approach of feeding the entire video into the model directly with a harness that treats VLMs as tools rather than as the reasoning substrate. We call this system MULE, and it is the product of months of careful tuning against failure modes that frontier models systematically get wrong.

The rest of this post explains how MULE is architected and how it performs on two very different benchmarks, Video-MME-v2 and EgoLifeQA.

The bottleneck: temporal and compositional reasoning

For AI to be genuinely useful, it has to understand and interact with the world, not just describe screenshots of it. In language, this has driven the field toward ever-larger context windows and the rise of agents that can plan, call tools, and act. But the audiovisual world we actually live in is still largely underexplored by comparison, and the gap shows up sharply when you ask current models to reason across time.

This is the dominant failure mode in the field today. Recent work on continuous, multi-day egocentric video has argued that existing methods, including frontier LLMs and standard retrieval-augmented generation, are fundamentally constrained by limited context windows and lack the ability to perform compositional, multi-hop reasoning over very long video streams.[1]

We see this as the central bottleneck for ambitious multimodal applications, from humanoid robots that need to remember what happened an hour ago, to assistants that track habits over weeks, to analysts who want to understand how a narrative unfolds across thousands of hours of footage.

MULE: a harness for verified multimodal reasoning

MULE is an agentic system that takes a question and a video corpus and returns a grounded answer. Instead of ingesting the full video into a single forward pass, it works iteratively, locating evidence, verifying it, composing partial findings, and only then committing to an answer. This is what lets it operate on inputs ranging from a 30-minute clip to a week of egocentric footage with the same machinery.

Building this harness took a substantial amount of human curation. We ran the system against real workloads and had researchers walk through agent reasoning traces by hand, cataloging where it jumped to conclusions, missed partial evidence, or confused similar entities. Each failure mode became a refinement to tool calling, task decomposition, or edge-case handling, until the system was tuned against thousands of real failures on real video.

Concretely, this means that on a question like "How many times did I drink coffee this week, and did it track with how late I went to bed?", MULE narrows the relevant footage from 50 hours down to a handful of short, targeted clips before the perception step runs. The VLM is asked targeted questions about seconds of video, not asked to reason over the whole week. This is dramatically cheaper at inference time and, as we will show, way more accurate.

MULE is by nature an agentic system but shares a deep conceptual kinship with Recursive Language Models (RLMs).[2]

We are working on a multimodal RLM as a natural extension of the current system.

Results on Video-MME-v2

Video-MME-v2 is the successor to the original Video-MME benchmark, built explicitly because v1 had saturated. It contains 800 videos with an average length of around 10 minutes, paired with 4 questions per video and 8 answer options per question. The questions span four high-level categories and 31 subcategories, and over 80% of the videos were uploaded to YouTube in 2025 or later, which sharply limits memorization-style contamination.

The v2 design is built around two ideas that, taken together, make this benchmark genuinely punishing for monolithic VLMs.

Progressive multi-level evaluation Capabilities are organized into three cognitive stages: Level 1 (multi-point information aggregation across frames, audio, and subtitles), Level 2 (temporal understanding, including action recognition, ordering, change detection, and causal reasoning), and Level 3 (complex reasoning, including physical-world reasoning, social behavior analysis, complex plot comprehension, and video-based knowledge acquisition).

Grouped non-linear scoring Rather than scoring questions independently, v2 organizes the 4 questions per video into groups of two kinds. Capability Consistency groups probe the same capability at four different granularities (e.g., counting latte-art appearances, take-away orders, dine-in orders, and total cups in a barista video). The group score is (N/4)² where N is the number of correct answers, so getting 3 of 4 right scores 0.5625, not 0.75. Reasoning Coherence groups string four questions into a logical chain where each step depends on the previous answer, with a "first-error truncation" rule: as soon as the model errs, every subsequent answer in the chain is discarded.

Why it is hard The v2 leaderboard tells the story directly. Gemini-3-Pro, currently the best model on the benchmark, scores 66.1% on traditional per-question average accuracy but only 49.4% under grouped non-linear scoring. The Video-MME-v2 authors note this gap explicitly. Even SOTA models rarely answer all four related questions in a group correctly, which means their per-question accuracy is buoyed by lucky guesses and partial credit on questions where the underlying capability is not actually robust. Humans remain comfortably above all of them.

The v2 design also closes the most common shortcut in video QA. Every question is run through Gemini-3-Pro in text-only mode during quality control, and any question solvable from text priors alone is removed. Models are forced to actually look at the video.

Why an agentic system makes sense here Both group types reward exactly what an agent does that a one-shot VLM cannot. For Capability Consistency, the agent treats each of the four questions as its own retrieval problem with its own frame budget, rather than trying to satisfy all four from a single sample. For Reasoning Coherence, the agent verifies each intermediate step against retrieved evidence before committing to the next, which is precisely what the benchmark's first-error truncation rule rewards.

Results on EgoLifeQA

EgoLifeQA is the question-answering benchmark built on top of the EgoLife dataset, a week of footage from six participants living together while wearing Meta Aria glasses. The QA benchmark itself comprises 500 multiple-choice questions over roughly 50 hours of one participant's footage, with 66% of questions requiring the model to look back more than 2 hours and over 15% requiring more than 24 hours of past activity. The five question categories target different aspects of long-horizon egocentric reasoning:

• EntityLog tracking object usage and locations over time
• EventRecall recalling details from the last time a critical task was performed
• HabitInsight identifying behavioral patterns
• RelationMap interpersonal interaction and person identification
• TaskMaster task assignment grounded in prior actions (the "buy a pen when the ink is low" class of problems)

Why it is hard A HabitInsight question requires aggregating across hundreds of short moments scattered over a week. A RelationMap question requires the system to maintain stable identities for multiple people across changes in lighting, clothing, and viewpoint. Frontier VLMs struggle accordingly. With uniform sampling at 3000 frames, Gemini 2.5 Pro reaches only 46.8% overall on EgoLifeQA, and GPT-4.1 with 1-FPS captions sits at 36.0%. RelationMap and TaskMaster, the two categories that most stress multi-hop reasoning, are the categories where these models fall furthest behind.

What comes next

We are building a scalable system that sets a new standard for annotations on multimodal data.

This matters because the labs training the next generation of multimodal models need exactly this kind of data. Long-form video, with verified answers to compositional and temporal questions, is the bottleneck for further scaling, both for evaluation and for post-training. We are using MULE to produce that data at scale, across the kinds of multimodal sources we believe will define the next phase of frontier model capability, including long-horizon egocentric video, multi-day security footage, broadcast and news archives, and conversational audio.

If this research is interesting to you, we'd love to hear from you! Please reach out here.