GUI-Primitives: Spatial Reasoning Failures in Vision Models

Computer-use agents, AI systems that interact with graphical interfaces by reading screenshots and issuing mouse/keyboard commands, are gaining capability but failing at a fundamental task: reliably locating UI elements. A new benchmark published on arXiv this week exposes why.

GUI-Primitives, a 994-item evaluation dataset, isolates a critical gap in vision-language models: they struggle to bind spatial language to the correct UI element even when the screenshot is fixed and only the instruction changes. Nineteen tested models hit a maximum of 32% strict point-in-box accuracy on contrastive spatial reasoning tasks. That’s not a rounding error; it’s a fundamental failure mode with security and reliability implications.

What the Benchmark Tests

Unlike text-only LLM benchmarks, GUI-Primitives tests whether a model can ground spatial instructions in a graphical interface. Each contrastive pair holds the screenshot and anchor element fixed. Two paired instructions change the spatial relation, shifting the correct target between two candidate elements on that same screenshot.

The benchmark covers seven spatial relations common to interface design:

• Horizontal: left/right positioning
• Vertical: above/below positioning
• Containment: elements nested or grouped
• Alignment: shared edges or centers
• Proximity: nearby elements
• List ordinal: Nth item in a list
• Occlusion: elements partially or fully hidden behind others

A 196-item subset was validated by five annotators (κ=0.94 for well-formedness, κ=0.79 for target selection). This is solid inter-rater agreement, which rules out dataset ambiguity as an excuse for model failure.

The Findings: Major Blind Spots

Performance varied sharply by relation type. When the model’s predicted coordinate falls within one of the two candidate regions (classified predictions), accuracy reaches 0.82–0.90 for horizontal/vertical and proximity tasks. These conditional results show relatively strong relation understanding once a candidate is localized. Containment and occlusion remain near chance (0.50) even under that condition, indicating weaker understanding of those relations.

On strict point-in-box accuracy, 19 models maxed out at 32%. Their unconstrained coordinates fall outside both candidate regions 60–92% of the time. Most failures therefore involve candidate localization, rather than choosing the wrong spatial relation between successfully localized candidates.

The paper tests an oracle diagnostic: visually marking the two ground-truth candidate regions in the screenshot. Performance jumped 35–57 percentage points. This intervention supplies candidate locations that the model would otherwise have to discover; it is not a deployable solution for finding them in an unfamiliar interface. The improvement identifies candidate localization as the dominant bottleneck, while the conditional containment and occlusion results show that relation understanding also needs work.

Why This Matters for Automated Agents

Computer-use agents rely on vision-language models to parse screenshots and emit coordinates. If the model cannot accurately locate the target element, the agent either clicks the wrong button, submits a form field to the wrong text box, or fails outright. For a system orchestrating routine IT tasks, including filling out expense reports, configuring cloud infrastructure, and navigating configuration menus, precision at the element level is not optional.

Consider an agent tasked with changing a security policy setting buried in a nested menu. If it misidentifies “the dropdown below the label” or fails to see an element hidden behind a modal, the agent either hangs, takes an incorrect action, or falls back to a human operator. In a compliance or security context, incorrect actions can have teeth.

OWASP LLM08 (Excessive Agency) flags this precise risk: “Granting LLMs unchecked autonomy to take action can lead to unintended consequences.” When the underlying perception mechanism is unreliable, autonomy without oversight becomes a liability. The question is not whether models can perform GUI tasks; they can. It is whether they do so with predictable precision.

The Larger Ecosystem of GUI Benchmarks

GUI-Primitives is not alone. The past year has seen a surge in fine-grained GUI evaluation:

• DiffSpot (May 2026) tests detection of subtle visual changes in web interfaces
• FineState-Bench (April 2026) evaluates state-conditioned grounding across desktop, web, and mobile
• AutoGUI-v2 (April 2026) focuses on GUI comprehension and interaction outcome prediction
• GUI-360° (November 2025) is a large-scale desktop automation suite

Each surfaces different failure modes. Together, they sketch a picture of what reliable agent deployment requires: not just a capable model, but one that degrades gracefully, provides uncertainty signals, and operates within defined guardrails. Compare evaluation methods and benchmarks when choosing which failure modes to test before deployment.

What Defenders Should Take Away

Organizations deploying computer-use agents should treat spatial reasoning failures as a first-order risk, not as a known limitation that QA will shake out:

1. Test on deployment UIs. General benchmarks do not establish performance on proprietary or internal interfaces. Teams should run GUI-Primitives-style contrastive tests on the tools their agents will actually use. When choosing frameworks for agent evaluation, look for a workflow that records the instruction, screenshot, predicted coordinates and per-item outcome so localization failures can be reviewed separately from relation errors.

2. Mandate uncertainty signals. The model should report confidence, not just coordinates. Low confidence on a high-stakes action (changing permissions, modifying infrastructure) should require human review.

3. Layer detection guards. Image processing should validate that the clicked element matches semantic expectations. If the model clicked a “Cancel” button when aiming for “Confirm,” the action should be blocked before it completes.

4. Monitor for silent failures. Agents that emit coordinates outside the UI bounds or that click in empty space are not failures; they are diagnostic signals. Teams should log them, alert on them, and retrain on them.

5. Scope autonomy tightly. Teams should constrain agents to read-only tasks or low-risk actions until spatial reasoning is demonstrably reliable on their own interfaces.