CUA-Bench by Vals.ai: a computer-use eval that puts agents in front of six video games

What it is

CUA-Bench is a public benchmark from Vals AI that scores computer-use agents on their ability to play commercial PC video games with nothing but screen pixels in and keyboard/mouse events out. It asks whether an AI agent can play six commercial video games with only a keyboard and a mouse. Each model gets one three-hour trial per game and sees nothing but the screen: no game API, no save files, no memory inspection. Three of the six games are public and three are held out, which is the part that matters for anyone worried about training-set contamination.

The next-work-session test

You're a developer deciding which computer-use model to wire into an agent loop next sprint — say, a QA bot that drives a desktop app your customers use. You need signal on which frontier CUA model actually handles messy, real-time GUIs versus which one only looks good on curated screenshot benchmarks. CUA-Bench gives you a fresh, adversarial datapoint to weigh against OSWorld or WindowsAgentArena before you commit an SDK choice. It won't replace your own eval harness, but it will change your shortlist.

Pricing

The benchmark itself is free to read on Vals AI's leaderboard — the CUA-Bench page publishes results, per-game breakdowns, and video clips at no cost. Vals AI's broader business is private evaluation infrastructure: it publishes public leaderboards (e.g., the Vals Index, Finance Agent benchmark, Vals Legal AI Report) and sells private evaluation infrastructure to labs and enterprise engineering teams. Pricing for private/enterprise runs is not published — treat that as unverified and expect a sales conversation.

What we'd actually use it for

Two things, both narrow:

Limits

• It's games, not your app. Performance on Minecraft doesn't linearly predict performance on Salesforce. Use it as one signal among several.
• No pricing/latency-per-task table on the CUA-Bench page yet the way Vals publishes for their other benchmarks — Vals publishes per-task accuracy with standard error, latency, and cost for every model it runs on Terminal-Bench, so the same granularity may arrive but isn't guaranteed here.
• Three-hour trials are expensive to reproduce. You are not running this locally on a laptop.
• Held-out games mean you can't fully audit the eval. That's the trade-off for contamination resistance — you trust Vals or you don't.
• This is not the Cua AI "Cua-Bench" for professional software (separate project on cua.ai). Same acronym, different benchmark, different focus. Don't cite one when you mean the other.

Try it if

• You're picking between Claude Computer Use, OpenAI CUA, and Gemini Computer-Use for a production agent loop and want a third-party datapoint.
• You maintain an internal eval and want to compare your findings against a public leaderboard with held-out tasks.
• You need to temper stakeholder expectations about what today's CUA models can actually do end-to-end.
• You're tracking model releases and want a benchmark where the scores still have room to move — among five tested frontier models, the highest average across public and held-out titles was 19.2 points out of 100, so ceiling is not the problem.

Skip it if

• Your agent runs headless in a browser or terminal — Vals' Terminal-Bench or a browser-agent eval will map more cleanly to your workload.
• You need reproducible per-task cost and latency numbers today to build a routing layer.
• You're looking for an SDK to integrate. CUA-Bench is a leaderboard and methodology, not a library you import.
• Your stakeholders only care about task-completion percentages on their own workflow — in which case, build the eval in-house and use CUA-Bench purely as external calibration.

---

Bottom line for developers: treat CUA-Bench as a contamination-resistant reality check on frontier computer-use claims, not as a proxy for your own agent's production readiness. The 19.2% ceiling is the whole story — it says "keep humans in the loop" more loudly than any vendor blog post will.