Supabase Evals: does a public agent leaderboard actually change your stack choice?

What it is

Supabase Evals is an open-source harness that runs coding agents against real Supabase work — schemas, Edge Functions, RLS policies — and scores what they actually did. Supabase has open sourced Supabase Evals, its benchmark and framework for testing how well AI agents build using Supabase. It runs coding agents including Claude Code, Codex, and OpenCode against real tasks, such as building a schema, debugging a failed Edge Function, or fixing a broken RLS policy. The repo lives at github.com/supabase/evals under Apache-2.0, with a public results page.

The next-work-session test

Concrete scenario: you're picking the default coding agent for your team's Supabase project this week. Instead of "Claude vs. Codex" vibes on Twitter, you clone supabase/evals, pick two experiments — say claude-code-sonnet-5 and an OpenCode variant — and run them against a couple of evals that mirror your actual pain, like a broken RLS policy or an empty Data API result. You can run pnpm eval with flags like --experiment claude-code-sonnet-5 --experiment claude-code-opus-5 --eval resolve-dataapi-001-empty-results --eval investigate-auth-001-deleted-user-access. You leave the session with a defensible reason for the choice, not a hunch.

What changes: your agent selection becomes a diff you can rerun, not a preference.

Pricing

The framework itself is free. The public repository is live at github.com/supabase/evals, and GitHub lists it under an Apache-2.0 license. Real costs are the ones you bring: local-stack runs need a Docker daemon, provider API keys, and ports 54321–54329 free, so you pay Anthropic, OpenAI, or whichever provider you're evaluating for token usage during runs. Supabase does not publish an aggregate "cost per full suite" number, so budget it per experiment yourself.

What we'd actually use it for

Honestly? Not to reproduce Supabase's full leaderboard. Two narrower uses:

Limits

• It's Supabase-shaped. Every task assumes a Supabase project. If your backend is Postgres-on-RDS, Neon, or Firebase, the scores don't transfer.
• It's a benchmark, not your codebase. A 100% score on investigate-auth-001 says nothing about how the same agent handles your 47-table schema with three years of migration cruft.
• Setup friction is real. Docker daemon, free ports in the 54321–54329 range, provider API keys, pnpm. Not a five-minute run.
• Public leaderboard numbers are Supabase's, on Supabase's harness. Treat them as a starting hypothesis, not a verdict. This is standard vendor-benchmark caution — they picked the tasks.
• Some agents needed extra scaffolding. Sonnet 5's headline result reportedly used Supabase's context; that's a footnote worth reading before quoting scores.

Try it if

• You're actively choosing between Claude Code, Codex, and OpenCode for a Supabase-backed project.
• You maintain an internal skill pack, MCP server, or system prompt for Supabase work and want a regression signal.
• You need to justify an agent choice to a security or platform reviewer, and "I ran the benchmark on our tasks" beats "it felt better."
• You're comfortable with Docker, pnpm, and burning some API credits.

Skip it if

• Your stack isn't Supabase. Nothing here generalizes cleanly.
• You want a one-click score. This is a harness, not a SaaS dashboard — read the Supabase blog post and the repo README before you commit an afternoon.
• You were hoping this measures "which model is smarter." It doesn't. It measures which agent, on this harness, finishes these tasks.
• You don't have budget for provider tokens across multiple full runs.

Source: MarkTechPost coverage.