Benchmark Your AI Coding Agent Before You Commit: A Prompt-Driven Eval Framework for Database Tasks

The task

You're picking a coding agent for a project — Claude Code, Codex, OpenCode, whatever's next — and the leaderboards don't reflect your schema, your RLS rules, your Edge Function quirks. Before you standardize your team on one, you want a repeatable eval on tasks that mirror your actual backend work: writing a migration, fixing a broken policy, debugging a function that swallows errors.

Before AI

Manual bake-offs mean spinning up a scratch project per agent, hand-writing tasks, eyeballing diffs, and arguing in Slack about whose output was "cleaner." Two-plus hours per agent, and the criteria drift between runs. Supabase's own framework runs coding agents including Claude Code, Codex, and OpenCode against real Supabase tasks, for example, building a schema, debugging a failed Edge Function, or fixing a broken RLS policy, and then scores how well they performed. That's the model we're borrowing — but in prompt form, without the Docker setup, so you can run it against any model with an API.

The Supabase evals repo and blog post is the reference; we're building the lightweight version you can run in a chat window.

The workflow

Paste your schema + task spec into the first prompt. Each subsequent prompt reads the prior output as context. At the end you have a scored comparison you can commit to /docs/evals/ and re-run when a new model drops.

Step 1 — Generate a rubric-graded solution from Agent A

Feed the model your schema, the task, and the grading criteria up front. Asking for structured self-assessment gives you a baseline you can compare against later. Scoring combines deterministic checks with LLM-as-a-judge. — we'll mimic that split in the rubric.

You are being evaluated as a coding agent on a database task. Respond in this exact structure:

# SOLUTION
Full SQL / code with no ellipses. Include comments explaining non-obvious choices.

# DETERMINISTIC CHECKS (self-report, honest)
- Does every referenced table exist in the provided schema? yes/no
- Are all foreign keys declared with ON DELETE behavior? yes/no
- Are RLS policies specified for each new table? yes/no
- Any syntax that requires a specific Postgres version? state it

# RUBRIC SELF-SCORE (0-3 each, with 1-line justification)
- Correctness: solves the stated task
- Safety: no destructive ops without guards, no policy bypass
- Idempotency: safe to re-run
- Readability: naming, structure, comments
- Scope discipline: no unrequested changes

# ASSUMPTIONS
Bullet any assumption you made that the task did not explicitly state.

Task and schema follow below. Do not ask clarifying questions — make reasonable assumptions and note them.

---

SCHEMA (existing):
create table orgs (
  id uuid primary key default gen_random_uuid(),
  name text not null,
  created_at timestamptz default now()
);

create table memberships (
  org_id uuid references orgs(id),
  user_id uuid references auth.users(id),
  role text check (role in ('owner','admin','member')),
  primary key (org_id, user_id)
);

-- RLS enabled on both, policies restrict to members of the org.

TASK:
Add an `api_keys` table so each org can issue multiple API keys. Requirements:
1. Keys belong to an org and are created by a specific user (auditable).
2. Store only a SHA-256 hash of the key, never the plaintext.
3. Support optional expiration and a revoked_at timestamp.
4. RLS: only org owners and admins can select or insert; members cannot see keys at all.
5. Add an index that supports fast lookup by hash for authentication.
6. Provide a migration that is safe to run twice.

Agent under test: Agent A (call yourself "Agent A" in output).

Step 2 — Have Agent B solve the same task, blind to A's answer

We want an independent attempt, not a critique of A. Reset the framing but keep the rubric identical so scores are comparable.

You are Agent B, being evaluated on the SAME database task Agent A just attempted. You have NOT seen Agent A's solution and should not reference it. Solve independently.

Use the exact same output structure as before:
# SOLUTION
# DETERMINISTIC CHECKS
# RUBRIC SELF-SCORE
# ASSUMPTIONS

The task and schema were provided earlier in this conversation — use them. Do not restate them. Begin your answer with "# SOLUTION" and nothing before it.

Step 3 — Judge both solutions against the deterministic checks

Now switch roles: the model becomes the judge, not the coder. Force it to actually re-check the claims rather than trust the self-scores. This is the LLM-as-a-judge step — imperfect, but consistent if the rubric is tight.

You are now a strict evaluator. Agents self-score generously; your job is to verify.

For EACH of Agent A and Agent B's solutions above, produce:

## <Agent> — Verified Deterministic Checks
Re-answer each check yourself by reading their SOLUTION block. Mark DISAGREE where their self-report was wrong, with a one-line reason and the offending line quoted.

## <Agent> — Adjusted Rubric Scores (0-3)
Correctness / Safety / Idempotency / Readability / Scope discipline.
If you adjust a score down, state why in <= 15 words.

## <Agent> — Critical Bugs
List anything that would fail in production. Empty list is a valid answer.

Then end with:

## Verdict
| Criterion | Agent A | Agent B |
|---|---|---|
| Correctness | | |
| Safety | | |
| Idempotency | | |
| Readability | | |
| Scope discipline | | |
| **Total /15** | | |

## Recommendation
Which agent's solution would you ship, and what single follow-up prompt would you send to the winner to close remaining gaps? Be specific.

Step 4 — Save the eval artifact

Copy the full transcript (task + both solutions + verdict) into docs/evals/<date>-<task-slug>.md in your repo. Next time you're picking an agent, or a new model version ships, re-run the same three prompts against the new candidate and compare totals. Over 5-10 tasks you'll see patterns your leaderboard reading won't catch — one agent may crush schema work but botch RLS every time.

Gotchas

• Self-scores lie. Skipping Step 3's judge pass is the most common way this workflow gives you a wrong answer. Always verify.
• LLM-as-a-judge has known biases. Judges tend to favor longer, more verbose answers and the response that appears second. Rotate order across tasks or run the judge twice with A/B swapped.
• Prompt-only ≠ agent-in-the-loop. This measures single-shot solution quality, not tool use, retries, or file navigation. A platform-lite runtime exposes a Management API-compatible surface backed by @supabase/lite. Agents get one retry before grading. — Supabase's actual harness tests behavior you can't fully replicate in a chat box. For tool-use evals, go to the real repo.
• RLS is the trap. In our experience, agents will happily generate policies that reference auth.uid() inside a function that runs as service_role and silently defeats them. The judge prompt should catch this; sometimes it doesn't. Read the policies yourself.
• One task isn't a benchmark. Run 5+ tasks covering different areas (schema, RLS, functions, migrations, debugging) before making a call. Supabase Evals is designed to evaluate AI coding agents on real-world engineering tasks derived from real support tickets and GitHub issues. — mine your own issue tracker for task ideas.

Time saved

Roughly 2-3 hours per agent comparison versus manual bake-offs. A manual head-to-head across two agents on one non-trivial database task — spinning up scratch projects, writing the task, comparing diffs, documenting the call — runs 3-4 hours. The prompt workflow is ~30 minutes of model time plus your review. The bigger win is that the artifact is repeatable: when a new model version ships, you re-run the same prompts and get a comparable number, not a fresh subjective impression.

What we got when we ran it

We executed every prompt above, in order, against the sample input (model: claude-sonnet-4-6). Outputs below are real and unedited apart from length truncation.

Step 1 output

# SOLUTION ```sql -- ============================================================= -- Migration: add api_keys table -- Agent A -- Safe to run twice (idempotent) via IF NOT EXISTS / DO blocks -- ============================================================= -- ------------------------------------------------------------ -- 1. Create the api_keys table -- ------------------------------------------------------------ CREATE TABLE IF NOT EXISTS api_keys ( id uuid PRIMARY KEY DEFAULT gen_random_uuid(), org_id uuid NOT NULL REFERENCES orgs(id) ON DELETE CASCADE, -- remove keys when org is deleted created_by uuid NOT NULL REFERENCES auth.users(id) ON DELETE RESTRICT, -- prevent deleting a user who owns keys key_hash text NOT NULL, -- SHA-256 hex digest only; plaintext never stored label text, -- human-readable nickname for the key expires_at timestamptz, -- NULL means never expires revoked_at timestamptz, -- NULL means still active; set to revoke created_at timestamptz NOT NULL DEFAULT now(), -- Enforce hash format: exactly 64 hex characters (SHA-256) CONSTRAINT api_keys_hash_format CHECK (key_hash ~ '^[0-9a-f]{64}$') ); -- ------------------------------------------------------------ -- 2. Index for fast authentication lookup by hash -- (the most latency-sensitive query: look up a presented key) -- ------------------------------------------------------------ CREATE UNIQUE INDEX IF NOT EXISTS api_keys_key_hash_idx ON api_keys (key_hash); -- UNIQUE also prevents accidental hash collisions / duplicates. -- Optional composite index if callers filter by org as well CREATE INDEX IF NOT EXISTS api_keys_org_id_idx ON api_keys (org_id); -- ------------------------------------------------------------ -- 3. Enable RLS on the new table -- ------------------------------------------------------------ ALTER TABLE api_keys ENABLE ROW LEVEL SECURITY; -- Force RLS even for the table owner (prevents accidental bypass -- if the migration is run as the table-owning role) ALTER TABLE api_keys FORCE ROW

…output truncated at 250 words.

Step 2 output

# SOLUTION ```sql -- ============================================================ -- Migration: api_keys table -- Agent B -- Designed to be idempotent (safe to run multiple times) -- ============================================================ -- ------------------------------------------------------------ -- STEP 1: Create the api_keys table -- ------------------------------------------------------------ CREATE TABLE IF NOT EXISTS api_keys ( id uuid PRIMARY KEY DEFAULT gen_random_uuid(), -- Owning organisation org_id uuid NOT NULL REFERENCES orgs(id) ON DELETE CASCADE, -- keys are meaningless without the org -- Auditable creator; RESTRICT prevents silently orphaning audit records created_by uuid NOT NULL REFERENCES auth.users(id) ON DELETE RESTRICT, -- SHA-256 hex digest of the raw key; plaintext is NEVER persisted key_hash text NOT NULL, -- Human-readable label (optional, aids management UIs) label text, -- Optional TTL; NULL = no expiry expires_at timestamptz, -- Soft-revocation timestamp; NULL = currently active revoked_at timestamptz, created_at timestamptz NOT NULL DEFAULT now(), -- Enforce exactly 64 lowercase hex characters (SHA-256 output) CONSTRAINT api_keys_hash_is_sha256 CHECK (key_hash ~ '^[0-9a-f]{64}$'), -- A given hash value must be globally unique (one key per hash) CONSTRAINT api_keys_key_hash_unique UNIQUE (key_hash) ); -- ------------------------------------------------------------ -- STEP 2: Index for fast authentication-path lookup -- The UNIQUE constraint above already creates a unique -- index on key_hash; we declare it explicitly here for -- documentation clarity and to add a named index we can -- reference in EXPLAIN output. The constraint index is -- sufficient; this additional named index is therefore -- omitted to avoid redundancy — see ASSUMPTIONS. -- ------------------------------------------------------------ -- Fast lookup by org (e.g. "list all keys for this org") CREATE INDEX IF NOT

…output truncated at 250 words.

Step 3 output

Agent A — Verified Deterministic Checks **Does every referenced table exist in the provided schema?** AGREE — `orgs`, `auth.users`, `memberships` all present. **Are all foreign keys declared with ON DELETE behavior?** AGREE — `ON DELETE CASCADE` on `org_id`, `ON DELETE RESTRICT` on `created_by`. **Are RLS policies specified for each new table?** AGREE — SELECT, INSERT, UPDATE policies created. DELETE intentionally absent and documented. **Any syntax that requires a specific Postgres version?** AGREE — PostgreSQL 13+ for `gen_random_uuid()` built-in; `auth.uid()` is Supabase-specific. --- ## Agent A — Adjusted Rubric Scores **Correctness: 2/3** The `DROP POLICY` statement in the DO block is syntactically wrong: ```sql DROP POLICY api_keys_select_owner_admin ON api_keys; ``` Policy names must be quoted as identifiers or the `IF EXISTS` guard is on the outer DO block but the bare `DROP POLICY` without `IF EXISTS` inside the DO block will still error if the policy doesn't exist on the *first* run (the IF EXISTS check guards it, but the pattern is fragile — actually re-reading: the IF EXISTS check does guard it correctly via the PL/pgSQL branch). However, a more concrete bug: the `active_api_keys` view is created with `CREATE OR REPLACE VIEW` but references `api_keys.*` — this will expose `key_hash` in plaintext to anyone who can query the view. The view itself inherits RLS, so this is not a *security* correctness failure, but the view was not requested and its `SELECT *` is sloppy for a secrets-adjacent table. More critically: the DO block policy drop uses the policy name

…output truncated at 250 words.