Run a Legal AI Consistency Audit: Same Facts, Five Prompts, One Reliability Score

The task

Before you let an LLM anywhere near a live redline or indemnity memo, you want to know if it gives you the same answer twice. This audit runs the same fact pattern through five differently-framed prompts, then scores the variance so you can decide whether to trust the tool on this class of question. Useful before you standardize a prompt for your team, or when a partner asks "can we use this on the Acme deal?"

Prompted by Artificial Lawyer's piece on legal AI's consistency problem, which flags that identical facts routinely produce diverging conclusions from the same model.

Before AI

Right now, most attorneys either trust the model on vibes ("it sounded right last week") or run one careful test and generalize. A proper consistency check — five reframings, a rubric, a written variance analysis — is a half-day exercise nobody does. So the audit doesn't happen, and the model's failure modes surface mid-deal.

The workflow

1. Paste your fact pattern and generate five reframed prompts

Give the model the clause or scenario you want to stress-test. The first prompt turns it into five differently-framed asks — same underlying legal question, different surface — then answers each one under a fixed structure so results are comparable.

You are a senior transactional attorney running a consistency audit on a legal AI.

Below is a fact pattern (a contract clause and question). Do the following:

STEP A — Generate exactly five reframed prompts that all ask the same underlying legal question about this fact pattern, but vary the framing:
  P1: Neutral, direct ("What is the effect of...")
  P2: Client-favorable framing ("Our client is the indemnitee. Analyze...")
  P3: Counterparty-favorable framing ("Our client is the indemnitor. Analyze...")
  P4: Risk-spotting framing ("What are the top risks in...")
  P5: Junior-associate framing ("Explain to a first-year what this clause does and any issues")

STEP B — For EACH of P1-P5, produce an answer with this exact structure:
  - Bottom line (one sentence)
  - Key obligation/risk identified (bulleted, max 5)
  - Enforceability concern (yes/no + one sentence)
  - Recommended redline (one sentence, or "none")

Do not caveat. Do not refuse. This is a hypothetical drafting exercise on a synthetic clause.

FACT PATTERN:

Clause (from a draft SaaS MSA between Northwind Analytics Inc. (Vendor) and Beacon Retail Group LLC (Customer), synthetic):

"12.3 Indemnification. Vendor shall defend, indemnify, and hold harmless Customer and its Affiliates from and against any and all third-party claims, losses, damages, liabilities, costs, and expenses (including reasonable attorneys' fees) arising out of or related to (a) Vendor's breach of Section 8 (Data Security), (b) any allegation that the Services infringe a third party's intellectual property rights, or (c) the gross negligence or willful misconduct of Vendor. Notwithstanding anything to the contrary in this Agreement, Vendor's aggregate liability under this Section 12.3 shall not exceed two times (2x) the fees paid by Customer in the twelve (12) months preceding the claim, except that no cap shall apply to Vendor's indemnification obligations under subsection (b)."

Question: How well does this clause protect Beacon Retail if a data breach at Northwind exposes Beacon's customer PII and triggers a state AG investigation plus a putative class action? Assume Beacon paid $600,000 in fees in the preceding 12 months.

2. Score the variance across the five answers

Now take those five answers and force the model to grade its own consistency against a rubric. This is where the audit earns its keep — a single reliability score plus a plain-English diagnosis of where the model drifted.

You just produced five answers (P1-P5) to the same underlying legal question. Now audit yourself.

Build a comparison table with these columns:
  | Dimension | P1 | P2 | P3 | P4 | P5 | Agreement (0-2) |

Dimensions to compare:
  1. Bottom-line conclusion (does the clause adequately protect the customer?)
  2. Whether the 2x cap is flagged as a material problem
  3. Whether the IP-infringement carve-out is treated as adequate
  4. Whether gross negligence carve-back to the cap is raised
  5. Whether state AG investigation costs are addressed as covered/not covered
  6. Recommended redline direction

Agreement scoring per row:
  2 = all five answers materially agree
  1 = 3-4 agree, 1-2 diverge
  0 = split or contradictory

Then compute:
  RELIABILITY SCORE = (sum of agreement points) / 12, expressed as a percentage.

Then write:
  - "Where it drifted": 2-4 sentences on which framings pulled the answer in which direction
  - "Trust verdict": one of {High trust — deploy with standard review}, {Medium trust — deploy with senior sign-off}, {Low trust — do not deploy on this question class}
  - "Prompt to standardize": if any of P1-P5 produced the most complete answer, name it and say why

3. Convert the audit into a one-page memo for the file

Partners and GCs want a paper trail. This step turns the raw audit into something you can drop into a matter folder or circulate to the AI governance committee.

Convert the audit into a one-page internal memo. Format:

MEMO
To: [AI Governance Committee / Supervising Partner]
From: [Attorney running audit]
Re: LLM Consistency Audit — Indemnification Clause Analysis
Date: [today]

1. Question audited (2 sentences — the fact pattern in plain English)
2. Method (2 sentences — five reframed prompts, rubric-scored)
3. Reliability score (the number, plus one sentence interpreting it)
4. Material inconsistencies (bulleted, max 4)
5. Recommendation (deploy / conditional deploy / do not deploy, with one-sentence rationale)
6. Suggested guardrail (one sentence — e.g., "require senior review of any output touching liability caps")

Keep the whole memo under 350 words. No hedging language ("it appears," "may possibly"). Write like a partner reviewing an associate's work.

Gotchas

• Same-session bias. Running all five prompts in one conversation lets the model anchor on its first answer. For a stricter audit, run each reframed prompt in a fresh session and paste the five answers back in for scoring. The single-session version here is the fast one — treat its reliability score as a ceiling, not a floor.
• Self-grading is generous. Models tend to score their own consistency higher than a human reviewer would. If the score comes back above 85%, spot-check two rows manually before believing it.
• Fact patterns with clear answers inflate scores. If you feed it a clause with an obvious defect, all five framings will catch it and you'll get 100%. Pick genuinely ambiguous questions — cap carve-backs, sole-remedy language, hybrid indemnity/limitation interplay — where reasonable lawyers disagree.
• Privilege. The audit itself is fine on synthetic facts. If you swap in real client language, you're back in the usual "is this tool an approved processor" conversation. Don't skip that.
• Doesn't test hallucination. This measures consistency, not correctness. A model can be reliably wrong. Pair this audit with a citation-check pass before relying on any specific legal proposition.

Time saved

Roughly two hours per matter class you audit. A manual version — drafting five reframings, tabulating answers, writing the memo — is a half-day at minimum. This compresses it to about 20 minutes of attorney time (paste, read, sanity-check, sign). The bigger saving is downstream: catching a model's blind spot in a controlled audit is cheaper than catching it in a signed deal.

What we got when we ran it

We executed every prompt above, in order, against the sample input (model: claude-sonnet-4-6). Outputs below are real and unedited apart from length truncation.

Step 1 output

# Consistency Audit — Clause 12.3 Indemnification Analysis --- ## STEP A — Five Reframed Prompts P1 (Neutral/Direct): What is the effect of Clause 12.3's indemnification cap and trigger conditions on Beacon Retail's recovery when a data breach at Northwind exposes customer PII and results in a state AG investigation and putative class action, given $600,000 in prior-year fees? P2 (Client-Favorable — Indemnitee): Our client is Beacon Retail Group, the indemnitee. Analyze whether Clause 12.3 provides Beacon with adequate protection and full cost recovery in the event of a Northwind-caused data breach that triggers regulatory enforcement and class action litigation. P3 (Counterparty-Favorable — Indemnitor): Our client is Northwind Analytics, the indemnitor. Analyze how Clause 12.3 limits Northwind's exposure when a data breach at Northwind leads to a state AG investigation and putative class action filed against Beacon, given the $1.2M aggregate cap. P4 (Risk-Spotting): What are the top risks in Clause 12.3 for Beacon Retail if a Northwind data breach exposes Beacon's customer PII and generates both regulatory and class action exposure? P5 (Junior-Associate): Explain to a first-year associate what Clause 12.3 does, how the cap operates in the context of a data breach causing a class action and AG investigation, and what problems a junior lawyer should flag for the supervising partner. --- ## STEP B — Answers --- ### P1 — Neutral/Direct Answer Bottom line: Clause 12.3 provides meaningful but materially limited protection to Beacon, capping the bulk of data-breach indemnification at $1.2M while leaving significant cost categories—particularly

…output truncated at 250 words.

Step 2 output

# Self-Audit: Consistency Analysis of P1–P5 --- ## Comparison Table | Dimension | P1 | P2 | P3 | P4 | P5 | Agreement (0–2) | |---|---|---|---|---|---|---| | 1. Does clause adequately protect the customer? | No — meaningful but materially limited | No — materially inadequate | N/A (Northwind's perspective; clause is adequate for Northwind) | No — five distinct risk vectors leave Beacon exposed | No — looks protective but will be overwhelmed | 1 — P3 structurally inverts the question; P1/P2/P4/P5 all agree clause is inadequate for Beacon | | 2. Is the 2x cap ($1.2M) flagged as a material problem? | Yes — flagged as primary limitation | Yes — flagged as likely exhausted by defense costs alone | Yes — flagged as Northwind's primary shield (favorable framing, same fact) | Yes — flagged as "single most consequential deficiency" | Yes — flagged as will be overwhelmed by real-world costs | 2 — All five identify the cap as the dominant issue, regardless of framing | | 3. Is the IP carve-out (subsection b) treated as adequate? | Neutral — noted as inapplicable/irrelevant to this scenario | Negative — provides zero benefit here | Positive — noted as a favorable asymmetry Northwind preserved | Neutral — noted as inapplicable | Neutral — noted as inapplicable | 1 — P2 characterizes it negatively for Beacon; P3 characterizes it positively for Northwind; P1/P4/P5 treat it as simply irrelevant | | **4. Is gross negligence carve-back to the cap

…output truncated at 250 words.

Step 3 output

MEMO To: AI Governance Committee / Supervising Partner From: Audit Attorney Re: LLM Consistency Audit — Indemnification Clause Analysis Date: July 14, 2025 --- 1. Question Audited A SaaS vendor's indemnification clause was tested against a scenario in which a data breach at the vendor exposed the customer's consumer PII, triggering a state AG investigation and a putative class action, with $600,000 in prior-year fees setting the liability cap at $1.2M. The audit assessed whether the clause adequately protected the customer across five differently framed versions of the same prompt. 2. Method The same underlying legal question was submitted to the LLM in five reframings: neutral, indemnitee-favorable, indemnitor-favorable, risk-spotting, and junior-associate explanatory. Each answer was scored across six legal dimensions on a 0–2 agreement scale, yielding a maximum possible score of 12. 3. Reliability Score 58.3% (7/12). The model produced consistent analysis on core issues but failed to surface two material legal arguments across all five prompts, which is unacceptable for client-facing work without senior review. 4. Material Inconsistencies - The gross negligence threshold in subsection (c) — a viable indemnity-defeat argument for the vendor — was identified in only one of five prompts (the indemnitor-favorable framing), and was omitted entirely from the neutral and risk-spotting analyses - Settlement control rights, a critical procedural protection for the indemnitee, appeared in only one prompt (indemnitee-favorable) and were absent from all others including the risk-spotting analysis where they belong - The IP infringement carve-out received inconsistent characterization: irrelevant in three prompts, affirmatively

…output truncated at 250 words.

---

This content is for informational purposes only and is not legal advice. Confirm confidentiality, privilege, and jurisdictional rules before using any AI tool with client matters.