← All work · 2026

AdventureBreaker

An AI whose only job is to break my AI

Agentic QA for an AI-narrated game engine. An autonomous, adversarial playtester drives ZorkAI through its real production backend, walks it into deep game states, and attacks each one to break the engine, the parser and, above all, the narrator. Then it files the bugs.

122: durable findings logged
81: filed as issues against ZorkAI
86: merged pull requests
0: dependencies: pure standard library

What I did: The idea, the architecture (spine, ribs, the oracle stack, the narrator A/B) and the rule that every finding is verified against the engine source before it’s logged.
What AI did: Most of the code. And the critic itself, the reasoning model that judges what the cheap checks can’t.

The problem

You can’t unit-test a hallucination. Traditional QA asserts on fixed outputs, and that model breaks the moment an LLM is in the loop: an AI narrator’s prose is non-deterministic, open-ended and unbounded. ZorkAI has thousands of tests for its engine, and none of them can tell you whether the narrator just invented a sword.

Spine and ribs

The goal is not to win the game. It’s to break it.

The spine replays a known-good walkthrough, extracted from ZorkAI’s own test fixtures, to drive the game into deep, varied states. Each step carries the expected output, so the spine doubles as a golden transcript.
The ribs attack each state before moving on: adversarial inputs aimed at the engine, the parser and the narrator.
Save and restore let it probe destructively, then fork back. Attacking never derails progress.
Narrator A/B. Every command can run with the narrator on or off. Wrong with the narrator off is an engine bug. Right with it off but wrong with it on is a narrator bug.

Cheapest checks first

Four layers of verification. The first three are free and deterministic: contract checks (errors, malformed responses, leaked stack traces), consistency checks (the prose says you took the item, but the inventory didn’t change), and anchor checks against known text. Only what passes all three reaches the fourth: an LLM critic that judges hallucinations, character breaks, prompt-injection compliance, spoilers and lore contradictions.

It has receipts

Every finding gets a stable ID, is deduplicated, and is tracked from open to filed to fixed in a committed ledger that remembers what has been tested across runs and computes what hasn’t. The worst one so far: in production, roughly fourteen consecutive wait commands silently reset a Planetfall session, which made one of the game’s own scripted sequences impossible to finish.

It also caught something I’d noticed myself while building this site. Address Floyd or Blather when they aren’t in the room, and the game hands your command to the player instead of saying they aren’t here.

1"""Deterministic oracles -- the free, no-LLM bug detectors.
2
3L0  contract     : transport/HTTP/JSON/score-bounds/leaked-internals checks on the raw result.
4L1  consistency  : the narrator prose vs the structured state envelope.
5L2  anchors      : known static strings + narrator-leak heuristics.
6
7Each oracle returns OracleHit objects. They never raise; a noisy oracle is
8tuned down to low/info, not removed, so the interactive driver (Claude) makes
9the final call on whether a hit is a real bug.
10"""
11from __future__ import annotations
12
13import re
14from dataclasses import dataclass, field
15from typing import List, Optional
16
17from .client import ApiResult
18from .models import GameResponse
19
20SEVERITIES = ["info", "low", "medium", "high", "critical"]
21
22# Substrings that should never appear in player-facing prose: leaked engine
23# internals / .NET or Python stack traces.
24INTERNALS_MARKERS = [
25    "   at ", "System.", "GameEngine.", "Microsoft.AspNetCore",
26    "StackTrace", "Traceback (most recent call last)", "NullReferenceException",
27    "Unhandled exception", "InvalidOperationException", "at lambda_method",
28]
29# Markers that suggest the AI narrator broke character / leaked its prompt.
30NARRATOR_LEAK_MARKERS = [
31    "as an ai", "language model", "system prompt", "i cannot fulfill",
32    "i'm sorry, but i can", "openai", "gpt-", "as a narrator, i",
33    "you are a parser", "you are the narrator of", "i am an ai",
34]
35
36
37@dataclass
38class OracleHit:
39    oracle: str
40    layer: str          # L0 / L1 / L2
41    severity: str       # info..critical
42    category: str       # contract / consistency / narrator / scoring / parser
43    title: str
44    detail: str
45    evidence: str = ""
46
47    def __post_init__(self):
48        if self.severity not in SEVERITIES:
49            self.severity = "medium"
50
51
52_MOVE_WORDS = set(d.lower() for d in
53                  ["n", "s", "e", "w", "ne", "nw", "sw", "se", "u", "d",
54                   "up", "down", "in", "out", "north", "south", "east", "west",
55                   "northeast", "northwest", "southeast", "southwest", "enter",
56                   "exit", "go", "climb"])
57_FAIL_MOVE = re.compile(r"can(?:'|no)t go that way|cannot go|you can't go", re.I)
58
59
60def _clip(s: str, n: int = 280) -> str:
61    s = (s or "").strip().replace("\n", " ")
62    return s if len(s) <= n else s[:n] + " ..."
63
64
65# --------------------------------------------------------------------------
66# L0 -- contract
67# --------------------------------------------------------------------------
68def check_contract(result: ApiResult, max_score: int) -> List[OracleHit]:
69    hits: List[OracleHit] = []
70    if result.transport_error and result.status >= 500:
71        hits.append(OracleHit("transport", "L0", "critical", "contract",
72                              "Server error / transport failure",
73                              f"status={result.status} {result.transport_error}",
74                              _clip(result.raw_text)))
75    elif result.transport_error:
76        hits.append(OracleHit("transport", "L0", "high", "contract",
77                              "Transport failure",
78                              f"status={result.status} {result.transport_error}",
79                              _clip(result.raw_text)))
80    elif not result.ok:
81        hits.append(OracleHit("http_status", "L0", "high", "contract",
82                              f"Non-2xx HTTP status {result.status}",
83                              f"status={result.status}", _clip(result.raw_text)))
84
85    if result.raw_text and result.json is None and result.status < 500:
86        hits.append(OracleHit("malformed_json", "L0", "high", "contract",
87                              "Response body is not valid JSON object",
88                              result.parse_error or "no json", _clip(result.raw_text)))
89
90    text = result.raw_text or ""
91    low = text.lower()
92    for m in INTERNALS_MARKERS:
93        if m.lower() in low:
94            hits.append(OracleHit("leaked_internals", "L0", "critical", "contract",
95                                  "Engine internals / stack trace leaked to player",
96                                  f"marker={m!r}", _clip(text)))
97            break
98
99    g = result.game()
100    if g is not None:
101        if g.score < 0 or g.score > max_score:
102            hits.append(OracleHit("score_bounds", "L0", "high", "scoring",
103                                  f"Score {g.score} outside [0, {max_score}]",
104                                  f"score={g.score} max={max_score}"))
105        if g.moves < 0:
106            hits.append(OracleHit("moves_negative", "L0", "medium", "contract",
107                                  f"Negative move count {g.moves}", ""))
108        if result.method == "POST" and (g.response or "").strip() == "":
109            hits.append(OracleHit("empty_response", "L0", "medium", "narrator",
110                                  "Empty narrator response to a command", ""))
111        if (g.response or "").strip() in ("The narrator is silent.",
112                                          "Your companion says nothing."):
113            hits.append(OracleHit("silent_narrator", "L0", "medium", "narrator",
114                                  "Narrator fell back to silent placeholder",
115                                  "", _clip(g.response)))
116    return hits
117
118
119# --------------------------------------------------------------------------
120# L1 -- prose vs state
121# --------------------------------------------------------------------------
122def check_consistency(command: str, prev: Optional[GameResponse],
123                      cur: Optional[GameResponse]) -> List[OracleHit]:
124    hits: List[OracleHit] = []
125    if cur is None:
126        return hits
127    resp = cur.response or ""
128    low = resp.lower()
129    cmd = (command or "").strip().lower()
130    first = cmd.split()[0] if cmd else ""
131
132    if prev is not None:
133        # Moves should never go backwards.
134        if cur.moves < prev.moves:
135            hits.append(OracleHit("moves_regress", "L1", "medium", "contract",
136                                  "Move counter went backwards",
137                                  f"{prev.moves} -> {cur.moves}"))
138        # Score regressions are rare-but-legal (e.g. broken egg); flag low.
139        if cur.score < prev.score:
140            hits.append(OracleHit("score_regress", "L1", "low", "scoring",
141                                  "Score decreased -- verify it's intended",
142                                  f"{prev.score} -> {cur.score}", _clip(resp)))
143
144        # Movement consistency.
145        is_move = first in _MOVE_WORDS or cmd in _MOVE_WORDS
146        moved = cur.location_name != prev.location_name
147        if is_move and not moved and not _FAIL_MOVE.search(resp) and cur.moves > prev.moves:
148            # Some legit non-moves (blocked by gate w/ custom msg) exist -> low.
149            hits.append(OracleHit("move_no_change", "L1", "low", "consistency",
150                                  "Movement command consumed a turn but location "
151                                  "did not change and no 'can't go' message shown",
152                                  f"loc={cur.location_name!r}", _clip(resp)))
153        if moved and cur.previous_location_name not in (None, prev.location_name):
154            hits.append(OracleHit("prevloc_mismatch", "L1", "low", "consistency",
155                                  "previousLocationName disagrees with actual prior location",
156                                  f"prev={prev.location_name!r} "
157                                  f"reported={cur.previous_location_name!r}"))
158
159        # Take / drop vs inventory size.
160        inv_delta = len(cur.inventory) - len(prev.inventory)
161        if re.search(r"\btaken\b", low) and inv_delta <= 0 and "drop" not in cmd:
162            hits.append(OracleHit("take_no_inv", "L1", "medium", "consistency",
163                                  "Prose says 'Taken' but inventory did not grow",
164                                  f"inv {len(prev.inventory)} -> {len(cur.inventory)}",
165                                  _clip(resp)))
166        if re.search(r"\bdropped\b", low) and inv_delta >= 0 and first not in ("take", "get"):
167            hits.append(OracleHit("drop_no_inv", "L1", "medium", "consistency",
168                                  "Prose says 'Dropped' but inventory did not shrink",
169                                  f"inv {len(prev.inventory)} -> {len(cur.inventory)}",
170                                  _clip(resp)))
171    return hits
172
173
174# --------------------------------------------------------------------------
175# L2 -- anchors + narrator-leak heuristics
176# --------------------------------------------------------------------------
177def check_anchors(result: ApiResult) -> List[OracleHit]:
178    hits: List[OracleHit] = []
179    g = result.game()
180    if g is None:
181        return hits
182    low = (g.response or "").lower()
183    for m in NARRATOR_LEAK_MARKERS:
184        if m in low:
185            hits.append(OracleHit("narrator_leak", "L2", "high", "narrator",
186                                  "Narrator may have broken character / leaked meta",
187                                  f"marker={m!r}", _clip(g.response)))
188            break
189    return hits
190
191
192def run_all(command: str, result: ApiResult, prev: Optional[GameResponse],
193            max_score: int) -> List[OracleHit]:
194    hits = []
195    hits += check_contract(result, max_score)
196    hits += check_consistency(command, prev, result.game())
197    hits += check_anchors(result)
198    return hits

1"""Deterministic oracles -- the free, no-LLM bug detectors.
2
3L0  contract     : transport/HTTP/JSON/score-bounds/leaked-internals checks on the raw result.
4L1  consistency  : the narrator prose vs the structured state envelope.
5L2  anchors      : known static strings + narrator-leak heuristics.
6
7Each oracle returns OracleHit objects. They never raise; a noisy oracle is
8tuned down to low/info, not removed, so the interactive driver (Claude) makes
9the final call on whether a hit is a real bug.
10"""
11from __future__ import annotations
12
13import re
14from dataclasses import dataclass, field
15from typing import List, Optional
16
17from .client import ApiResult
18from .models import GameResponse
19
20SEVERITIES = ["info", "low", "medium", "high", "critical"]
21
22# Substrings that should never appear in player-facing prose: leaked engine
23# internals / .NET or Python stack traces.
24INTERNALS_MARKERS = [
25    "   at ", "System.", "GameEngine.", "Microsoft.AspNetCore",
26    "StackTrace", "Traceback (most recent call last)", "NullReferenceException",
27    "Unhandled exception", "InvalidOperationException", "at lambda_method",
28]
29# Markers that suggest the AI narrator broke character / leaked its prompt.
30NARRATOR_LEAK_MARKERS = [
31    "as an ai", "language model", "system prompt", "i cannot fulfill",
32    "i'm sorry, but i can", "openai", "gpt-", "as a narrator, i",
33    "you are a parser", "you are the narrator of", "i am an ai",
34]
35
36
37@dataclass
38class OracleHit:
39    oracle: str
40    layer: str          # L0 / L1 / L2
41    severity: str       # info..critical
42    category: str       # contract / consistency / narrator / scoring / parser
43    title: str
44    detail: str
45    evidence: str = ""
46
47    def __post_init__(self):
48        if self.severity not in SEVERITIES:
49            self.severity = "medium"
50
51
52_MOVE_WORDS = set(d.lower() for d in
53                  ["n", "s", "e", "w", "ne", "nw", "sw", "se", "u", "d",
54                   "up", "down", "in", "out", "north", "south", "east", "west",
55                   "northeast", "northwest", "southeast", "southwest", "enter",
56                   "exit", "go", "climb"])
57_FAIL_MOVE = re.compile(r"can(?:'|no)t go that way|cannot go|you can't go", re.I)
58
59
60def _clip(s: str, n: int = 280) -> str:
61    s = (s or "").strip().replace("\n", " ")
62    return s if len(s) <= n else s[:n] + " ..."
63
64
65# --------------------------------------------------------------------------
66# L0 -- contract
67# --------------------------------------------------------------------------
68def check_contract(result: ApiResult, max_score: int) -> List[OracleHit]:
69    hits: List[OracleHit] = []
70    if result.transport_error and result.status >= 500:
71        hits.append(OracleHit("transport", "L0", "critical", "contract",
72                              "Server error / transport failure",
73                              f"status={result.status} {result.transport_error}",
74                              _clip(result.raw_text)))
75    elif result.transport_error:
76        hits.append(OracleHit("transport", "L0", "high", "contract",
77                              "Transport failure",
78                              f"status={result.status} {result.transport_error}",
79                              _clip(result.raw_text)))
80    elif not result.ok:
81        hits.append(OracleHit("http_status", "L0", "high", "contract",
82                              f"Non-2xx HTTP status {result.status}",
83                              f"status={result.status}", _clip(result.raw_text)))
84
85    if result.raw_text and result.json is None and result.status < 500:
86        hits.append(OracleHit("malformed_json", "L0", "high", "contract",
87                              "Response body is not valid JSON object",
88                              result.parse_error or "no json", _clip(result.raw_text)))
89
90    text = result.raw_text or ""
91    low = text.lower()
92    for m in INTERNALS_MARKERS:
93        if m.lower() in low:
94            hits.append(OracleHit("leaked_internals", "L0", "critical", "contract",
95                                  "Engine internals / stack trace leaked to player",
96                                  f"marker={m!r}", _clip(text)))
97            break
98
99    g = result.game()
100    if g is not None:
101        if g.score < 0 or g.score > max_score:
102            hits.append(OracleHit("score_bounds", "L0", "high", "scoring",
103                                  f"Score {g.score} outside [0, {max_score}]",
104                                  f"score={g.score} max={max_score}"))
105        if g.moves < 0:
106            hits.append(OracleHit("moves_negative", "L0", "medium", "contract",
107                                  f"Negative move count {g.moves}", ""))
108        if result.method == "POST" and (g.response or "").strip() == "":
109            hits.append(OracleHit("empty_response", "L0", "medium", "narrator",
110                                  "Empty narrator response to a command", ""))
111        if (g.response or "").strip() in ("The narrator is silent.",
112                                          "Your companion says nothing."):
113            hits.append(OracleHit("silent_narrator", "L0", "medium", "narrator",
114                                  "Narrator fell back to silent placeholder",
115                                  "", _clip(g.response)))
116    return hits
117
118
119# --------------------------------------------------------------------------
120# L1 -- prose vs state
121# --------------------------------------------------------------------------
122def check_consistency(command: str, prev: Optional[GameResponse],
123                      cur: Optional[GameResponse]) -> List[OracleHit]:
124    hits: List[OracleHit] = []
125    if cur is None:
126        return hits
127    resp = cur.response or ""
128    low = resp.lower()
129    cmd = (command or "").strip().lower()
130    first = cmd.split()[0] if cmd else ""
131
132    if prev is not None:
133        # Moves should never go backwards.
134        if cur.moves < prev.moves:
135            hits.append(OracleHit("moves_regress", "L1", "medium", "contract",
136                                  "Move counter went backwards",
137                                  f"{prev.moves} -> {cur.moves}"))
138        # Score regressions are rare-but-legal (e.g. broken egg); flag low.
139        if cur.score < prev.score:
140            hits.append(OracleHit("score_regress", "L1", "low", "scoring",
141                                  "Score decreased -- verify it's intended",
142                                  f"{prev.score} -> {cur.score}", _clip(resp)))
143
144        # Movement consistency.
145        is_move = first in _MOVE_WORDS or cmd in _MOVE_WORDS
146        moved = cur.location_name != prev.location_name
147        if is_move and not moved and not _FAIL_MOVE.search(resp) and cur.moves > prev.moves:
148            # Some legit non-moves (blocked by gate w/ custom msg) exist -> low.
149            hits.append(OracleHit("move_no_change", "L1", "low", "consistency",
150                                  "Movement command consumed a turn but location "
151                                  "did not change and no 'can't go' message shown",
152                                  f"loc={cur.location_name!r}", _clip(resp)))
153        if moved and cur.previous_location_name not in (None, prev.location_name):
154            hits.append(OracleHit("prevloc_mismatch", "L1", "low", "consistency",
155                                  "previousLocationName disagrees with actual prior location",
156                                  f"prev={prev.location_name!r} "
157                                  f"reported={cur.previous_location_name!r}"))
158
159        # Take / drop vs inventory size.
160        inv_delta = len(cur.inventory) - len(prev.inventory)
161        if re.search(r"\btaken\b", low) and inv_delta <= 0 and "drop" not in cmd:
162            hits.append(OracleHit("take_no_inv", "L1", "medium", "consistency",
163                                  "Prose says 'Taken' but inventory did not grow",
164                                  f"inv {len(prev.inventory)} -> {len(cur.inventory)}",
165                                  _clip(resp)))
166        if re.search(r"\bdropped\b", low) and inv_delta >= 0 and first not in ("take", "get"):
167            hits.append(OracleHit("drop_no_inv", "L1", "medium", "consistency",
168                                  "Prose says 'Dropped' but inventory did not shrink",
169                                  f"inv {len(prev.inventory)} -> {len(cur.inventory)}",
170                                  _clip(resp)))
171    return hits
172
173
174# --------------------------------------------------------------------------
175# L2 -- anchors + narrator-leak heuristics
176# --------------------------------------------------------------------------
177def check_anchors(result: ApiResult) -> List[OracleHit]:
178    hits: List[OracleHit] = []
179    g = result.game()
180    if g is None:
181        return hits
182    low = (g.response or "").lower()
183    for m in NARRATOR_LEAK_MARKERS:
184        if m in low:
185            hits.append(OracleHit("narrator_leak", "L2", "high", "narrator",
186                                  "Narrator may have broken character / leaked meta",
187                                  f"marker={m!r}", _clip(g.response)))
188            break
189    return hits
190
191
192def run_all(command: str, result: ApiResult, prev: Optional[GameResponse],
193            max_score: int) -> List[OracleHit]:
194    hits = []
195    hits += check_contract(result, max_score)
196    hits += check_consistency(command, prev, result.game())
197    hits += check_anchors(result)
198    return hits