Why a system prompt can’t enforce a tool allowlist or a budget
Because a system prompt is a request the model can choose to follow, not a control the runtime enforces. OWASP’s LLM06:2025 Excessive Agency (opens in a new tab) names excessive functionality, excessive permissions and excessive autonomy as the root causes, and its mitigation is blunt: authorization belongs in the downstream system, not in the LLM’s judgment.
Anthropic’s own agent-design guidance (opens in a new tab) recommends stopping conditions such as a maximum iteration count, plus human checkpoints before higher-risk actions. It names no enforcement mechanism for either. That’s left to the code wrapped around the model.
The gap is concrete in the tool_result lifecycle (opens in a new tab): a failed tool call comes back with is_error: true, and nothing in the Messages API stops the model calling the same tool again. Anthropic’s docs note the model typically retries two or three times on its own before giving up — a courtesy, not a limit you can rely on.
sequenceDiagram
participant Model
participant Loop as Agent loop
participant Tool
Model->>Loop: tool_use delete_record(id=42)
Loop->>Tool: execute delete_record
Tool-->>Loop: throws (permission denied)
Loop->>Model: tool_result is_error=true
Model->>Loop: tool_use delete_record(id=42) - retry
Loop->>Tool: execute delete_record
Tool-->>Loop: throws again
Loop->>Model: tool_result is_error=true
Note over Model,Loop: Nothing in the Messages API stops this on its own
Three independent Show HN projects posted within days of each other converge on the same fix from different angles. ModelFuzz (opens in a new tab) validates tool arguments before they reach a function. Wattage (opens in a new tab) gates CI on token-cost regressions. SHACKLE’s SP/1.0 (opens in a new tab) trips a circuit breaker on budget or repeat-call thresholds. This article builds all four controls — a hard budget, a default-deny allowlist, a circuit breaker and a kill switch — around one loop, in that order.
Scaffold a minimal tool-calling agent loop in TypeScript
Start from an ordinary Claude tool-use loop with no guardrails, then wrap it in the rest of the article. The example uses @anthropic-ai/sdk and one mock tool, get_weather, so it runs without a real backend.
The shape is the same for any tool-calling agent: send the conversation, check stop_reason, and if it’s tool_use, execute the tool and append a tool_result block before sending again. The loop ends when stop_reason comes back as end_turn. See Tool use with Claude (opens in a new tab) for the full content-block reference.
Every response carries a usage object with input_tokens and output_tokens — the numbers every guardrail below consumes. Log them now; the budget guard in the next section does nothing more than accumulate what’s already on the wire.
Keep tool dispatch as one function. Sections 4 and 5 wrap it with an allowlist check and a circuit breaker, and section 6 threads an abort signal through it, so it has to stay the single choke point every tool call passes through.
// src/agent.ts
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();
type Message = { role: "user" | "assistant"; content: any };
const tools = [
{
name: "get_weather",
description: "Get the current weather for a city",
input_schema: {
type: "object",
properties: { city: { type: "string" } },
required: ["city"],
},
},
];
async function callTool(name: string, input: Record<string, unknown>): Promise<string> {
if (name === "get_weather") {
return JSON.stringify({ city: input.city, tempC: 19, condition: "cloudy" });
}
throw new Error(`unknown tool: ${name}`);
}
export async function runLoop(userMessage: string): Promise<Message[]> {
const messages: Message[] = [{ role: "user", content: userMessage }];
while (true) {
const response = await client.messages.create({
model: "claude-sonnet-4-5",
max_tokens: 512,
tools,
messages: messages as any,
});
console.log("usage:", response.usage.input_tokens, response.usage.output_tokens);
messages.push({ role: "assistant", content: response.content });
if (response.stop_reason !== "tool_use") return messages;
const toolResults = [];
for (const block of response.content) {
if (block.type === "tool_use") {
const result = await callTool(block.name, block.input as Record<string, unknown>);
toolResults.push({ type: "tool_result", tool_use_id: block.id, content: result });
}
}
messages.push({ role: "user", content: toolResults });
}
}
runLoop("What's the weather in Lisbon?").then((messages) => {
console.log(JSON.stringify(messages, null, 2));
});This article is tested against Node 24.4.0 (opens in a new tab) and TypeScript 5.9.3 (opens in a new tab). Pin @anthropic-ai/sdk and zod to whatever’s current when you set this up — those two aren’t version-pinned by this article.
{
"name": "agent-guardrails",
"version": "1.0.0",
"type": "module",
"engines": { "node": ">=24.4.0" },
"scripts": {
"build": "tsc",
"start": "node dist/agent.js",
"test": "node --test dist"
},
"dependencies": {
"@anthropic-ai/sdk": "^0.32.0",
"zod": "^3.23.0"
},
"devDependencies": {
"typescript": "5.9.3"
}
}{
"compilerOptions": {
"target": "ES2023",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"rootDir": "src",
"outDir": "dist",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true
},
"include": ["src/**/*.ts"]
}npm install
npm run build
npm startExpected output:
You should now see: a finished conversation array with one tool call, one tool result and a final assistant turn — the shape every guardrail below attaches to, not the content itself, which depends on the model’s response.
Enforce a hard token and cost budget that halts the run
A budget guard has one job: turn the usage numbers from the previous section into a running dollar total, and refuse to start another turn once that total crosses a cap. Price it per model, since input and output tokens aren’t priced the same.
The check has to run before the next turn is dispatched, not after the response comes back — by then the tokens are already spent. hasRemaining() is that pre-flight check; recordUsage() is what feeds it after every response.
That ordering has a hole worth naming directly. A check that runs before a turn can’t stop the turn already in flight: if the model’s completion runs to the full max_tokens, its cost posts to the total after the check that let it through already passed. The mitigation is capping max_tokens per turn, so the worst-case overshoot is bounded to one turn’s worth of tokens instead of unbounded.
When the cap is crossed, throw a BudgetExceededError rather than truncating output or quietly returning nothing. A silent truncation looks like a model failure to whatever consumes the conversation; a thrown, named error is something calling code can catch — retry with a fresh budget, page an operator, or stop.
// src/budget.ts
export class BudgetExceededError extends Error {
constructor(spent: number, cap: number) {
super(`budget exceeded: spent $${spent.toFixed(4)} of a $${cap.toFixed(4)} cap`);
this.name = "BudgetExceededError";
}
}
const PRICE_PER_MILLION_TOKENS = {
"claude-sonnet-4-5": { input: 3, output: 15 },
} as const;
export class BudgetGuard {
private spentUsd = 0;
constructor(
private readonly capUsd: number,
private readonly model: keyof typeof PRICE_PER_MILLION_TOKENS = "claude-sonnet-4-5"
) {}
recordUsage(usage: { input_tokens: number; output_tokens: number }): void {
const price = PRICE_PER_MILLION_TOKENS[this.model];
this.spentUsd +=
(usage.input_tokens / 1_000_000) * price.input +
(usage.output_tokens / 1_000_000) * price.output;
}
hasRemaining(): boolean {
return this.spentUsd < this.capUsd;
}
get spent(): number {
return this.spentUsd;
}
get cap(): number {
return this.capUsd;
}
}// src/agent.ts — checked immediately before every new turn is sent
import Anthropic from "@anthropic-ai/sdk";
import { BudgetGuard, BudgetExceededError } from "./budget.js";
const client = new Anthropic();
type Message = { role: "user" | "assistant"; content: any };
const tools = [
{
name: "get_weather",
description: "Get the current weather for a city",
input_schema: {
type: "object",
properties: { city: { type: "string" } },
required: ["city"],
},
},
];
async function callTool(name: string, input: Record<string, unknown>): Promise<string> {
if (name === "get_weather") {
return JSON.stringify({ city: input.city, tempC: 19, condition: "cloudy" });
}
throw new Error(`unknown tool: ${name}`);
}
export async function runLoop(userMessage: string, budget: BudgetGuard): Promise<Message[]> {
const messages: Message[] = [{ role: "user", content: userMessage }];
while (true) {
if (!budget.hasRemaining()) {
throw new BudgetExceededError(budget.spent, budget.cap);
}
const response = await client.messages.create({
model: "claude-sonnet-4-5",
max_tokens: 512,
tools,
messages: messages as any,
});
budget.recordUsage(response.usage);
messages.push({ role: "assistant", content: response.content });
if (response.stop_reason !== "tool_use") return messages;
const toolResults = [];
for (const block of response.content) {
if (block.type === "tool_use") {
const result = await callTool(block.name, block.input as Record<string, unknown>);
toolResults.push({ type: "tool_result", tool_use_id: block.id, content: result });
}
}
messages.push({ role: "user", content: toolResults });
}
}
const budget = new BudgetGuard(0.5);
runLoop("What's the weather in Lisbon?", budget).then((messages) => {
console.log(JSON.stringify(messages, null, 2));
});// src/budget.test.ts
import { test } from "node:test";
import assert from "node:assert/strict";
import { BudgetGuard } from "./budget.js";
test("hasRemaining() goes false once cumulative spend crosses the cap", () => {
const guard = new BudgetGuard(0.001);
assert.equal(guard.hasRemaining(), true);
guard.recordUsage({ input_tokens: 2000, output_tokens: 2000 });
assert.equal(guard.hasRemaining(), false);
});
test("a single in-flight turn can overshoot the cap by up to one max_tokens response", () => {
const capUsd = 0.001;
const maxTokensPerTurn = 512;
const guard = new BudgetGuard(capUsd);
assert.equal(guard.hasRemaining(), true);
guard.recordUsage({ input_tokens: 500, output_tokens: maxTokensPerTurn });
const overshootUsd = guard.spent - capUsd;
assert.ok(overshootUsd > 0, `expected an overshoot, got spent=${guard.spent}`);
console.log(`in-flight turn overshoot: $${overshootUsd.toFixed(4)}`);
});npm testExpected output — Node’s test runner prints a checkmark per passing test plus a pass/fail tally; exact formatting and timings vary by reporter, but the console.log line below is deterministic arithmetic:
✔ hasRemaining() goes false once cumulative spend crosses the cap
✔ a single in-flight turn can overshoot the cap by up to one max_tokens response
in-flight turn overshoot: $0.0082
# pass 2
# fail 0The second test computes exactly how far over budget one in-flight turn can push the total, using a fixed max_tokens and input size. At $3/$15 per million tokens, a 512-token turn can overshoot a very small cap by roughly $0.008 — scale your own max_tokens cap against your actual budget using this number, not a guess.
You should now see: both tests pass, and the second logs the dollar amount one turn overshot the cap by — the figure to check against your own max_tokens setting.
Build a default-deny tool allowlist with argument validation
Default-deny means a tool runs only if it’s explicitly marked allowed: true — the same first-violation-stops-the-call approach ModelFuzz’s policy engine (opens in a new tab) uses, where each policy inspects the call and the engine halts at the first one that objects.
Argument validation is the other half: a schema check with zod before the underlying function ever sees the input, so a call with the right tool name but a malformed or unexpected field is rejected rather than passed through. This is where “authorize in the downstream system” becomes a line of code, not a principle.
Mark destructive tools — delete_record, send_email — as allowed: false by default, even when the model calls them with perfectly valid arguments. That’s the gap a prompt instruction like “never delete without confirmation” can’t close on its own: the instruction has no enforcement point, and this does.
Blocked calls return a structured tool_result with is_error: true, the same shape a normal tool failure uses, rather than throwing out of the loop. The model sees a denial and can react — ask for confirmation, try something else — instead of the run crashing on a call it was never going to be allowed to make.
// src/policy.ts
import { z } from "zod";
export interface ToolPolicy {
allowed: boolean;
schema: z.ZodType;
}
export class AllowlistPolicy {
constructor(private readonly policies: Record<string, ToolPolicy>) {}
check(
name: string,
input: unknown
): { ok: true; input: unknown } | { ok: false; reason: string } {
const rule = this.policies[name];
if (!rule || !rule.allowed) {
return { ok: false, reason: `tool "${name}" is not on the allowlist` };
}
const parsed = rule.schema.safeParse(input);
if (!parsed.success) {
return { ok: false, reason: `invalid arguments for "${name}": ${parsed.error.message}` };
}
return { ok: true, input: parsed.data };
}
}
export const policy = new AllowlistPolicy({
get_weather: { allowed: true, schema: z.object({ city: z.string() }) },
delete_record: { allowed: false, schema: z.object({ id: z.string() }) },
});// src/dispatch.ts
import { policy } from "./policy.js";
type ToolFn = (input: any) => Promise<string>;
const toolFunctions: Record<string, ToolFn> = {
get_weather: async (input) => JSON.stringify({ city: input.city, tempC: 19 }),
delete_record: async (input) => `deleted record ${input.id}`,
};
export async function dispatchTool(
name: string,
input: unknown
): Promise<{ content: string; is_error?: true }> {
const check = policy.check(name, input);
if (!check.ok) {
return { content: check.reason, is_error: true };
}
const fn = toolFunctions[name];
if (!fn) {
return { content: `no implementation registered for "${name}"`, is_error: true };
}
const content = await fn(check.input);
return { content };
}// src/dispatch.test.ts
import { test } from "node:test";
import assert from "node:assert/strict";
import { dispatchTool } from "./dispatch.js";
test("a disallowed tool never reaches the underlying function", async () => {
const result = await dispatchTool("delete_record", { id: "row-42" });
assert.equal(result.is_error, true);
assert.match(result.content, /not on the allowlist/);
});
test("invalid arguments are rejected before the tool function runs", async () => {
const result = await dispatchTool("get_weather", { city: 12345 });
assert.equal(result.is_error, true);
assert.match(result.content, /invalid arguments/);
});Expected output:
✔ a disallowed tool never reaches the underlying function
✔ invalid arguments are rejected before the tool function runs
# pass 2
# fail 0Both tests assert something stronger than “the call returned an error”: they assert toolFunctions[name] never runs. For delete_record, the mock delete never executes; for get_weather, the malformed city: 12345 never reaches the function that would have stringified it.
You should now see: two passing tests, with the disallowed-tool and invalid-argument cases both short-circuiting inside dispatchTool before toolFunctions[name] is ever called.
Trip a circuit breaker on repeated tool failures
The canonical breaker is a three-state machine, laid out in Microsoft’s Circuit Breaker pattern reference (opens in a new tab): Closed, where calls pass through and failures count against a threshold; Open, where calls fail immediately and a reset timer starts; and Half-Open, where one trial call decides whether to reset to Closed or revert to Open.
stateDiagram-v2
[*] --> Closed
Closed --> Open: failures >= threshold
Open --> HalfOpen: reset timeout elapses
HalfOpen --> Closed: trial call succeeds
HalfOpen --> Open: trial call fails
Count a failure whenever a tool_result comes back with is_error: true, keyed by tool name, so a broken call_api doesn’t also block get_weather. Three consecutive failures trips that tool to Open; further calls short-circuit immediately, without touching the underlying function, until the reset timeout elapses.
Microsoft’s own guidance is worth repeating: circuit breakers add overhead without benefit for in-memory or local-only resources, where a call either always succeeds or reflects a bug a breaker won’t fix. Reserve this for tools calling something external, remote or costly — a paid API, a flaky webhook — not a pure function in the same process.
The state is process-local. A restarted worker, or a second instance behind a load balancer, starts back at Closed with no memory of what the first instance counted — this isn’t a substitute for a shared rate limit sitting in front of the same tools.
// src/breaker.ts
type State = "closed" | "open" | "half-open";
export class CircuitBreaker {
private state = new Map<string, State>();
private failures = new Map<string, number>();
private openedAt = new Map<string, number>();
constructor(
private readonly failureThreshold = 3,
private readonly resetTimeoutMs = 30_000
) {}
private stateFor(tool: string): State {
const current = this.state.get(tool) ?? "closed";
if (current === "open") {
const openedAt = this.openedAt.get(tool) ?? 0;
if (Date.now() - openedAt >= this.resetTimeoutMs) {
this.state.set(tool, "half-open");
return "half-open";
}
}
return current;
}
isOpen(tool: string): boolean {
return this.stateFor(tool) === "open";
}
recordFailure(tool: string): void {
const count = (this.failures.get(tool) ?? 0) + 1;
this.failures.set(tool, count);
if (count >= this.failureThreshold) {
this.state.set(tool, "open");
this.openedAt.set(tool, Date.now());
}
}
recordSuccess(tool: string): void {
this.failures.set(tool, 0);
this.state.set(tool, "closed");
}
}
export const breaker = new CircuitBreaker(3, 30_000);// src/dispatch.ts — extended with the circuit breaker
import { policy } from "./policy.js";
import { breaker } from "./breaker.js";
type ToolFn = (input: any) => Promise<string>;
const toolFunctions: Record<string, ToolFn> = {
get_weather: async (input) => JSON.stringify({ city: input.city, tempC: 19 }),
delete_record: async (input) => `deleted record ${input.id}`,
};
export async function dispatchTool(
name: string,
input: unknown
): Promise<{ content: string; is_error?: true }> {
if (breaker.isOpen(name)) {
return { content: `circuit open for "${name}" after repeated failures`, is_error: true };
}
const check = policy.check(name, input);
if (!check.ok) {
return { content: check.reason, is_error: true };
}
const fn = toolFunctions[name];
if (!fn) {
return { content: `no implementation registered for "${name}"`, is_error: true };
}
try {
const content = await fn(check.input);
breaker.recordSuccess(name);
return { content };
} catch (err) {
breaker.recordFailure(name);
return { content: String(err), is_error: true };
}
}// src/breaker.test.ts
import { test, mock } from "node:test";
import assert from "node:assert/strict";
import { CircuitBreaker } from "./breaker.js";
test("breaker opens after 3 consecutive failures and short-circuits the 4th call", () => {
const breaker = new CircuitBreaker(3, 30_000);
const flaky = mock.fn(() => {
throw new Error("upstream timeout");
});
for (let i = 0; i < 3; i++) {
assert.equal(breaker.isOpen("call_api"), false);
try {
flaky();
} catch {
breaker.recordFailure("call_api");
}
}
assert.equal(breaker.isOpen("call_api"), true);
if (!breaker.isOpen("call_api")) {
flaky();
}
assert.equal(flaky.mock.callCount(), 3);
});Expected output:
✔ breaker opens after 3 consecutive failures and short-circuits the 4th call
# pass 1
# fail 0You should now see: the mock tool function invoked exactly three times, then the fourth call blocked by isOpen() before flaky() runs again — confirmed by mock.fn’s call count.
Wire a kill switch with AbortController across the loop
Node ships everything this needs already: AbortController, the static AbortSignal.timeout(delay), and AbortSignal.any(signals) (opens in a new tab) — all built in, no extra package.
AbortSignal.any() is what turns three unrelated triggers into one kill switch. Feed it the manual controller’s signal, a signal the budget guard aborts on BudgetExceededError, and a per-turn timeout; whichever fires first supplies the combined signal’s abort reason. The rest of the loop only has to check one signal.
flowchart LR
K["Manual kill switch"] --> A["AbortSignal.any()"]
B["BudgetGuard abort"] --> A
C["CircuitBreaker abort"] --> A
T["Per-turn timeout"] --> A
A --> M["Messages API call"]
A --> D["dispatchTool - fetch / subprocess"]
Pass the combined signal into both the Messages API call and every tool execution — not just the outer while loop. A real tool would do await fetch(url, { signal }) or pass { signal } to a subprocess spawn; a fetch or subprocess started before the abort fires keeps running unless it was handed the signal directly.
Handle SIGINT by calling .abort() on the manual controller, not by letting the process exit. An unhandled Ctrl+C kills the Node process but not a detached child process or an in-flight request to another host — the abort signal is what tells those to stop.
// src/kill-switch.ts
export function composeSignals(signals: AbortSignal[]): AbortSignal {
return AbortSignal.any(signals);
}
export function createKillSwitch(): { signal: AbortSignal; abort: (reason?: string) => void } {
const controller = new AbortController();
process.once("SIGINT", () => {
controller.abort("SIGINT: manual kill switch triggered");
});
return {
signal: controller.signal,
abort: (reason?: string) => controller.abort(reason ?? "manual kill switch"),
};
}// src/policy.ts — slow_task added as a stand-in for a slow network call
import { z } from "zod";
export interface ToolPolicy {
allowed: boolean;
schema: z.ZodType;
}
export class AllowlistPolicy {
constructor(private readonly policies: Record<string, ToolPolicy>) {}
check(
name: string,
input: unknown
): { ok: true; input: unknown } | { ok: false; reason: string } {
const rule = this.policies[name];
if (!rule || !rule.allowed) {
return { ok: false, reason: `tool "${name}" is not on the allowlist` };
}
const parsed = rule.schema.safeParse(input);
if (!parsed.success) {
return { ok: false, reason: `invalid arguments for "${name}": ${parsed.error.message}` };
}
return { ok: true, input: parsed.data };
}
}
export const policy = new AllowlistPolicy({
get_weather: { allowed: true, schema: z.object({ city: z.string() }) },
delete_record: { allowed: false, schema: z.object({ id: z.string() }) },
slow_task: { allowed: true, schema: z.object({}) },
});// src/dispatch.ts — signal threaded through, aborts rethrown instead of swallowed
import { policy } from "./policy.js";
import { breaker } from "./breaker.js";
type ToolFn = (input: any, signal: AbortSignal) => Promise<string>;
const toolFunctions: Record<string, ToolFn> = {
get_weather: async (input) => JSON.stringify({ city: input.city, tempC: 19 }),
delete_record: async (input) => `deleted record ${input.id}`,
slow_task: (_input, signal) =>
new Promise<string>((resolve, reject) => {
if (signal.aborted) return reject(signal.reason);
const timer = setTimeout(() => resolve("done"), 5_000);
signal.addEventListener(
"abort",
() => {
clearTimeout(timer);
reject(signal.reason);
},
{ once: true }
);
}),
};
export async function dispatchTool(
name: string,
input: unknown,
signal: AbortSignal
): Promise<{ content: string; is_error?: true }> {
if (signal.aborted) {
return { content: `aborted before dispatch of "${name}"`, is_error: true };
}
if (breaker.isOpen(name)) {
return { content: `circuit open for "${name}" after repeated failures`, is_error: true };
}
const check = policy.check(name, input);
if (!check.ok) {
return { content: check.reason, is_error: true };
}
const fn = toolFunctions[name];
if (!fn) {
return { content: `no implementation registered for "${name}"`, is_error: true };
}
try {
const content = await fn(check.input, signal);
breaker.recordSuccess(name);
return { content };
} catch (err) {
if (signal.aborted) {
throw err;
}
breaker.recordFailure(name);
return { content: String(err), is_error: true };
}
}// src/agent.ts — the kill switch, budget guard and per-turn timeout combined into one signal
import Anthropic from "@anthropic-ai/sdk";
import { BudgetGuard, BudgetExceededError } from "./budget.js";
import { dispatchTool } from "./dispatch.js";
import { composeSignals, createKillSwitch } from "./kill-switch.js";
const client = new Anthropic();
type Message = { role: "user" | "assistant"; content: any };
const tools = [
{
name: "get_weather",
description: "Get the current weather for a city",
input_schema: {
type: "object",
properties: { city: { type: "string" } },
required: ["city"],
},
},
];
export async function runLoop(
userMessage: string,
budget: BudgetGuard,
killSwitch: AbortSignal
): Promise<Message[]> {
const messages: Message[] = [{ role: "user", content: userMessage }];
const budgetController = new AbortController();
while (true) {
if (!budget.hasRemaining()) {
budgetController.abort("budget exceeded");
throw new BudgetExceededError(budget.spent, budget.cap);
}
const turnSignal = composeSignals([
killSwitch,
budgetController.signal,
AbortSignal.timeout(60_000),
]);
const response = await client.messages.create(
{ model: "claude-sonnet-4-5", max_tokens: 512, tools, messages: messages as any },
{ signal: turnSignal }
);
budget.recordUsage(response.usage);
messages.push({ role: "assistant", content: response.content });
if (response.stop_reason !== "tool_use") return messages;
const toolResults = [];
for (const block of response.content) {
if (block.type === "tool_use") {
if (turnSignal.aborted) break;
const result = await dispatchTool(block.name, block.input, turnSignal);
toolResults.push({ type: "tool_result", tool_use_id: block.id, ...result });
}
}
messages.push({ role: "user", content: toolResults });
}
}
const { signal: killSignal } = createKillSwitch();
const budget = new BudgetGuard(0.5);
runLoop("What's the weather in Lisbon?", budget, killSignal).then((messages) => {
console.log(JSON.stringify(messages, null, 2));
});// src/kill-switch.test.ts
import { test } from "node:test";
import assert from "node:assert/strict";
import { dispatchTool } from "./dispatch.js";
test("aborting mid-run rejects the in-flight call with an AbortError and blocks the next dispatch", async () => {
const controller = new AbortController();
const inFlight = dispatchTool("slow_task", {}, controller.signal);
controller.abort();
await assert.rejects(inFlight, (err: any) => err.name === "AbortError");
const after = await dispatchTool("get_weather", { city: "Porto" }, controller.signal);
assert.equal(after.is_error, true);
assert.match(after.content, /aborted before dispatch/);
});Expected output:
✔ aborting mid-run rejects the in-flight call with an AbortError and blocks the next dispatch
# pass 1
# fail 0The test aborts mid-run and checks two things: the in-flight slow_task call rejects with an AbortError rather than resolving, and a second call issued afterward is blocked before it reaches the allowlist check or the tool function — confirmed by the "aborted before dispatch" message rather than a network or breaker error.
You should now see: the in-flight call reject, the second call blocked before dispatch, and zero tool executions after .abort() is called.
Know what this guardrail layer does not cover
None of this is a complete safety net, and none of the projects it draws on claim otherwise. SHACKLE’s own documentation (opens in a new tab) is direct about it: LLM orchestration is non-deterministic, and SHACKLE is a best-effort circuit breaker that does not guarantee preventing all API spend overruns. The same caveat applies to BudgetGuard — it bounds the overshoot, it doesn’t eliminate it.
The budget total and the breaker’s failure counts are both process-local, held in a class field and a Map. A distributed deployment — a worker pool, a queue of retried jobs, anything horizontally scaled — needs a shared store such as Redis for either number to mean anything across instances. Otherwise each worker enforces its own cap against its own slice of the spend.
None of these four controls stop a model from choosing a bad action it’s actually permitted to take. The allowlist restricts which tools can run, not whether calling send_email with these particular arguments right now is a good idea.
Two things this article didn’t test: spend attribution across a multi-agent or sub-agent setup, and cost estimation for streamed or unpriced, newly released models — the case Wattage (opens in a new tab) signals with a dedicated exit code rather than guessing.
flowchart TB
subgraph InProcess["Covered here - single process"]
Budget["Token / cost budget"]
Allow["Tool allowlist + schema"]
Breaker["Per-tool circuit breaker"]
Kill["AbortController kill switch"]
end
subgraph Beyond["Needs more infrastructure"]
Shared["Shared budget / failure store - e.g. Redis"]
Judgment["Judging whether an allowed action is wise"]
MultiAgent["Multi-agent spend attribution"]
Pricing["Cost for streamed / unpriced models"]
end
InProcess -.->|does not extend to| Beyond
Limitations
- The budget cap and circuit breaker state are process-local; a restart or a second worker instance resets both to zero, so they don’t hold across a distributed deployment without a shared store.
- The pre-turn budget check can’t stop a turn already in flight — cap
max_tokensper turn to bound the overshoot, don’t rely on the check alone. - The allowlist governs which tools run and whether their arguments parse, not whether a permitted call is the right call to make.
- Spend attribution across multiple cooperating agents, and cost estimation for streamed or unpriced models, are both outside what was built and tested here.