Building Reliable LLM Pipelines
Patterns for schema validation, failure-aware retries, model fallback chains, and structured logging in production LLM applications.
The reliability problem
LLM APIs behave differently from standard web services. A database query either returns records or throws an error, but a language model call can return an HTTP 200 with output that is malformed, factually inconsistent, or unrelated to the prompt.
Model invocations are non-deterministic service calls. Their application boundary needs output schemas, failure-specific retry rules, fallback tiers, and sanitized telemetry.
Structured output with schema validation
Enforcing output structure at the application boundary prevents invalid payloads from propagating downstream. Rather than accepting unconstrained natural language, define a schema upfront and validate the parsed JSON against it:
import { z } from "zod";
const ExtractionSchema = z.object({
entities: z.array(
z.object({
name: z.string(),
type: z.enum(["person", "organization", "location"]),
confidence: z.number().min(0).max(1),
})
),
summary: z.string().max(500),
});
type Extraction = z.infer<typeof ExtractionSchema>;
The model receives the expected schema definition in its system or task prompt. After receiving the response, validate the payload:
function parseResponse(raw: string): Extraction | null {
try {
const json = JSON.parse(raw);
return ExtractionSchema.parse(json);
} catch {
return null;
}
}
When parsing fails, retry with a corrective prompt that includes the specific validation error. Providing the parse error output allows the model to address missing fields or invalid types directly.
Failure-aware retry strategies
Repeating an identical prompt after a failure wastes tokens and frequently reproduces the same malformed response. Effective retry logic differentiates by error type:
async function callWithRetry<T>(
fn: () => Promise<T>,
validate: (result: T) => boolean,
maxRetries = 3
): Promise<T> {
let lastError: Error | null = null;
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const result = await fn();
if (validate(result)) return result;
lastError = new Error("Validation failed");
} catch (err) {
lastError = err as Error;
// Rate limit: back off exponentially
if ((err as { status?: number }).status === 429) {
await sleep(Math.pow(2, attempt) * 1000);
continue;
}
// Server error: retry immediately
if ((err as { status?: number }).status === 500) {
continue;
}
// Client error: don't retry
throw err;
}
}
throw lastError ?? new Error("Max retries exceeded");
}
function sleep(ms: number) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
Key decisions:
- Rate limits (429): Exponential backoff allows upstream rate limits to reset.
- Server errors (500): Retry immediately or with short jitter for transient infrastructure interruptions.
- Client errors (400): Do not retry; the payload or request configuration requires correction.
- Validation failures: Retry with a modified prompt containing the validation error message.
Fallback chains
Not every task requires the same model tier. A fallback chain can try a fast model first, then move to a higher-capability model when validation fails or execution errors occur:
const MODELS = [
{ name: "fast-model" },
{ name: "high-capability-model" },
];
async function extractWithFallback(
input: string
): Promise<Extraction> {
for (const model of MODELS) {
try {
const result = await callLLM(model.name, input);
const parsed = parseResponse(result);
if (parsed) return parsed;
} catch {
console.warn(`${model.name} failed, trying next`);
}
}
throw new Error("All models failed");
}
Measure each tier's validation pass rate, latency, and token usage before choosing the order. A fallback chain is useful only when the first tier's lower cost outweighs the extra retry path.
Structured logging and data handling
Debugging non-deterministic systems requires tracing request and response state, but prompts and completions frequently contain sensitive data, personal identifiers, or credentials. Production logging must combine comprehensive telemetry with redaction and access controls.
A structured telemetry record should capture:
- Unique request identifiers and model configurations
- Sanitized prompts and responses (with sensitive values masked or omitted)
- Validation status and parsed payloads
- Request latency and token usage
- Retry attempt counts and failure classifications
interface LLMLog {
requestId: string;
model: string;
sanitizedPrompt: string;
sanitizedResponse: string;
sanitizedParsed: unknown | null;
valid: boolean;
attempt: number;
latencyMs: number;
tokens: { input: number; output: number };
timestamp: string;
}
Orchestration layers and evaluation datasets
As pipelines add parallel branches, tool executions, and multi-step recovery logic, chaining asynchronous functions quickly becomes unwieldy. Explicit state machines or graph-based orchestration frameworks make state transitions, timeouts, and checkpointing predictable and inspectable.
A fixed evaluation set can compare prompt or model changes before release. Assertions work for structural requirements; task-specific scoring or reviewed examples are needed for semantic quality.