Batch inference looks harmless when the input file is small. You load rows, call the model, write outputs, and feel the satisfying rhythm of a simple loop. Then the job gets large enough to matter. Row 341 fails. Row 612 times out. The process restarts after writing some outputs but before updating state. Someone asks whether it is safe to run the job again, and suddenly the loop is not so simple.
That is the part I care about in this issue. Batch inference is not only a throughput problem. It is a job-control problem. A production AI pipeline needs to know which requests were admitted, which model deployment handled them, which rows already committed, which rows failed permanently, which rows should be retried, and which outputs can be replayed later without guessing.
In this issue, we build a .NET 10 batch inference runner for support-ticket triage. The repo uses Microsoft Foundry through the OpenAI-compatible chat completions endpoint for live model calls. Around that probabilistic model call, the runner adds deterministic request manifests, idempotency keys, retry envelopes, checkpoints, partial failure handling, failed-row quarantine, and JSONL output records that can be replayed and audited.
The Boundary I Care About Here
A weak batch pipeline treats the model call as the center of the system. A stronger pipeline treats the model call as one row-level operation inside a durable job boundary.
There are good primitives around this problem. The OpenAI Batch API has explicit batch statuses and separates output and error files. Kubernetes Jobs give us vocabulary for completion, restart policy, and work-queue style execution. Kueue LocalQueues show how tenant-scoped work can enter a queue before cluster capacity admits it.
Those primitives are useful, but they do not remove the application contract. Your AI job still needs row identity, idempotency, provider retry policy, checkpoint state, quarantine evidence, and replayable outputs. That is what we build here.
What We Are Actually Building
The companion repository contains a .NET console project named BatchInferenceJobControl. It reads a versioned manifest of support tickets, validates the job contract, and sends each admitted row to Microsoft Foundry for structured triage.
The interesting part is not the ticket classifier. The interesting part is what happens around it. Each row has an idempotency key. Each completed row is written to a checkpoint. Provider failures are retried only when they are safe to retry. Rows that cannot be trusted are quarantined instead of being mixed into the successful output set.
The result is a small batch runner you can stop, inspect, and run again. If the same manifest is replayed after a successful run, the completed rows are skipped rather than sent to the model a second time.
The Shape Of The Batch Runner
The architecture is a small control loop. The manifest defines the work. The checkpoint defines what is already terminal. Foundry produces row-level structured results. The runner decides whether each row commits, retries, or goes to quarantine.
The model is important, but it is not in charge. It classifies one row at a time. Deterministic code owns the job contract around those calls.
Foundry Is The Model Boundary
The runtime integration posts to Microsoft Foundry's Azure OpenAI chat endpoint. The Microsoft Foundry chat completions reference documents the /openai/v1/chat/completions route and API-key header shape. The repo supports either a direct /openai/v1 endpoint or a Foundry project URL that can be resolved to the OpenAI-compatible endpoint.
using var message = new HttpRequestMessage(HttpMethod.Post, BuildChatCompletionsUrl(settings.Endpoint));
ApplyCredential(message, credential);
var payload = new ChatCompletionRequest
{
Model = string.IsNullOrWhiteSpace(settings.DeploymentName) ? request.ModelId : settings.DeploymentName,
Temperature = settings.Temperature,
MaxTokens = settings.MaxOutputTokens,
ResponseFormat = new ResponseFormat { Type = "json_object" },
Messages =
[
new ChatMessage { Role = "system", Content = InferencePromptBuilder.BuildSystemPrompt(request.OutputSchemaVersion) },
new ChatMessage { Role = "user", Content = InferencePromptBuilder.BuildUserPrompt(request) }
]
};The model is asked to return a JSON object with summary, category, priority, recommendedAction, and confidence. The runner parses and validates that contract before committing the row output.
For GPT-5-class deployments, the adapter sends max_completion_tokens rather than the older max_tokens field. That detail matters in live Foundry runs because unsupported provider parameters are quarantined as non-retryable row failures, leaving a clear error record instead of a half-written batch.
A live run needs Foundry configuration:
$env:BATCHAI_Foundry__Endpoint = "https://YOUR-RESOURCE.services.ai.azure.com/openai/v1"
$env:BATCHAI_Foundry__DeploymentName = "gpt-5.4-mini"
$env:BATCHAI_FOUNDRY_CREDENTIAL = "<foundry-api-key>"
dotnet run --project BatchInferenceJobControlThe deployment name matters because the manifest also names the expected model. A batch should not silently drift to another deployment just because the shell environment changed.
The Manifest Is The Work Contract
The manifest is the job boundary. It names the job, version, model, source dataset, output schema, and rows. Each row carries a request ID and idempotency key.
{
"jobId": "support-ticket-triage-2026-09-26",
"version": "2026.09.26",
"modelId": "gpt-5.4-mini",
"sourceDatasetId": "support-ticket-export-2026-09-26",
"outputSchemaVersion": "ticket-triage-v1",
"rows": [
{
"requestId": "ROW-001",
"idempotencyKey": "support-ticket-triage-2026-09-26:ROW-001:4c908a",
"tenantId": "tenant-alpha",
"priorityHint": "urgent",
"inputText": "Customer reports that the production status page is green but API requests to /v1/orders have returned intermittent 503 responses for 18 minutes.",
"metadata": {
"source": "zendesk",
"product": "orders-api",
"region": "eu"
}
}
]
}I like this shape because the input is reviewable before execution. The runner rejects duplicate request IDs, duplicate idempotency keys, missing job identity, missing model identity, and model mismatch against the configured Foundry deployment.
Checkpoints Make Resume Boring
A batch runner should be able to restart without a debate. If a row is already terminal in the checkpoint, the runner skips it. It does not call the model again. It does not append a duplicate output record.
if (checkpoint.Rows.TryGetValue(row.RequestId, out var terminal))
{
if (terminal.Status == RowTerminalStatus.Succeeded)
{
succeeded++;
}
else
{
quarantined++;
}
skipped++;
continue;
}The checkpoint includes the manifest hash. If the manifest changes, the runner refuses to load the old checkpoint. That small guard prevents a dangerous class of accidental replay bugs: using terminal state from one input set against another input set.
Retries Are Envelopes, Not Hope
Retries should be bounded, classified, and visible. The runner treats HTTP 408, 429, and 5xx provider failures as retryable. It does not retry invalid rows, credential failures, malformed model output, or other permanent failures.
catch (InferenceClientException ex) when (ex.IsRetryable && attempt < settings.Retry.MaxAttempts)
{
await DelayBeforeRetryAsync(cancellationToken);
}
catch (InferenceClientException ex)
{
var quarantine = BuildQuarantine(
manifest,
manifestHash,
row,
ex.ErrorCode,
ex.Message,
ex.IsRetryable,
attempt);
await QuarantineAsync(checkpoint, quarantine, cancellationToken);
quarantined++;
break;
}This is a small piece of code, but it changes the operating model. A retry is no longer a hidden loop. It is part of the row evidence.
Quarantine Keeps Partial Failure Honest
The wrong response to partial failure is pretending the batch succeeded because enough rows completed. The better response is to keep the successful rows committed and put failed rows somewhere explicit.
{
"jobId": "support-ticket-triage-2026-09-26",
"jobVersion": "2026.09.26",
"requestId": "ROW-003",
"idempotencyKey": "support-ticket-triage-2026-09-26:ROW-003:59d95d",
"tenantId": "tenant-charlie",
"manifestHashSha256": "4f...",
"errorCode": "foundry_http_429",
"errorMessage": "Foundry chat completion returned HTTP 429",
"retryable": true,
"attempts": 3,
"quarantinedAtUtc": "2026-09-26T00:04:12Z"
}A quarantine record is not a dead end. It is a review queue. It tells you whether the row was invalid, whether the provider failed, whether retries were exhausted, and which manifest produced the failure.
Outputs Are Replay Records
A successful output is not just the model answer. It is the committed evidence for one row: job identity, row identity, idempotency key, model identity, manifest hash, input hash, prompt hash, output hash, schema version, attempt count, token usage, and structured result.
{
"jobId": "support-ticket-triage-2026-09-26",
"jobVersion": "2026.09.26",
"requestId": "ROW-001",
"idempotencyKey": "support-ticket-triage-2026-09-26:ROW-001:4c908a",
"modelId": "gpt-5.4-mini",
"manifestHashSha256": "4f...",
"inputHashSha256": "b7...",
"promptHashSha256": "a2...",
"outputHashSha256": "8c...",
"outputSchemaVersion": "ticket-triage-v1",
"attempts": 1,
"result": {
"summary": "EU order API is returning intermittent 503 responses after deployment.",
"category": "outage",
"priority": "urgent",
"recommendedAction": "Route to incident response and verify EU deployment health.",
"confidence": 0.91
}
}Those hashes are not decoration. They let you prove which input and prompt contract produced the stored result without logging raw content everywhere.
A Live Run Tells The Story
Once Foundry is configured, run the repo:
dotnet run --project BatchInferenceJobControlA successful run prints the job-control summary:
Batch Inference Job Control for AI Pipelines
Job: support-ticket-triage-2026-09-26
Manifest hash: 4f...
Rows: total=4 succeeded=4 quarantined=0 skipped=0
Attempts: 4
Checkpoint: data/checkpoints/support-ticket-batch.checkpoint.json
Outputs: data/outputs/support-ticket-batch.outputs.jsonl
Quarantine: data/quarantine/support-ticket-batch.quarantine.jsonl
Report: data/reports/support-ticket-batch-report.jsonIn the live Foundry check for this issue, gpt-5.4-mini processed all four rows successfully. The output JSONL records include structured ticket classifications, token usage, prompt hashes, input hashes, and output hashes. A provider request-shape error during testing was quarantined cleanly, then fixed by sending max_completion_tokens for the GPT-5-class deployment.
If you run the same manifest again with the same checkpoint, the runner skips the terminal rows. That is the moment the architecture earns its keep. You can resume without manually trimming input files or hoping the provider deduplicates your calls.
The Tests Protect The Job Boundary
The tests use scripted clients in the test project. They do not pretend to be a runtime provider. They prove the deterministic job-control behavior around the provider boundary.
dotnet test BatchInferenceJobControl.slnxThe current suite passes 9 tests covering:
- successful rows producing output records and checkpoints
- checkpoint resume skipping completed rows
- transient retry followed by success
- retry exhaustion to quarantine
- invalid row quarantine without model execution
- duplicate idempotency key rejection
- Foundry project endpoint resolution
- Foundry
api-keyauthentication and JSON request shape - retryable provider HTTP failure classification
That is the right test surface for this issue. We are not testing whether a small ticket classifier is brilliant. We are testing whether the batch job remains controllable when rows succeed, fail, retry, quarantine, and resume.
The Deterministic Boundary
The probabilistic layer reads one ticket and returns one structured classification. It does not decide what the batch is, whether a row can be replayed, whether a failed row should be hidden, or whether a prior output can be overwritten.
In this implementation, deterministic code owns:
- manifest validation
- deployment identity checks
- idempotency key uniqueness
- row admission
- retry classification and attempt limits
- checkpoint resume behavior
- quarantine records
- output hashes and replay evidence
The model can classify. The job controller decides what becomes durable state.
Why This Architecture Works
The value is that the batch job becomes inspectable and restartable.
- The manifest gives the job a stable identity.
- Idempotency keys prevent row-level replay ambiguity.
- The configured Foundry deployment must match the manifest model.
- The checkpoint prevents duplicate model calls on resume.
- Retries are bounded and classified.
- Quarantine keeps partial failure visible instead of contaminating outputs.
- Output records carry enough hashes to support replay and incident review.
That is the practical difference between running a script and operating an AI pipeline. Scripts finish or fail. Pipelines leave behind state you can reason about.
Potential Enhancements
The next version should add bounded concurrency, because real batch jobs eventually need more than one row in flight. I would start with a simple limit per model deployment and a simple limit per tenant. That gives the runner better throughput without letting one manifest consume the entire quota.
After that, I would add three practical pieces: row-level telemetry, cost totals for the run, and a replay command that can build a new manifest from selected quarantine rows. Those additions would make failures easier to investigate and successful runs easier to trust.
Final Notes
Batch inference deserves more respect than it usually gets. The hard part is not writing a loop over rows. The hard part is knowing what happened after the loop was interrupted, retried, resumed, partially failed, or questioned during an incident.
The practical lesson is simple: put a job contract around the model call. Use manifests, idempotency keys, checkpoints, retry envelopes, quarantine, and replayable outputs. Then the batch pipeline can be operated as software, not remembered as a one-off run.
Explore the companion repository at the GitHub repository.
See you in the next issue.
Stay curious.