Streaming AI Gateways

Streaming feels like a small product decision until you have to operate it. The first part of the answer appears sooner, the interface feels more responsive, and the demo looks better. Then a real request arrives: the client disconnects halfway through, the model emits something unsafe across two chunks, or the system logs a partial answer as if it were complete.

That is the part I care about in this issue. A streamed AI response is not just the same answer arriving earlier in smaller pieces. It is a different runtime contract. The gateway has to decide what is allowed to leave the server before the full answer exists.

In this issue, we build a .NET streaming AI gateway with Microsoft Foundry as the default runtime provider. It exposes an SSE endpoint, validates requests before model execution, applies partial-output policy, handles cancellation, and writes a finalization record for every run. The deterministic mock stream stays in the repo for tests; the application path uses a real Foundry/OpenAI-compatible stream.

The Boundary I Care About Here

The weak version of streaming is simple: call the model with stream: true, forward every chunk to the browser, and concatenate the same chunks somewhere in the background. That works until the stream does not finish cleanly.

A production gateway needs a stronger boundary. The upstream model stream is not the same thing as the downstream product stream. The model emits deltas. The gateway emits admitted events. Those are different things.

Server-sent events give us a practical one-way wire format for incremental updates. Microsoft Foundry's chat completions API streams from /openai/v1/chat/completions when stream is true. See the Microsoft Foundry chat completions reference.

Those docs explain the transport. They do not design the product boundary for you. The important engineering question is what your gateway does between provider chunks and client-visible output.

What We Are Actually Building

The companion repository contains an ASP.NET Core app named StreamingAiGateway. It defaults to Microsoft Foundry so the issue shows a real provider boundary, while the control layer stays testable with deterministic streams.

The gateway does this:

• accepts POST /gateway/stream
• emits named SSE events
• validates requests before model execution
• streams from Microsoft Foundry
• holds back a small output tail for policy checks
• handles cancellation and terminal run records

The repo is small, but the shape is the point. Streaming is treated as an execution contract, and the model client is provider-neutral even though Foundry is the active runtime path.

The Shape Of The Gateway

The architecture has three flows in mind: the incoming HTTP request, the upstream model stream, and the downstream SSE stream. The gateway sits between the last two and decides what is allowed to move forward.

The model provider can be replaced. The gateway contract should hold. The central file is StreamingGatewayService.cs; the endpoint is thin and the service owns the control loop.

SSE Is The Downstream Contract

The endpoint exposes named events so the client can distinguish started, admitted text, policy block, validation rejection, and finalization.

The response writer sets text/event-stream, disables caching, and writes frames as event plus JSON data.

public async Task WriteAsync(
  string eventName,
  object payload,
  CancellationToken cancellationToken = default)
{
  var json = JsonSerializer.Serialize(payload, JsonOptions);

  await response.WriteAsync($"event: {eventName}\n", cancellationToken);
  await response.WriteAsync($"data: {json}\n\n", cancellationToken);
  await response.Body.FlushAsync(cancellationToken);
}

That frame shape matters. If everything is a default message event, the browser receives text but the product loses state.

The Request Is Validated Before The Stream Starts

A gateway should not start a model stream just because the HTTP request arrived. Rejection belongs before the first model token.

public ValidationResult Validate(StreamRequest request)
{
  var gateway = options.Value;
  var reasons = new List<string>();

  if (string.IsNullOrWhiteSpace(request.UserMessage))
      reasons.Add("user_message_required");

  if (request.UserMessage?.Length > gateway.MaxInputCharacters)
      reasons.Add($"user_message_too_long:{request.UserMessage.Length}>{gateway.MaxInputCharacters}");

  if (string.IsNullOrWhiteSpace(request.RiskTier))
      reasons.Add("risk_tier_required");

  return reasons.Count == 0
      ? ValidationResult.Accept()
      : new ValidationResult(false, reasons);
}

If validation fails, the gateway writes stream.rejected, persists a finalization record, and never calls the model client.

Partial Output Is Still Output

This is the part many streaming demos skip. A partial answer can still leak something, mislead someone, or get copied into a ticket. The repo uses a hold-back buffer so the newest tail of generated text is checked before it leaves the server.

public PolicyChunkDecision Observe(string delta)
{
  ReceivedCharacters += delta.Length;
  _pending.Append(delta);

  var blockReason = ValidateCandidate();
  if (blockReason is not null)
      return PolicyChunkDecision.Block(blockReason);

  if (_pending.Length <= _holdBackCharacters)
      return PolicyChunkDecision.Allow(string.Empty);

  var flushLength = _pending.Length - _holdBackCharacters;
  var safeText = _pending.ToString(0, flushLength);
  _pending.Remove(0, flushLength);
  _emitted.Append(safeText);

  return PolicyChunkDecision.Allow(safeText);
}

The deterministic test stream emits The api_ and then key should not be visible.. The forbidden phrase only exists after the second chunk arrives. With the hold-back window enabled, the gateway blocks before api_key is emitted.

This is not a complete content safety system. It is a streaming control. A serious product may still need provider safety filters, moderation, redaction, reviewer workflows, and policy by risk tier.

Progressive And Buffered Are Different Modes

The request includes a display mode. Standard workflows can use Progressive, where admitted text is released as it passes policy. Higher-risk workflows can use BufferedUntilFinal, where the gateway consumes the stream but releases nothing until completion. Streaming is not always the safer product choice just because it feels fast.

Cancellation Is A Terminal State

Client cancellation is not just noise in the logs. The user did not receive the committed final answer, and some partial text may already have left the server. The endpoint passes HttpContext.RequestAborted into the gateway, and the service handles cancellation separately from provider failure.

catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
  state = state with
  {
      Status = StreamRunStatus.Canceled,
      ClientCanceled = true
  };

  return await FinalizeAsync(state, sink, canWriteFinal: false);
}

If the client disconnected, the server may not be able to send stream.finalized. The JSONL record still gets written.

Finalization Is The Commit Boundary

A streaming client needs a clear rule: do not treat the answer as committed until the final event says the run completed. Without that rule, partial text can become product state.

The finalization record carries operational evidence without storing raw prompt or output content by default:

{
  "runId": "run_...",
  "status": "Completed",
  "provider": "foundry",
  "modelId": "gpt-5.1",
  "promptSha256": "...",
  "outputSha256": "...",
  "receivedChunks": 60,
  "emittedCharacters": 293,
  "finishReason": "stop",
  "clientCanceled": false,
  "finalEventWritten": true
}

Raw content can be sensitive. The default record stores hashes, model identity, status, counters, and reason fields. If a product needs raw capture, make it an explicit governed store, not an accidental trace field.

OpenTelemetry's GenAI semantic conventions are relevant when this grows into real telemetry. This sample keeps the finalization log local, but the same fields can map to spans, metrics, and events later.

Foundry Is The Upstream Boundary

The default provider is Microsoft Foundry. If the endpoint, deployment, or credential is missing, the app fails fast at startup. That is intentional. A streaming gateway that claims to teach real integration should not silently fall back to a fake provider.

The adapter posts to the chat completions endpoint with stream: true:

var payload = new
{
  model = deployment,
  stream = true,
  messages = new[]
  {
      new { role = "system", content = request.SystemPrompt },
      new { role = "user", content = request.UserMessage }
  }
};

Then it reads data: lines from the provider stream:

var data = line["data:".Length..].Trim();
if (data == "[DONE]")
{
  yield return ModelStreamChunk.Complete();
  yield break;
}

var parsed = TryParseChatCompletionChunk(data);
if (parsed.Delta is not null)
{
  yield return ModelStreamChunk.Text(parsed.Delta);
}

The important detail is that the Foundry client does not write to the HTTP response. It only produces internal chunks. The gateway still owns downstream admission, cancellation, and finalization.

A Foundry Run Tells The Story

Configure Foundry first, then run the gateway from the companion repo:

$env:STREAMGW_Foundry__Endpoint = "<project-or-openai-v1-endpoint>"
$env:STREAMGW_Foundry__DeploymentName = "<deployment-name>"
$env:STREAMGW_Gateway__ModelId = "<deployment-name>"
$env:STREAMGW_Foundry__CredentialHeaderName = "api-key"
$env:STREAMGW_FOUNDRY_CREDENTIAL = "<foundry-api-key>"
dotnet run --project StreamingAiGateway

I would set those values in the shell, CI/CD secret store, or production hosting configuration that starts the gateway. I would not put the API key in the repository. appsettings.json is fine for default policy values; credentials belong outside the source tree.

Send the standard sample request:

curl.exe -N -H "Content-Type: application/json" --data '@StreamingAiGateway/data/sample-requests/standard.json' http://localhost:5296/gateway/stream

The output is an SSE stream:

event: stream.started
data: {"runId":"run_...","requestId":"REQ-LOCAL-001","modelId":"gpt-5.1","displayMode":"Progressive","riskTier":"standard"}

event: stream.delta
data: {"runId":"run_...","requestId":"REQ-LOCAL-001","delta":"I don\u2019t ha"}

event: stream.finalized
data: {"runId":"run_...","requestId":"REQ-LOCAL-001","status":"Completed","finishReason":"stop","blockedReason":null,"errorCode":null,"emittedCharacters":293,"receivedChunks":60}

The answer is not interesting by itself. The sample request has no real case body, so the model says it needs more information. The important part is the contract around it: Foundry streamed chunks, the gateway admitted deltas, and the run finalized cleanly. The deltas look uneven because the gateway holds back the newest tail before flushing.

The policy-violation path tells the other half of the story. After the live Foundry path is proven, I switch only this replay to the deterministic test provider. That gives the gateway a known unsafe split across chunks without asking a real model to invent a secret:

$env:STREAMGW_Gateway__Provider = "mock"
dotnet run --project StreamingAiGateway

curl.exe -N -H "Content-Type: application/json" --data '@StreamingAiGateway/data/sample-requests/policy-violation.json' http://localhost:5296/gateway/stream

event: stream.policy_blocked
data: {"runId":"run_...","requestId":"REQ-LOCAL-002","blockedReason":"forbidden_output_pattern:\bapi[_-]?key\b"}

event: stream.finalized
data: {"runId":"run_...","requestId":"REQ-LOCAL-002","status":"PolicyBlocked","finishReason":null,"blockedReason":"forbidden_output_pattern:\bapi[_-]?key\b","errorCode":null,"emittedCharacters":9,"receivedChunks":3}

The gateway emitted a harmless prefix, then stopped when the held-back text completed the forbidden phrase. It did not send the forbidden token to the client. The final status is not Completed. That distinction is the product contract.

The Tests Protect The Stream Boundary

The tests focus on behavior that is easy to lose when streaming is treated as a UI detail.

dotnet test StreamingAiGateway.slnx

The current suite passes 8 tests covering split forbidden output, successful streaming, finalization, policy blocking, length limits, chunk limits, rejected input, and client cancellation. Those tests are more useful than a screenshot of a fast typing animation.

Why This Architecture Works

The value is that every important transition is explicit.

• Requests fail before model execution when they are invalid.
• Provider chunks are separated from client-visible deltas.
• Policy runs before text reaches the browser.
• Canceled and blocked streams get explicit terminal states.
• Run records make later incident review possible without storing raw content by default.

The model writes text. The gateway owns admission, status, and commitment.

Potential Enhancements

The next production steps would be OpenTelemetry spans and metrics, policy by risk tier, managed identity for Foundry where the hosting model supports it, heartbeat comments for long-running streams, browser-level stream tests, provider timeout classification, and pre-stream retry policy.

I would not retry in the middle of an active user-visible stream unless the product has an explicit merge contract. Mixed partial answers are hard to explain later.

Final Notes

Streaming is worth building, but production streaming needs more than a provider flag and a typing animation.

The practical lesson is simple: treat streamed text as uncommitted until the gateway has admitted it, tracked it, and finalized the run. The model can emit chunks. The product should emit a contract.

Explore the companion repository at the GitHub repository.

See you in the next issue.

Stay curious.