Azure OpenAI on Your Data: Streaming Response Performance with JSON Property Filtering

Hi Kaja Sherief,

It looks like you’re working to optimize streaming JSON responses from Azure OpenAI and reduce the latency introduced by post-processing before displaying data in your UI. In your current approach, you wait for the entire response to arrive before parsing, which adds several seconds to the overall response time.

To improve this, the key is to process the response incrementally as it streams rather than after it completes. Since Azure OpenAI “on your data” returns text-based JSON, you can use a streaming JSON parser to read the data token-by-token and extract the UI-specific property as soon as it appears in the stream. This allows you to send that property to the UI immediately while the rest of the data continues to stream and process in the background. Examples of streaming JSON parsers include JsonTextReader from Newtonsoft.Json (.NET), ijson (Python), and Clarinet or stream-json (JavaScript).

Although Azure OpenAI does not support server-side selective property streaming, you can make extraction faster through prompt engineering — for example, structuring your request so the model outputs the UI property first in the JSON object, followed by other properties. This ensures the critical field appears earlier in the stream. Content filtering is always enabled in Azure OpenAI and cannot be turned off, but it is applied in parallel and generally does not block incremental streaming of the main content.

For large or mixed workloads, you can split processing into two pipelines:

Primary (UI) – extracts and sends the needed property to the UI as soon as it’s detected in the stream.

Secondary (background) – handles parsing and storing or further processing the rest of the response asynchronously.

Finally, you can reduce unnecessary delay by minimizing payload size and, where possible, adjusting max_tokens to limit overly large responses. This combination of incremental parsing, prompt structuring, and dual-path processing helps maintain the desired 3.5–4.5 second UI response time while still supporting complete background processing of all returned data.
For reference:

Azure OpenAI Responses API (Preview)
Content streamingContent streaming
Learn how to use JSON mode