Azure Container App with EventHubConsumerClient gets stuck after idle period – requires restart to resume

@YUGAN NITHISH KUMAR T I understand that your Python app using EventHubConsumerClient stops receiving events after an idle period (10–30 minutes), without logging any errors or disconnections. The only workaround so far is restarting the container app.

From the error you mentioned, we observed that

1.Silent Idle Disconnects Are a Known Issue

This behavior has been observed and discussed in Microsoft Q&A forums and internal emails. The SDK does not always log idle disconnects, especially when the AMQP session times out silently due to inactivity.

2.No Built-in Auto-Reconnect in SDK

The Python SDK for Azure Event Hub does not automatically reconnect after idle disconnects.

3.Idle Session Timeout Behavior

While not specific to Event Hub, idle session timeout mechanisms are documented for other Azure services. These timeouts can affect long-lived connections in containerized environments.

To resolve these,

Implement Manual Reinitialization

Yes, you should consider reinitializing the EventHubConsumerClient periodically or upon detecting inactivity. This can be done by wrapping the receive() call in a loop with timeout checks.

Use Heartbeat + Timeout Logic

Your heartbeat logger is a good start. Enhance it by tracking the time since the last event and restarting the client if it exceeds a threshold.

Enable AMQP-Level Diagnostics

Use Azure Monitor and Log Analytics to track AMQP session drops. The Azure Container Apps - Workshop explains how to surface logs via stdout/stderr and query them.

Azure Container Apps - Workshop

Check for BlobCheckpointStore Issues

Ensure that the blob checkpointing mechanism isn’t causing stalls. If the checkpoint store becomes unreachable or corrupted, it may block event processing silently.

Use CloseAsync() and DisposeAsync() Properly

Ensure that you’re gracefully closing the client using CloseAsync() and DisposeAsync() methods to release resources and avoid lingering connections.

Monitor with Azure Diagnostic Settings

You can configure Azure Monitor to detect dropped AMQP sessions. The ingestion troubleshooting guide provides queries to detect failures and mismatches in consumer group configurations.

Here’s a simplified loop structure to help manage idle disconnects:

import asyncio

from azure.eventhub.aio import EventHubConsumerClient

from azure.eventhub.extensions.checkpointstoreblobaio import BlobCheckpointStore

import time

last_event_time = time.time()

async def on_event(partition_context, event):

    global last_event_time

    print(f"Received event from partition: {partition_context.partition_id}")

    await partition_context.update_checkpoint(event)

    last_event_time = time.time()

async def receive_events(client):

    while True:

        try:

            await client.receive(on_event=on_event, starting_position="@latest")

        except Exception as e:

            print(f"Error: {e}")

        await asyncio.sleep(10)

        if time.time() - last_event_time > 1800:  # 30 minutes

            print("Idle timeout reached. Restarting client.")

            await client.close()

            break

def run_dispatcher():

    checkpoint_store = BlobCheckpointStore.from_connection_string("", "")

    client = EventHubConsumerClient.from_connection_string("", consumer_group="$Default", eventhub_name="", checkpoint_store=checkpoint_store)

    loop = asyncio.get_event_loop()

    loop.run_until_complete(receive_events(client))

I hope these recommendations help resolve the issue. Please feel free to reach out for any further assistance.

1. Is the silent idle disconnect behavior officially documented for EventHubConsumerClient in Azure Container Apps?
   • If so, can you point to the docs or internal knowledge base entry?
   • Is there a known idle timeout threshold at the AMQP/TCP level within ACA infrastructure?
2. How can I detect that a silent disconnect occurred?
   • Can we log or trace AMQP session closures (like idle-timeout or TCP FIN) from ACA or Event Hubs side?
   • Which diagnostic settings in Azure Monitor or Container Apps help detect such drops?
3. Is there an Azure-native way to auto-restart consumers upon inactivity, or should watchdog logic be embedded in the app as standard?
   • Any supported retry pattern or auto-reconnect framework planned for the Python SDK?
4. Does the use of WebSocket vs AMQP over TCP impact idle timeout behavior in ACA?
   • Are there recommendations for production environments inside VNETs (e.g., is TCP more stable than WebSocket or vice versa)?
5. What metrics or logs can help determine whether checkpoint blob access is contributing to silent hangs?
   • Any known guidance for blob checkpointing monitoring with Azure Event Hubs?
6. Is the reinitialization interval (e.g., restart client every 3–5 minutes of idle) considered safe and scalable in production?
   • Could this create issues with partition rebalancing or consumer group stability under load?
7. Is EventProcessorClient recommended over EventHubConsumerClient for better resilience in these scenarios?
   • Would switching to EventProcessorClient help with automatic recovery or fault detection in ACA?

@YUGAN NITHISH KUMAR T answering to your questions:

1.

No, silent idle disconnect behavior is not officially documented for EventHubConsumerClient in Azure Container Apps

The default timeout is 4 minutes, and it can be configured between 4 and 30 minutes for TCP and client-dependant for AMQP.

2.

Enable AMQP heartbeats and monitor idle timeouts.

Apply ACA resiliency policies for TCP retries and timeouts.

Use watchdogs and health probes to monitor connection health.

Log and alert on inactivity thresholds using Azure Monitor or custom diagnostics.

To trace AMQP session closures effectively:

Enable Diagnostic Logs on Event Hubs and ACA endpoints.

Use Azure Monitor to query AMQP errors and TCP session metrics.

Configure Idle Timeout Policies on ACA and Load Balancer endpoints.

Monitor Firewall Logs for session drops due to IDPS or scale-in events.

Use Heartbeats in AMQP clients to detect silent disconnects proactively.

Diagnostic Settings in Azure Monitor

Platform Metrics & Logs

Log Analytics & Alerts

Application Insights

Network Watcher

Diagnostic Settings in Azure Container Apps

Log Analytics Workspace Integration

Console & System Logs

Metrics Collection

Alerts Based on Metrics and Logs

Embed Watchdog Logic: Implement heartbeat checks or activity counters in your app. If inactivity is detected, reinitialize the consumer or restart the container.

Use Azure Monitor Alerts: Configure alerts on inactivity metrics and link them to automated recovery workflows.

Enable Diagnostic Logging: Use Log Analytics to trace consumer behavior and detect silent disconnects.

Consider Scheduled Restarts: For long-lived consumers, schedule periodic restarts using Logic Apps or Automation Accounts.

Use SDK retry settings for transient errors.

Implement watchdog logic in your app to detect inactivity and reinitialize consumers.

Leverage Azure Monitor alerts to trigger container restarts or SDK reinitialization.

Consider resilience libraries like tenacity or R9 for structured retry behavior.

For AMQP over TCP:

Configure idle timeout to a higher value (up to 30 minutes) if supported.

Use TCP keep-alive or heartbeat logic in your client.

For WebSocket:

Define custom health probes in ACA to avoid silent disconnects.

Monitor logs for handshake failures and EOF errors.

5.

To detect and mitigate silent hangs due to blob checkpointing:

Enable Diagnostic Settings for Event Hubs and Blob Storage.

Monitor Runtime Audit Logs for checkpoint write failures or access errors.

Use Azure Monitor and Log Analytics to correlate blob latency with consumer inactivity.

Implement Retry Logic in your consumer app for checkpoint writes.

Alert on Missing Checkpoints: Set alerts if no checkpoint is written within a defined interval.

Restarting clients every 3–5 minutes of idle time is not recommended for production environments. It can destabilize consumer groups, trigger excessive partition rebalancing, and amplify downstream failures. Instead, use structured retry logic, monitor idle metrics, and apply watchdogs judiciously.

7.

Switching to EventProcessorClient is recommended for ACA-based workloads that require:

Automatic recovery from silent disconnects or idle hangs.

Scalable partition management across multiple consumers.

Built-in checkpointing and fault detection mechanisms.

This client is designed to handle the complexities of production environments and integrates well with ACA’s autoscaling and monitoring capabilities.

I hope this helps in resolving the issue, do let me know if you have any further questions on this

@YUGAN NITHISH KUMAR T Just checking to see if you have a chance to check my previous response and helped, do let me know if you have any further questions on this.