Managed Redis fails with StackExchange.Redis.RedisConnectionException after a few days of usage and does not return to e healthy state

Hi Sven Carstensen,

Yes, you're absolutely right.

var credential = new DefaultAzureCredential(new DefaultAzureCredentialOptions { ManagedIdentityClientId = "our-user-managed-identity-id"} );

only fetches the token once when you call GetTokenAsync(). It does not automatically refresh the token later unless you implement that logic yourself.

How would I refresh that token?

To handle token refresh properly, you need to:

1. Cache the token and track its expiration.
2. Refresh it proactively before it expires (e.g., 5 minutes before).
3. Recreate the Redis connection with the new token.

I hope this information helps. Please do let us know if you have any further queries.

Aren't the services I added fixed after I called Build() on the app builder?

How would I bring the recreated Redis connection into usage with dependency injection?

I tried to use the custom ConnectionMultiplexer created in CreateRedisConnectionAsync as an option to AddStackExchangeRedisCache by setting ConnectionMultiplexerFactory. But that gives authentication errors.

Returning to the old code ConfigureForAzureWithUserAssignedManagedIdentityAsync still produced the Auth failures.

I am at a loss why the Redis Cache is not allowing our user managed identity to connect.

New code snippet:

        var configurationOptions = ConfigurationOptions.Parse(cacheOptions.DistributedCacheEndpoint!);

        services.AddStackExchangeRedisCache(options =>  
        {  
            options.InstanceName = cacheOptions.InstanceName;  
            options.ConfigurationOptions = configurationOptions;  
            options.ConnectionMultiplexerFactory = async () =>  
            {  
                var connectionMultiplexer = await CreateRedisConnectionAsync(cacheOptions);  
                return connectionMultiplexer;

            };  
        });

I tried your proposed scope "https://*.cacheinfra.windows.net:10225/appid/.default" as well as "https://redis.azure.com/.default". Both returned a token, but both would lead to Auth failures on Redis

Old code is adding the MSI ID like so and does not use CreateRedisConnectionAsync:

        var configurationOptions = ConfigurationOptions.Parse(cacheOptions.DistributedCacheEndpoint!);

configurationOptions.ConfigureForAzureWithUserAssignedManagedIdentityAsync(cacheOptions.UserManagedIdentityClientId!.Value.ToString());

Auth error looks like so:

AuthenticationFailure on xyz.westeurope.redis.azure.net:10000/Interactive, Flushed/ComputeResult, last: ECHO, origin: SetResult, outstanding: 0, last-read: 0s ago, last-write: 0s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 0s ago, v: 2.8.24.3255 Error: NOAUTH Authentication required Verify if the Redis password provided is correct. Attempted command: ECHO

Is it really a token problem or is Redis in a corrupt state?

After redeploying the Managed Redis resource our old code which uses

configurationOptions.ConfigureForAzureWithUserAssignedManagedIdentityAsync(cacheOptions.UserManagedIdentityClientId!.Value.ToString());

services.AddStackExchangeRedisCache(options =>             {                 options.InstanceName = cacheOptions.InstanceName;                 options.ConfigurationOptions = configurationOptions;
});

The connection is working again.

Switching to CreateRedisConnectionAsync which creates a token on its own, I get authentication errors. The tokens look ok: I find our tenant id, the client ID of the user managed identity and the audience "https://redis.azure.com" in that token.

options.ConnectionMultiplexerFactory = async () =>
{
   var connectionMultiplexer = await CreateRedisConnectionAsync(cacheOptions);
   return connectionMultiplexer;
};

Is there any documentation how to create a working token for Managed Redis with a user managed identity?

Some more digging: if I pass the token as password I need to set the User name to the ID of the managed identity - "1234-5678" in the following example.

public static async Task CreateRedisConnectionAsync()
{
    var credential = new DefaultAzureCredential(new DefaultAzureCredentialOptions { ManagedIdentityClientId = "1234-5678");
    var tokenRequestContext = new TokenRequestContext(new[] { "https://redis.azure.com/.default" });
    var token = await credential.GetTokenAsync(tokenRequestContext);
    var options = new ConfigurationOptions
    {
        EndPoints = { "your-redis-name.redis.azure..net:somePort" },
        Ssl = true,
        AbortOnConnectFail = false,
        User = "1234-5678"
        Password = token.Token
    };
    return await ConnectionMultiplexer.ConnectAsync(options);
}

Now I can connect to Redis.

Question remains, how to refresh the token after I called ConnectionMultiplexer.ConnextAsync and added that to the service collection.

And for our previous code, where we used this nuget, the documentation states that the refresh takes place: 'configures a StackExchange.Redis connection to use the token for authentication. It also maintains the token, proactively refreshing it and re-authenticating the connection to maintain uninterrupted communication with the cache over multiple days'.

Not sure why they say "over multiple days" and not "forever".

Hi Sven Carstensen,

Could you please share the details requested in the private message?

Hi Sven Carstensen,

Typically, we see TimeOutException errors being raised by StackExchange Redis due to client-side threading or performance issues. I notice the following output from the exception:

(RedisServerException) Error: WRONGPASS invalid username-password pair ConnectTimeout, command=HMSET, timeout: 5000, inst: 0, qu: 1, qs: 0, aw: False, bw: CheckingForTimeout, rs: ReadAsync, ws: Idle, in: 0, last-in: 0, cur-in: 0, sync-ops: 9, async-ops: 568, serverEndpoint: XYZ.westeurope.redis.azure.net:10000, conn-sec: 16.79, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: ABC (SE.Redis-v2.8.24.3255), PerfCounterHelperkeyHashSlot: 10240, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=77,Free=32690,Min=1,Max=32767), POOL: (Threads=106,QueuedItems=1,CompletedItems=2321851,Timers=27), v: 2.8.24.3255

•  You can see .NET threadpool worker thread busy count is currently 77 while the minimum thread count is 1.
•  Anytime busy counter for IOCP or worker threads is > the Min count the .NET thread pool will throttle the creation of new threads. This can lead to Redis timeouts.

You can find details here:

 https://azure.microsoft.com/en-us/documentation/articles/cache-faq/#important-details-about-threadpool-growth https://gist.github.com/JonCole/db0e90bedeb3fc4823c2#burst-of-traffic

Cache management FAQs - Azure Cache for Redis | Microsoft Learn

• Another thing that leads to the timeouts are client-side performance issues particular high CPU and/or memory usage of the Redis client application. You can find details about the other client-side issues that can cause Redis client timeouts here:   https://gist.github.com/JonCole/db0e90bedeb3fc4823c2

I have the following recommendations:

1. Please following the recommendation from this link to raise the minimum thread count. You can do this in your code. For Worker Roles you can do this at the beginning of your role's Run method, or for Web Roles or Web Apps you can do this as the first line of code in your global.asax.cs Application_Start(). Here is the line of code to add:

      System.Threading.ThreadPool.SetMinThreads(200, 200); 
   

   NOTE: When configuring this in code this is not a per-core setting, the value you specify in code will be the value used for the Min counts. The Min value you set will depend on the load you expect on the application. 200 is a good starting value, but if your busy count exceeds 200 then you may need to raise it higher.
2. When you raise the Min thread counts this will allow more requests to run simultaneously because you remove the throttling so this will put additional resource usage on your client instances. You need to monitor this web application and VM hosting the web application for memory usage and CPU utilization. If you see these spike during the time you are receiving the Redis timeout errors then I recommend fixing the issue causing high memory and/or CPU utilization. This might include scaling out to more instances or scaling up to a larger VM size. Please see https://gist.github.com/I90Runner/911cd00ceeeeb473074c4e320e9fa287 .

I hope this information helps. Please do let us know if you have any further queries.

I switched back to ConfigureForAzureWithUserAssignedManagedIdentityAsync and redeployed our containers. Now we typically have to wait a few days until the connection problems to Redis start.

Are you sure that our issue is a client side one? Should not restarting the containers fix that problem, if its client based? We cannot fix that error with client restarts, only deleting and recreating the Redis instance helped up.

When the error reoccurs I will add the call to SetMinThreads.

After our Redis did not allow connections from our container apps, we enabed access key access and we can connect with an access key. Managed Identity cannot connect. So, the problem seems to be related to the connection method used.

Anyone else seeing this problem when using managed identies to connect to Redis?