Azure Data Explorer - measures to improve concurrency, response time, throughput

Question

Azure Data Explorer - measures to improve concurrency, response time, throughput

databricksuser-5173 20

Hi

Reference: https://learn.microsoft.com/en-us/kusto/management/alter-merge-workload-group-command?view=microsoft-fabric

As per recommendations in above link, increasing "MaxConcurrentRequests" has improved concurrency. Are there any other settings designed for getting the best utilization of CPU, because I still see CPUs are not fully utilized.

Any recommendations to improve concurrent queries for below specific observations?

A) Response Times improvement

B) Throughput improvement

Regards

databricksuser-5173 20 Reputation points

2025-05-31T21:28:46.0166667+00:00

will it help creating a new workload along with default workload group? what are the performance benefits of such an additional user-defined workload group? what are the cost implications of an additional user-defined workload group?
Anonymous

2025-06-05T08:16:16.9+00:00

@databricksuser-5173
If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

1 answer

Your answer

databricksuser-5173 20 Reputation points

2025-05-31T21:28:46.0166667+00:00

will it help creating a new workload along with default workload group? what are the performance benefits of such an additional user-defined workload group? what are the cost implications of an additional user-defined workload group?
Anonymous

2025-06-05T08:16:16.9+00:00

@databricksuser-5173
If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

Anonymous

Hi @databricksuser-5173
It sounds like you're looking to optimize the performance of Azure Data Explorer, particularly in terms of CPU utilization, response times, and throughput for concurrent queries. Here’s what you can consider:

MaxConcurrentRequests: You've already increased the "MaxConcurrentRequests," which is great!
Workload Groups: Creating a new user-defined workload group can help, especially if you want to fine-tune the way queries are prioritized. This allows more control over the resources allocated to different types of queries. The performance benefits include:
1. More tailored resource management based on usage patterns.
2. Potentially improved response times for specific queries if they’re mapped to the right workload group with optimal settings.
Optimizing Data and Query Design:
1. Optimized Table Schema Design: Ensure your table schemas are optimized to minimize CPU usage. This includes choosing appropriate data types, using denormalization where beneficial, and avoiding large sparse tables.
2. Data Partitioning and Pre-aggregation: Using partitioning policies can boost performance. Materialized views can help pre-aggregate data to reduce the CPU load during query time.
3. Caching: Make use of caching strategies to minimize repeated CPU usage for frequently accessed queries.
Cluster Policies: Adjust the Request Rate Limit policy to allow for more concurrent requests as needed, but remember to test thoroughly to ensure your cluster can handle the updated limits.
writing efficient queries is essential. You could go use of .set query_trace=true or the explain operator to diagnose slow queries.
Keep an eye on your cluster's metrics to identify potential bottlenecks and areas for improvement. Azure Monitor can give you insights into query performance, resource usage, and more.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

databricksuser-5173 20 Reputation points

2025-06-02T21:30:29.4033333+00:00

Do both below approaches have the same impact on turning Cache to Hot?

Said otherwise, altering workload's Cache policy and altering the entire Database's Cache policy - do these both have the same effect?

Option 1

.alter-merge workload_group default

"RequestLimitsPolicy": { "DataScope": { "IsRelaxable": true, "Value": "HotCache" }

Option 2

.alter database MyDatabase policy caching hot = <some X days)
databricksuser-5173 20 Reputation points

2025-06-03T05:43:02.33+00:00

previously mentioned 2 options are as per below recommendations.

Do both below approaches have the same impact on turning Cache to Hot?

Said otherwise, altering workload's Cache policy and altering the entire Database's Cache policy - do these both have the same effect?

Option 1

https://learn.microsoft.com/en-us/kusto/management/alter-merge-workload-group-command?view=microsoft-fabric

Option 2

https://learn.microsoft.com/en-us/kusto/management/alter-database-cache-policy-command?view=microsoft-fabric
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Anonymous

2025-06-04T10:01:47.1066667+00:00
@databricksuser-5173

Do both below approaches have the same impact on turning Cache to Hot?

No, both approaches do not have the same effect , they target different layers of caching behavior in Azure Data Explorer:

Option 1:Workload Group Cache Policy

This setting controls the query behavior at the workload level.

Specifically, "DataScope": { "Value": "HotCache" } requests the engine to prefer hot cache reads for that query group.

However, this is only a request — the query might still access cold storage if data isn't cached or available.

Option 2: Database cache policy

This defines the retention duration of data in hot cache for the entire database.

For example: .alter database MyDatabase policy caching hot = 7d ensures that newly ingested data is kept in RAM for 7 days.

This guarantees hot cache availability of the data for the defined time window.

To summaries both options

Option 1: A query-level hint/request to prioritize hot cache when available.

Option 2: A data retention policy to actively keep data hot in memory.

Use both in tandem: Option 2 ensures data is in hot cache; Option 1 tells the query engine to use it.

I hope this info helps, If you have any queries, please reach out to us We are happy assist on further queries

Share via

Azure Data Explorer - measures to improve concurrency, response time, throughput

1 answer

Your answer