Architecture best practices for Log Analytics

Review service limits for Log Analytics workspaces: The service limits section explains restrictions on data collection, data retention, and other aspects of the service. These limits help you design an effective workload observability strategy. Make sure that you review Azure Monitor service limits because many functions, such as queries, work in tandem with Log Analytics workspaces.

Plan for workspace resilience and recovery: Log Analytics workspaces are regional and don't have built-in support for cross-regional redundancy or replication. Availability zone redundancy options are limited. Because of these limitations, you should determine the reliability requirements of your workspaces and strategize to meet those targets.

Your requirements might stipulate that your workspace must be resilient to datacenter failures or regional failures. Or they might stipulate that you must be able to recover your data to a new workspace in a failover region.

Each of these scenarios require extra resources and processes to be successful, so carefully consider how to balance your reliability targets with cost and complexity.

Choose the right deployment regions to meet your reliability requirements: Deploy your Log Analytics workspace and data collection endpoints (DCEs) co-located with the workload components that emit operational data. Where you deploy your workload should inform your choice of the appropriate region in which to deploy your workspace and your DCEs.

You might need to weigh the regional availability of specific Log Analytics functionality, such as dedicated clusters, against other factors that are more central to your workload's reliability, cost, and performance requirements.

Exclude workspaces from critical path dependencies: Log Analytics workspaces serve as important components of your observability system, but you shouldn't include them in your workload's critical path. These workspaces collect and store operational data that's essential for monitoring and troubleshooting. However, the core functionality of your workload must remain independent of workspace availability. This architectural separation ensures that observability system outages don't cascade into workload runtime failures.

Ensure that your observability systems are healthy: Like any other component of your workload, ensure that your monitoring and logging systems are functioning properly. To achieve reliable observability, enable features that send health data signals to your operations teams. Set up health data signals specific to your Log Analytics workspaces and associated resources.

Review security best practices: Review best practices for security in the Azure Monitor security baseline and Manage access to Log Analytics workspaces articles.

Deploy your workspaces with segmentation as a cornerstone principle: Implement segmentation at the networking, data, and access levels. Segmentation helps ensure that your workspaces are isolated to the appropriate degree. It also helps protect your workspaces from unauthorized access to the highest degree possible, while still meeting your business requirements for reliability, cost optimization, operational excellence, and performance efficiency.

Ensure that you can audit workspace reads and writes activities and associated identities: Attackers can benefit from viewing operational logs. A compromised identity can lead to log injection attacks. Enable auditing of operations that run from the Azure portal or through API interactions and the associated users.

If you aren't set up to audit your workspace, you might put your organization at risk of breaching compliance requirements.

Implement robust network controls: Help secure network access to your workspace and your logs through network isolation and firewall functions. Insufficiently configured network controls increase the risk of unauthorized or malicious access.

Determine what types of data need immutability or long-term retention: Your log data should be treated with the same rigor as workload data inside production systems. Include log data in your data classification practices to ensure that you successfully store sensitive log data according to its compliance requirements.

Protect log data at rest through encryption: Segmentation alone won't completely protect the confidentiality of your log data. If unauthorized raw access occurs, encrypting the log data at rest helps prevent bad actors from using that data outside of your workspace.

Protect sensitive log data through obfuscation: Just like workload data that resides in production systems, you must take extra measures to ensure that confidentiality is retained for sensitive information that might be intentionally or unintentionally present in operational logs. When you use obfuscation methods, it helps you hide sensitive log data from unauthorized access.

Perform cost modeling exercises: These exercises help you understand your current workspace costs and forecast your costs relative to workspace growth. Analyze your growth trends in your workload and ensure that you understand plans for workload expansion to properly forecast your future operational logging costs.

Choose the right billing model: Use your cost model to determine the best billing model for your scenario. How you currently use your workspaces and how you plan to use them as your workload evolves determines whether a pay-as-you-go or commitment tier model is best suited for your scenario.

Remember that you can choose different billing models for each workspace. You can also combine workspace costs in specific cases, so you can be granular in your analysis and decision-making.

Collect just the right amount of log data: Perform regularly scheduled analysis of your diagnostic settings on your resources, data collection rule configuration, and custom application code logging to ensure that you don't collect unnecessary log data.

Treat nonproduction environments differently than production environments: Review your nonproduction environments to ensure that your diagnostic settings and retention policies are configured appropriately. These settings and policies can often be significantly less robust than in production, especially for dev/test or sandbox environments.

Use infrastructure as code (IaC) for all functions related to your workload's Log Analytics workspaces: Minimize the risk of human error that can occur from manually administering and operating your log collection, ingestion, storage, and querying functions, including saved queries and query packs, by automating as many of those functions as possible through code.

Also, include alerts that report health status changes and the configuration of diagnostic settings for resources that send logs to your workspaces in your IaC code. Include the code with your other workload-related code to ensure that your safe deployment practices are maintained for the management of your workspaces.

Ensure that your workspaces are healthy and that you're notified when problems occur: Like any other component of your workload, your workspaces can encounter problems. These problems can consume valuable time and resources to diagnose and fix, and they might leave your team unaware of the production workload's status. Proactive monitoring of workspaces and early problem mitigation enable operations teams to reduce time spent on troubleshooting and repairs.

Separate your production from nonproduction workloads: Avoid unnecessary complexity that can cause extra work for an operations team by using different workspaces for your production environment than those used by nonproduction environments. Comingled data can also lead to confusion because testing activities might appear to be events in production.

Prefer built-in tools and functions over non-Microsoft solutions: Use built-in tools to extend the functionality of your monitoring and logging systems. You might need to put extra configurations in place to support requirements like recoverability or data sovereignty that aren't available out-of-the-box with Log Analytics workspaces. In these cases, when practical, use native Azure or Microsoft tools to minimize the number of tools that your organization must support.

Treat your workspaces as static rather than ephemeral components: Like other types of data stores, workspaces shouldn't be considered among the ephemeral components of your workload. The Well-Architected Framework generally favors immutable infrastructure and the ability to quickly and easily replace resources within your workload as part of your deployments. But the loss of workspace data can be catastrophic and irreversible.

For this reason, leave workspaces out of deployment packages that replace infrastructure during updates, and only perform in-place upgrades on the workspaces.

Ensure that the operations staff is trained on Kusto Query Language: Train staff to create or modify queries when needed. If operators are unable to write or modify queries, it can slow critical troubleshooting or other functions because operators must rely on other teams to do that work for them.

Familiarize yourself with the fundamentals of log data ingestion latency in Azure Monitor: There are several factors that contribute to latency when ingesting logs into your workspaces. Many of these factors are inherent to the Azure Monitor platform.

Understanding the factors and the normal latency behavior can help you set appropriate expectations within your workload operations teams.

Separate your nonproduction and production workloads: Production-specific workspaces mitigate any overhead that nonproduction systems might introduce. Separation reduces the overall footprint of your workspaces by requiring fewer resources to handle log data processing.

Choose the right deployment regions to meet your performance requirements: Deploy your Log Analytics workspace and DCEs close to your workload. Where you deploy your workload should inform your choice of the appropriate region in which to deploy your workspace and your DCEs.

You might need to weigh the performance benefits of deploying your workspaces and DCEs in the same region as your workload against your reliability requirements if you have already deployed your workload into a region that can't support those requirements for your log data.