Cohesity Azure cluster space Issue in Azure Clusters caused by Monitor Agent
How do I disable the azure monitor agent or reduce its logging limit, as per Microsoft recommendations?
Symptom
Alert reports DiskSpaceLow, on Cohesity in Azure clusters:
ALERT CODE : CE00101007 ALERT TYPE : 1007 ALERT CATEGORY : Disk ALERT SEVERITY : Info ALERT STATE : Open LAST RAISED TIME : Thursday, 14-Apr-22 09:00:00 EDT ALERT NAME : DiskSpaceLow ALERT DESCRIPTION : Disk free space is low on the node 904324852649638 with ip xxx.xxx.xxx.xxx. ALERT CAUSE : Free disk space 3% on / is below the threshold 5%
Cause
This is a known scenario caused by Azure Monitoring Agent overlogging.
Resolution
To confirm this is the exact scenario you're experiencing, follow the steps below: Step 1 of 3. The /var dir should be below 200 MB, use the command below to check:
$ sudo du -xh -d1 / | sort This example shows a problematic /var size.
Step 2 of 3. Check the amount of space being used by /azuremonitoragent:
$ sudo du -xhd1 /var/opt/microsoft/azuremonitoragent | sort This example shows most space being used by Azure Monitor Agent (events).
Step 3 of 3. Disable the azure monitor agent or reduce its logging limit, as per Microsoft recommendations.
- More details can be found at How to troubleshoot issues with the Log Analytics agent for Linux.
- For assistance with Azure's Monitoring Tool, please contact Microsoft Support and provide the above information.
See Also
- Article Number 000003089: Alert: CE00101007 DiskSpaceLow.
- Cohesity recommends enabling Support Channel for remote access by a qualified support engineer.