Hello Liam Woollery,
This could be expected because compliance policy checks and reports but doesn't enforce. It might have worked during enrollment when configured along with conditional access with the option "Require Device to be marked compliant".
To achieve your requirement, configure a Device Restriction Configuration Policy that will enforce and prompt to set/reset the password accordingly.
Refer to: Restrict devices features using policy in Microsoft Intune | Microsoft Learn
Android configuration list for Intune settings catalog | Microsoft Learn
Please note this setting only works for fully managed, dedicated, and corporate-owned work profile devices.
Hope that helps!
If you found the information above helpful, please Click Yes. This will assist others in the community who encounter a similar issue, enabling them to quickly find the solution and benefit from the guidance provided.