We are experiencing a failure in sending IOT direct calls to iot devices, even though we see the device receives the command

Hi Jonathan Volij

Thank you for pointing out that "rebooting the device fixes the issue, but not clear how come the device gets the call but the backend thinks its offline"

It means that error likely occurred because the direct method callback wasn't registered on the device.

Possible reasons

1. If the device hasn't sent a heartbeat or pinged the cloud recently, it may be marked as offline
2. Cloud-to-device (C2D) message routing delay
3. Device might be behind firewall causing delay in communication.
4. Some platforms queue commands for offline devices, but if the device doesn’t acknowledge receipt within a timeout, the backend assumes failure.

Remedials

• Check in activity logs on cause of Device failures
• Log connection status on the device and backend.
• Use device twin/shadow to track and sync online/offline state.
• Add retry logic in your backend for transient 404 errors.
• Monitor keep-alive pings and optimize TTL, timeout , retry to detect and fix stale connections.

Here is a more resilient code

const openDoorCommand = async function(request, response) {
    try {
        const payload = request.payload;
        track(`IoT open door command received: ${payload}`);
        metric("remote-open-door", 1);
        const deviceStatus = await checkDeviceStatus();
        if (!deviceStatus.online) {
            track(`Device offline: ${deviceStatus.reason || 'No heartbeat'}`);
            response.send(503, 'Device is offline. Try again later.');
            return;
        }
        const maxRetries = 3;
        const baseDelay = 500; // milliseconds
        let attempt = 0;
        let success = false;
        let lastError = null;
        while (attempt < maxRetries && !success) {
            try {
                const result = await openDoor();
                if (result.success) {
                    success = true;
                    response.send(200, 'Opening door');
                    return;
                } else {
                    lastError = result.error;
                    track(`Attempt ${attempt + 1} failed: ${result.error}`);
                }
            } catch (err) {
                lastError = err.message;
                track(`Attempt ${attempt + 1} threw error: ${err.message}`);
            }
            attempt++;
            if (attempt < maxRetries) {
                await new Promise(resolve => setTimeout(resolve, baseDelay * Math.pow(2, attempt)));
            }
        }
        response.send(500, `Failed to open door after ${maxRetries} attempts. Last error: ${lastError}`);
    } catch (err) {
        track(`Unexpected error: ${err.message}`);
        response.send(500, 'Unexpected error occurred while processing the command');
    }
};

Reference used -

https://learn.microsoft.com/en-us/azure/iot-hub/troubleshoot-error-codes?WT.mc_id=AZ-MVP-5002324#404103-device-not-online

https://learn.microsoft.com/en-us/azure/iot/iot-mqtt-connect-to-iot-hub#use-the-mqtt-protocol-directly-from-a-device (for SDK optimization)

Hope it helps.

Thank you.

Hi Jonathan Volij

Please let us know if the above pointers helped debug the issue better.

Thank you.