sequenceDiagram
participant S as Shadow
participant D as Device
S-->>D: delta {sendSlot: "5-100"}
Note over D: Send to the light (on)
D->>S: reported {sendSlot: null}
Note over S: sendSlot is removed from reported<br>→ they still differ
S-->>D: delta {sendSlot: "5-100"} (redelivered)
Note over D: Send to the light (off)
D->>S: reported {sendSlot: null}
S-->>D: delta {sendSlot: "5-100"} (redelivered)
Note over D: ... and so on
5 Pitfall 2: Infinite Loops
In the original project, partway through applying the countermeasures from the previous chapter, we caused an accident in which infrared kept being sent to the light. The light kept turning on and off, and in the end we stopped it by unplugging the ESP32’s USB cable. In this chapter we safely reproduce the same accident with the virtual device and learn its cause and how to prevent it.
The virtual device has a safety device (--max-exec, default 10) that stops it on its own when the number of sends reaches that count. Even so, after it stops, always perform an emergency stop (send.py stop).
5.1 The cause is a single rule
Recall rule 5 from Chapter 1.
After an update is accepted, if delta is not empty, the entire delta is delivered to the device. This happens also for reported updates, and even for updates to unrelated fields
In other words, if desired and reported still differ after the device writes back, that write-back itself triggers the same delta to arrive again. The device executes it again, writes back again, and it arrives again —— this is the infinite loop.
5.2 Trying four ways of writing back
The virtual device’s --bug option reproduces the write-back methods that were actually tried and failed in the original project (in bug-reproduction modes the duplicate check is also disabled, just as it was then).
5.2.1 Writing back null (--bug null)
The idea is “once processed, sendSlot is no longer needed, so let’s delete it”.
python device/virtual_device.py --bug null # terminal A
python device/send.py slot 5 # terminal C[ 5.3s] delta received (push): {"sendSlot": "5-1791382600466"}
[ 5.3s] >>> IR send #1: slot 5 "Light toggle"
[ 5.3s] <<< reported update: {"sendSlot": null}
[ 5.4s] delta received (push): {"sendSlot": "5-1791382600466"}
[ 5.4s] >>> IR send #2: slot 5 "Light toggle"
[ 5.4s] <<< reported update: {"sendSlot": null}
... (omitted) ...
[ 5.7s] >>> IR send #8: slot 5 "Light toggle"
[ 5.7s] !!! Safety device: sends reached 8, stopping.
It sent 8 times in 0.4 seconds (run with --max-exec 8). null means “delete the field”, so sendSlot disappears from reported, which makes the discrepancy with desired all the more certain.
5.2.2 Writing back with a different type (--bug number)
This is the case where the Lambda sends cmdId as a string ("1791382616543") but the device writes it back as a number (1791382616543).
python device/virtual_device.py --bug number
python device/send.py ac --power on[ 5.3s] delta received (push): {"power": true, "cmdId": "1791382616543"}
[ 5.3s] >>> IR send #1: air conditioner {...}
[ 5.3s] <<< reported update: {"power": true, ..., "cmdId": 1791382616543}
[ 5.3s] delta received (push): {"cmdId": "1791382616543"}
[ 5.3s] >>> IR send #2: air conditioner {...}
... (8 times in 0.4 s) ...
They look like the same number, but a different type means a different value. If you build JSON by string concatenation in C++, simply forgetting the double quotes causes this.
5.2.3 Not writing back (--bug nowrite)
[ 5.2s] delta received (push): {"sendSlot": "5-1791382638632"}
[ 5.2s] >>> IR send #1: slot 5 "Light toggle"
[ 10.1s] delta received (get): {"sendSlot": "5-1791382638632"}
[ 10.1s] >>> IR send #2: slot 5 "Light toggle"
[ 15.1s] delta received (get): {"sendSlot": "5-1791382638632"}
[ 15.1s] >>> IR send #3: slot 5 "Light toggle"
... (continues every 5 seconds) ...
There is only one push, but because the delta keeps remaining, it is re-executed at every 5-second poll (get). The speed of the loop equals the polling interval.
5.2.4 Writing back in two steps (--bug split)
This is the case where the air-conditioner state is written back, and then cmdId is written back in a separate publish.
[ 5.3s] delta received (push): {"temp": 25, "cmdId": "1791382664665"}
[ 5.3s] >>> IR send #1: air conditioner {...}
[ 5.3s] <<< reported update: {"power": false, "mode": "cool", "temp": 25, "fan": "auto"}
[ 5.3s] <<< reported update: {"cmdId": "1791382664665"}
[ 5.3s] delta received (push): {"cmdId": "1791382664665"}
[ 5.3s] >>> IR send #2: air conditioner {...}
Right after the first write-back, there is an intermediate state in which only cmdId is still old. AWS delivers a delta saying “cmdId still differs” at that moment, so the device sends once more. In this experiment the second write-back made them match and it stopped after 2 sends, but in the original project this was observed as a loop (probably in combination with other factors). Either way, infrared going out two or more times for a single operation is an accident. With a toggle-type light, it turns on and immediately off again.
5.3 The correct way to write back
| Rule | Reason |
|---|---|
| Write the processed value back to reported as is | Makes desired and reported match, clearing the delta |
| Write back with the same type (keep strings as strings) | Different types do not match |
Do not clear with null |
null means deletion, which makes the discrepancy certain |
| Combine multiple fields into a single publish | One update is applied atomically on the AWS side, so no intermediate state is visible from outside |
With the correct implementation (--bug none), a single operation sends exactly once and the delta disappears (Chapter 3).
5.4 Reproducing it in the simulator
Switch the “device implementation” and press the light button or the air conditioner. Pay attention to how version and the send count increase. When sends reach 20, the safety device stops it. For readability, the simulator slows one message round trip down to 0.35 seconds (on real AWS it was 8 times in 0.4 seconds).
5.5 Safety checks before trying on the production device
In the original project, we repeatedly “tested by talking to Alexa” after every change, and the real light malfunctioned with every loop. This is the procedure we settled on after the accident.
- After starting the device, run
send.py watchin another terminal - Before asking anyone to operate the real device (speaking to Alexa, etc.), send a command yourself just once
- Wait 15–20 seconds and check two things
- Did the device send exactly once?
- Is version not increasing continuously? (Normally it increases by 2 for one operation and stops: the desired update and the reported write-back)
- If version keeps increasing, perform an emergency stop immediately, then fix the cause
- Do not ask anyone to operate the real device until the single-send check passes
5.5.1 Emergency stop: clear desired
python device/send.py stop # writes {"state": {"desired": null}}Once desired is empty, no delta is produced and the loop stops (rule 2 in Chapter 1). The advantage of this method is that it can be stopped from the cloud side alone, however broken the device code may be. Unplugging the device is a last resort (after a reboot, it may receive the same delta again via the get right after connecting).
Even if desired says 35℃, if the air conditioner can only be set up to 30℃, reported becomes 30 and the discrepancy remains forever. With a duplicate check, the send is not repeated, but the delta keeps remaining and is delivered with every other reported update. Avoid leaving discrepancies by, for example, validating the range on the cloud side before writing to desired, or clearing the desired value of fields the device could not achieve ({"desired": {"temp": null}}).
- If a discrepancy remains, the device’s own write-back triggers redelivery of the delta, causing an infinite loop
- Write back with the same value and the same type, never null, combined into a single publish
- Before the production device, send once and monitor version. To stop, set desired to null