Optimize firmwares for larger networks

[Koenkk]

According to http://www.ti.com/lit/an/swra427c/swra427c.pdf (4 Z-Stack Knobs) there are a few parameters which can be tuned to improve the stability of larger networks.

I found MAX_RTG_SRC_ENTRIES the most interesting one. This parameter determines how much routes are saved. By default it only saves 12 routers, that means if having more than 12 devices, the coordinator has to rediscover routers to devices using the AODV mechanism.

@ptvoinfo @kirovilya I think some of the parameters can also be used to optimize the router firmwares.

[Koenkk]

For those wanting to test the new firmware: https://drive.google.com/open?id=1xcGP5tGBBeAaU8VNQG7BiE70H2Bty5pe

I've currently migrated my production zigbee network to this firmware.

First impression: bulbs respond better

Note: advised to be used with the latest dev branch.

[ptvoinfo]

@Koenkk What do you think it optimizes? Allows you to add more routers?

[Koenkk]

The coordinator can now keep more paths to other devices, which avoid rediscovery of paths.

[lolorc]

@Koenkk would you be able to provide a cc2530+cc2591 firmware with hardware flow control disabled please ? :-)
In the mean time, it might be the time I start compiling my own firmware :-)

[HKC001]

@lolorc Yes please do!

I think many users have raised issues with cc2531's short range, and cc2530 + cc2591 looks like a nice cost effective replacement. But for us non-technical people, we really need a good tutorial (covering the right equipment and firmware) like the one for the cc2531.

[Koenkk]

Just pushed the instructions to https://github.com/Koenkk/Z-Stack-firmware/tree/optimize/coordinator/CC2531. Will try this week if I can also compile it for CC2530+CC2591.

[lolorc]

@HKC001 will try, but I'm not a windows user, I wish this could compiled on linux with a "free" compiler

[Chrischi-]

for people who don't have a debugger, i compiled a .bin that works with TI's SerialBootTool.

See comment here: #543 (comment)

[layanto]

Please compile cc2530 (w/o cc2591) firmware too.

[layanto]

Is the max number of ZigBee end devices without any router still 15 with this firmware?

[Koenkk]

@layanto the number has been increased to 20

[ben423423n32j14e]

#Koenkk when you migrate your network, do you have a method for backup / restore of devices or are you manually repairing everything?

[Koenkk]

@ben423423n32j14e you have to manually pair it again.

[Koenkk]

@lolorc the CC2530_CC2591 is up: https://github.com/Koenkk/Z-Stack-firmware/tree/optimize/coordinator/CC2530_CC2591

[cbu99]

@Koenkk Could you please update the simple CC2530, too?

[Skeletorjus]

Is there a chance that this optimization will remedy the occasional error 205 (No network route) on some bulbs?
Network with 32 devices, mostly routers (Trådfri-bulbs).

[Koenkk]

@Skeletorjus yes

[HKC001]

Requesting that @ptvoinfo consider if this will enhance his very useful CC2531 Router firmware and update if necessary. :)

[ben423423n32j14e]

This is great work but isn't there Z-Stack 3 in progress which supports even more devices?

[ptvoinfo]

@Koenkk The document in the first message says that MAX_RTG_SRC_ENTRIES is related to concentrators (coordinators). Do you think it may help on routers too? What value do you use in your firmware?

[Koenkk]

@ptvoinfo here you can see the parameters I use: https://github.com/Koenkk/Z-Stack-firmware/tree/optimize/coordinator/CC2531

This parameter is not useful for routers. See http://www.ti.com/lit/an/swra427c/swra427c.pdf (4 Z-Stack Knobs), it specifies which parameters are relevant for routers/coordinator.

I think especially MAX_NEIGHBOR_ENTRIES is interesting for the routers. What is the current value?

[Koenkk]

@ben423423n32j14e I did some test with z-stack 3.0; my guess is that there is still lot of work needed to get this stable.

[ben423423n32j14e]

@Koenkk Thanks, my network got the can't find route error last night on previous firmware, I'll upgrade to this one. Cheers

[Imperial-Guard]

Using version CC2530_CC2591ZNP-Prod_20181030, flashed the CC2531 yesterday morning and till now looks only beter then the previous version 👍

[lolorc]

Thanks @Koenkk
I managed to upgrade without repairing all my devices.
All my devices have reported their status & values the last hour. The funny thing is most of them aren't "connected" through any router/coordinator when generating the network map. (this is not something new with this firmware)

[ben423423n32j14e]

@lolorc Interestingly some of my devices did not require repairing, in particular:

Xiaomi Temp Hum Baro sensor
Xiaomi Tilt Fall Vibration sensor

My other devices did require repairing, still interesting that some didn't, I wonder why.

[lolorc]

@ben423423n32j14e are you sure you're still using the same channel & panid ?
I've seen weird things, like some devices still paired while panid has been increased by 1.

[ben423423n32j14e]

@lolorc I have no idea, I just used the Texas SmartRF Flash Programmer tool.

Would be really nice if we could figure out how to upgrade firmware and keep device associations, seems it may be possible.

[Chrischi-]

It is possible via TI's Serial Boot Tool. I've never had to repair my devices since i use this method.

ATTN! You can't use the same firmware as for the flasher.

[lolorc]

@ben423423n32j14e you should be able to find that in your logs if you run in debug mode.
i guess this "zigbee-shepherd info" should be made available as an info log from zigbee2mqtt.
Or run a zigbee sniffer to see what panid your devices and your coordinator are using.

[HKC001]

The new firmware on a CC2530 + RF isn't working for me. I had 26 devices joined, and everything was working fine. After adding 2 more devices (WXKG03LM & WXKG01LM), 2 previously attached QBKG03LMs started getting 205 errors. Errors persisted even after removing and re-joining the QBKG03LMs.

Network Map:

digraph G {
node[shape=record];
  "0x00124b00076622b8" [label="{0x00124b00076622b8|Coordinator|No model information available|online}"];
  "0x00124b00076622b8" -> "0x00124b0014d9db92" [label="99"]
  "0x00124b0014d9db92" [label="{2531r_1|Router|Custom devices (DiY) [CC2530 router](http://ptvo.info/cc2530-based-zigbee-coordinator-and-router-112/) (CC2530.ROUTER)|online}"];
  "0x00124b0014d9db92" -> "0x00124b00076622b8" [label="61"]
  "0x00158d00024f1d15" [label="{stud_pow_1|Router|Xiaomi Mi power plug ZigBee (ZNCZ02LM)|online}"];
  "0x00158d00024f1d15" -> "0x00124b0014d9db92" [label="91"]
  "0x00158d0002b95ec5" [label="{stud_ipir_1|EndDevice|Xiaomi Aqara human body movement and illuminance sensor (RTCGQ11LM)|online}"];
  "0x00158d0002b95ec5" -> "0x00124b0014d9db92" [label="87"]
  "0x00158d0002b44a97" [label="{livr_ipir_1|EndDevice|Xiaomi Aqara human body movement and illuminance sensor (RTCGQ11LM)|online}"];
  "0x00158d0002b44a97" -> "0x00124b00076622b8" [label="117"]
  "0x00158d0002347f00" [label="{bhall_pir_1|EndDevice|Xiaomi MiJia human body movement sensor (RTCGQ01LM)|online}"];
  "0x00158d0002347f00" -> "0x00124b00076622b8" [label="109"]
  "0x00158d0002a9b999" [label="{balc_swds_1|Router|Xiaomi Aqara single key wired wall switch (QBKG04LM)|online}"];
  "0x00158d0002a9b999" -> "0x00124b00076622b8" [label="101"]
  "0x00158d00029ba7dc" [label="{mhall_swds_1|Router|Xiaomi Aqara single key wired wall switch (QBKG04LM)|online}"];
  "0x00158d00029ba7dc" -> "0x00124b00076622b8" [label="86"]
  "0x00158d0002b42f31" [label="{kitc_ipir_1|EndDevice|Xiaomi Aqara human body movement and illuminance sensor (RTCGQ11LM)|online}"];
  "0x00158d0002b42f31" -> "0x00124b00076622b8" [label="56"]
  "0x00158d00026a35a9" [label="{mhall_pir_1|EndDevice|Xiaomi MiJia human body movement sensor (RTCGQ01LM)|online}"];
  "0x00158d00026a35a9" -> "0x00124b00076622b8" [label="84"]
  "0xd0cf5efffe1078e3" [label="{bedr_tlight_1|Router|IKEA TRADFRI LED bulb E26/E27 980 lumen, dimmable, white spectrum, opal white (LED1545G12)|offline}"];
  "0xd0cf5efffe1078e3" -> "0x00124b0014d9db92" [label="0"]
  "0x00158d0002a35f6a" [label="{bedr_dwds_1|Router|Xiaomi Aqara double key wired wall switch (QBKG03LM)|online}"];
  "0x00158d00026e2964" [label="{livr_dwds_2|Router|Xiaomi Aqara double key wired wall switch (QBKG03LM)|online}"];
  "0x00158d00026e2964" -> "0x00124b00076622b8" [label="104"]
  "0x00158d0002a35f4e" [label="{bedr_dwds_2|Router|Xiaomi Aqara double key wired wall switch (QBKG03LM)|online}"];
  "0x00158d000282a0f6" [label="{livr_dwls_1|EndDevice|Xiaomi Aqara double key wireless wall switch (WXKG02LM)|online}"];
  "0x00158d000282a0f6" -> "0x00124b00076622b8" [label="110"]
  "0x000b57fffe8f6927" [label="{stud_tlight_1|Router|IKEA TRADFRI LED bulb E27 1000 lumen, dimmable, opal white (LED1623G12)|online}"];
  "0x000b57fffe8f6927" -> "0x00124b0014d9db92" [label="33"]
  "0x00158d000227d3c9" [label="{stud_tmp_1|EndDevice|Xiaomi MiJia temperature & humidity sensor  (WSDCGQ01LM)|online}"];
  "0x00158d000227d3c9" -> "0x00124b0014d9db92" [label="56"]
  "0x00158d00020f542e" [label="{livr_tmp_1|EndDevice|Xiaomi MiJia temperature & humidity sensor  (WSDCGQ01LM)|online}"];
  "0x00158d00020f542e" -> "0x00124b0014d9db92" [label="1"]
  "0x00158d0002379552" [label="{bedr_tmp_1|EndDevice|Xiaomi MiJia temperature & humidity sensor  (WSDCGQ01LM)|online}"];
  "0x00158d0002379552" -> "0x00124b0014d9db92" [label="1"]
  "0x00158d0002347d95" [label="{frnt_pir_1|EndDevice|Xiaomi MiJia human body movement sensor (RTCGQ01LM)|online}"];
  "0x00158d0002347d95" -> "0x00124b0014d9db92" [label="170"]
  "0x00158d0002367588" [label="{bath_tmp_1|EndDevice|Xiaomi MiJia temperature & humidity sensor  (WSDCGQ01LM)|online}"];
  "0x00158d0002367588" -> "0x00124b0014d9db92" [label="35"]
  "0x00158d0002a71718" [label="{stud_dwds_1|Router|Xiaomi Aqara double key wired wall switch (QBKG03LM)|online}"];
  "0x00158d0002b478f3" [label="{bath_ipir_1|EndDevice|Xiaomi Aqara human body movement and illuminance sensor (RTCGQ11LM)|online}"];
  "0x00158d0002b4785d" [label="{bedr_ipir_1|EndDevice|Xiaomi Aqara human body movement and illuminance sensor (RTCGQ11LM)|online}"];
  "0x00158d0002478a04" [label="{balc_ptmp_1|EndDevice|Xiaomi Aqara temperature, humidity and pressure sensor (WSDCGQ11LM)|online}"];
  "0x00158d0002478a04" -> "0x00124b0014d9db92" [label="4"]
  "0x00158d00028f7d0e" [label="{bedr_swls_1|EndDevice|Xiaomi Aqara single key wireless wall switch (WXKG03LM)|online}"];
  "0x00158d00028f7d0e" -> "0x00124b0014d9db92" [label="1"]
  "0x00158d000252d579" [label="{livr_tog_1|EndDevice|Xiaomi MiJia wireless switch (WXKG01LM)|online}"];
  "0x00158d000252d579" -> "0x00124b0014d9db92" [label="56"]
  "0x00158d0002a35f40" [label="{bath_dwds_1|Router|Xiaomi Aqara double key wired wall switch (QBKG03LM)|online}"];
  "0x00158d0002a35e54" [label="{bhall_dwds_1|Router|Xiaomi Aqara double key wired wall switch (QBKG03LM)|online}"];
}

Also, the network map may not be fully accurate. At least one device shown as having no connection (stud_dwds_1) is working fine.

[Koenkk]

Optimized CC2530 firmware is now available for testing: https://github.com/Koenkk/Z-Stack-firmware/tree/optimize/coordinator/CC2530/bin

[kquinsland]

If you want me to create a new issue, i will be happy to. I thought i'd post here, first though.

I have decent number of Aqara devices; for some reason, they keep "falling off" the network with this firmware. I can't seem to identify a pattern or a way to reliably reproduce it, but here's the general problem that i've observed:

0. I'll have no issue pairing the device to the network. Home Assistant gets data and all is good
1. some amount of time elapses; Not more than 24 hours or so
2. the device stops functioning. In the case of the WKXG11LM buttons, a button press does nothing. when in debug logging mode, i see no incoming packets
3. remove the battery, and re-insert into the device and press the pairing button. I will then see a devLeaving message:

  zigbee2mqtt:debug 2018-12-25 19:18:53 Received zigbee message of type 'devLeaving' with data '"0x00158dDEADBEEFbc"'

4. i can then enable pairing mode and then re-pair the device without having to edit database.db or state.json or having to restart z2m.

I do not know if this is related to a similar issue i observed (but didn't document!) earlier... when using the network map endpoint, i would get a graph that had an orphaned node. That is, the end device is there, but it has no line connecting it to either my router or my coordinator.

If there's anything specific you'd like me to check, please do let me know!\

[Koenkk]

@kquinsland what is the linkquality of the devices? This can happen in case of a bad connection.

[mrfoxbit]

Hi all,

I am using this firmware (https://github.com/Koenkk/Z-Stack-firmware/tree/master/coordinator/CC2530/bin) for my CC2530 .

Is max number of devices with this firmware?

[Koenkk]

@hoangtuit this one (note master but dev branch) has a limit of 15: https://github.com/Koenkk/Z-Stack-firmware/tree/dev/coordinator/CC2530/bin

[orientalsniper]

Which one should we use? Dev or Master?

[Koenkk]

I recommend dev, it will (probably) be the next default firmware as it's more stable and has group support.

[mrfoxbit]

@Koenkk Tks you.
What is the max number of devices with dev firmware?

[Koenkk]

@hoangtuit 15 devices connected directly to the coordinator.

[kquinsland]

@Koenkk

Thanks for asking / getting back to me.

One of the devices is "attached" to the coordinator and has a link quality of 0. I've moved the coordinator closer, now and hope to see the link-quality change. Oddly enough, the device with a link-quality of 0 had been publishing data to the coordinator just days ago. The coordinator had not moved in the time that the end device went from publishing data to a link quality of 0.

Here's the real kicker, though: I have a device that is currently not connected at all (using the network map graph...) but the state.json has a link quality of "linkquality": 60,. This particular device has been reliably publishing data.

As of right now, my state.json has two devices with a link-quality of 0, but the most recent network graph has them both connected to the coordinator.

[Koenkk]

state.json is not updated when a device is removed, you have to clear this part manually. This is probably the last known value of that device.

[kquinsland]

@Koenkk, thanks. that's good to know.

[wimpie007]

@lolorc:
"I managed to upgrade without repairing all my devices."
How did you do this? want to upgrade firmware, but obviously don't want te repair devices...
(and i've seen that i even can't reset my 2 HUE lamps as i don't have the hue-bridge, or a hue-dimmer)

[lolorc]

@wimpie007
#282 (comment)