ath79-generic (e.g. WR1043ND v4) - WLAN Mesh broken when upgrading to v2022.1.x because of timing issue in boot process

[rotanid]

When upgrading from Gluon v2021.1.x to v2022.1.x wlan mesh doesn't work anymore on a TP-Link TL-WR1043ND v4.
The 802.11s mesh interface is shown in "iwinfo" but not on the status page or "batctl if"
The upgrade process was tested from latest v2021.1.x branch (fresh install) to Gluon v2022.1, v2022.1.1 and v2022.1.2.

The problem does not appear when flashing with "forget settings" and reconfiguring the v2022.1.x firmware from scratch.
The problem does not appear with WR1043ND v2 or v3:

A problem with the migration from ar71xx-generic to ath79-generic may have happened, although @AiyionPrime stated in #2431 that everything was working fine - so maybe the issue was introduced later than v2022.1(.0) ?

[AiyionPrime]

This is a v4, that's been active and updated for the last four years: https://hannover.freifunk.net/karte/#/en/map/8416f99bd2d0

With vH31 it is currently running gluon v2022.1.1 (ab1fb05).

Meshing does work and can be seen without problems on it's statuspage:
http://[2001:678:978:213:8616:f9ff:fe9b:d2d0]/cgi-bin/status

Though I certainly might have overlooked something, I think the migration was fine.
Our sample size is fairly limited though; most of the routers (75) are still on vH25 (ar71xx)
and only three or four are ath79.

[rotanid]

tom reported on IRC that in darmstadt there was a similar issue with a TL-WR1043N v5:
https://forum.darmstadt.freifunk.net/t/meshausfall-nach-2-6-0-in-dieburger-innenstadt/944

[rotanid]

i just tested the tag v2022.1 and the issue also happens with the initial release of this branch.

@AiyionPrime can you be sure, the linked device was never reconfigured manually? one probably cannot see this from the available data.

i'm currently trying to get my hands on a TL-WR1043N v5 to check if i can reproduce it.

it would also help to check it on a second TL-WR1043ND v4 - does anyone have it laying around and can test?

[AiyionPrime]

No, I can not. We can send the owner an email and ask though, if that helps.

[Djfe]

Just some ideas. If they don't apply, then pls bear with me :)
@rotanid Have you taken a look at what logread returns? (Maybe there are errors in there?)
Once anyone gets their hands on an affected device: Is this reproducible on a fresh install? (Install 2021, set a few things in config mode, boot once, then update to 2022)

[rotanid]

Have you taken a look at what logread returns? (Maybe there are errors in there?)

serious question? well ok: no relevant info there as far as i can see.

Once anyone gets their hands on an affected device: Is this reproducible on a fresh install? (Install 2021, set a few things in config mode, boot once, then update to 2022)

that i have already done and also written in my bug report, i added "(fresh install)" now as it seems it hasn't been clear enough.

[Djfe]

ok my first suggestions were very basic. 😑😅

You probably figured this out, but maybe you can replicate what they did in Darmstadt and compare configs before and after saving config mode:
install 2021 fresh
config mode 2021, save
update 2021 to 2022
get relevant info (uci show, /etc/config/wireless, ...)
config mode 2022, save
get relevant info (uci show, /etc/config/wireless, ...) again
compare the two

[Djfe]

The only commit, that happened after adding the device would be
openwrt/openwrt@e826b64

Looking back at the old definition:
https://github.com/openwrt/openwrt/blob/openwrt-19.07/target/linux/ar71xx/files/arch/mips/ath79/mach-tl-wr1043nd-v4.c#L62-L67
absolute MAC offsets on flash are v4 0x1ff50008 vs v5 0x1ff00008
It's obvious that mac's are placed in the same partition but the partition has a different offset.
In the commit the definition is moved from the dtsi down to the correct dts files.
This should've improved from upgrading the old ar71xx target, but maybe the commit broke something else.
They also moved the calibration data for wifi along.

Maybe I found the issue?

#define TL_WR1043_V4_EEPROM_ADDR		0x1fff0000
#define TL_WR1043_V4_WMAC_CALDATA_OFFSET	0x1000

Old address of calibration data 0x1fff1000

New definition based on art partition, but art != eeprom(?)

art: partition@ff0000
mtd-cal-data = <&art 0x1000>;

New address of calibration data 0x1ff01000 (I added 0x1 manually, I assume it is calculated this way)

Possible fix?
mtd-cal-data = <&art 0xf1000>;

[rotanid]

the only difference in config i could find is as follows:

> config interface 'mesh_radio0'
> 	option proto 'gluon_mesh'
> 

looking at the code, there may be an issue with "get_wlan_mac" during the upgrade from 2021.1.x and therefore the upgrade script returns before setting up the above section.
https://github.com/freifunk-gluon/gluon/blob/master/package/gluon-core/luasrc/lib/gluon/upgrade/200-wireless#L129

this correlates with what @Djfe found in OpenWrt.

anyone else can follow this argumentation? @blocktrron @NeoRaider @adschm ?

[Djfe]

I feel like we should print an error when there is no wmac to be found (Lines 130-131).
This error could happen again for other devices and can be catched best by observing some form of log.
Since logread is probably no option for init scripts, could we create a script that writes a log to flash that is only overwritten the next time, init scripts are run? (yes, there are devices with small flash storage, so we either have to keep the log small and pipe it through gzip on write after init complete, or we disable this for tiny style targets)

Such a log could be useful for adding new devices, too.
It also allows catching mistakes in new code/dts files regarding the initialization.
It would be useful alone for all silent returns in the lua file @rotanid linked above.

[rotanid]

i bought a TL-WR1043N v5 and tested it.
this device has exactly the same problem :-(
it was therefore erroneously tested as "working" in #2483