AHI/AHRS/Angle mode failure after about 1 to 3 minutes

[OlivierC-FR]

Follow up on : #10943

Since I posted in this previous BR, I replaced the FC with the same make and model, and replaced the GPS, didn't change anything. Both FC worked fine on other planes, running under INAV 7.1, the chances that both these FCs are somehow faulty at the same time after being moved out of a plane is literally zero, I'd say.

Current Behavior

After a delay between 60 to 180 seconds flying in angle mode, the horizon goes bad, and within few seconds the plane is out of control and I have to switch to acro or manual mode to recover. I then have to fly long enough for the horizon to come back somehow, but it's never for very long, it stays more or less unstable until the end of the flight.

INAV v8.0.1, FC AtomRC F405 Navi, GPS 15+ satellites, plane is a forward swept 3D-printed flying wing of my own design, similar to say a Dart250 in size and performance.

In this flight the issue happens at 2:37 :

CLI Dump : INAV_8.0.1_DUMP.txt
CLI Diff : INAV_8.0.1_DIFF.txt
DVR : https://drive.google.com/file/d/1VXAqBhBZmMHfllpzaNu92wNJKTWBOjL1/view?usp=sharing
Raw log : LOG00062.TXT
Decoded log : LOG00062.01.csv
GPS trace : LOG00062.01.gps.zip
GPS :

Steps to Reproduce

I just have to fly the plane, it always happens, 100% so far over a dozen of flights, approximately 1 to 2.5 minutes after launch. I now have a sizable collection of about 10 DVRs of this happening. I'm not 100% ruling out vibrations, BUT, on a 3D printed plane these would be super easy to notice, and it does not appear to have any. Motor is clean and props are new.

For context, another DVR, trimmed down to the event : https://drive.google.com/file/d/1tzwkwAA5huQha9ohewhLkvsL319_37jk/view?usp=sharing

I have other similar planes running INAV 8.0.1 with identical settings (minor the hardware specific settings of course) and they do not do this ((Dolphin Pro with AtomRC F40 Navi Deluxe, Dolphin with Matek F405 Wing)

Expected behavior

Plane not diving to the ground to its mechanical death.

[OlivierC-FR]

Side note, on the particular flight I choose as sample there was a LOT of wind, and it's a small plane so it was shaken quite bad, but the issue is the same on a calm day. But now I'm wondering if the stabilization being hard at work was the cause why it seemed to trigger later than all previous flights ?

Used to be at 60 to 100 seconds the previous days, and for this flight in really rough conditions it was at 165 secs.
Also I enabled the glonass option so the sat count was higher than before, from 10-15 to 15-20.

[breadoven]

Pretty sure this is caused by problems with the Z position estimations. A quick look at the DVRs shows numerous examples of position and Z velocities that contradict each other which I assume confuses the AHRS. Would need to check the logs for abrupt large changes in altitude/vertical speed that are likely responsible for the AHI suddenly losing alignment.

I'm thinking possibly a problem with Nav acc values given I noticed unrealistic values of up to 50g in the logs from #10943. The 50g didn't didn't remotely compare with the x. y. z g values from the accelerometer. Maybe something to do with the Nac acc gain calculations, something that's been changed/fixed in #10818 ?

[breadoven]

Looking at the log above I don't see the abrupt changes in estimated altitude that were in the logs in #10943. The one thing that is obvious though is Acc Z is pretty noisy, flip flopping 0 to 2g constantly. Acc X and Y look OK in comparison. Unfortunately Nav Acc values weren't logged which makes it difficult to know exactly how acceleration is affecting the estimated positions.

[OlivierC-FR]

I'll make another flight with the accelerometers logged, I forgot to enable this indeed. Maybe this evening, or tomorrow.

[OlivierC-FR]

Ok so I harvested new datas !

Before that I change few settings :

Flight 81 :

The issue did NOT happen, to my surprise, first time in like 12 flights.
But it was a short flight, 4 minutes (I'm vetting some old batteries) :

DVR : https://drive.google.com/file/d/1HbUYruHeon8ZGhn7iA23-FqyAQeo8EHK/view?usp=sharing
Raw log : LOG00081.TXT
Decoded log : LOG00081.01.zip

Flight 82 :

... so for the next I fitted a bigger lipo, and it happened, at 0:52, 2:32, 5:00, 5:50, 6:50

DVR : https://drive.google.com/file/d/1EaPJxyA8QU5IrdMx_AtdxGA_kyxDdXa8/view?usp=sharing
Raw log : LOG00082.TXT
Decoded log : LOG00082.01.zip

[breadoven]

Looking at LOG00082 the problems seem to occur when estimated Vel Z and Pos Z start misbehaving possibly because Vel Z is noticeably higher than the GPS and Baro rates. e.g. around time 05.00 Vel Z looks overestimated by a facor of 2 and Vel Z and Pos Z contradict each other, i.e. Vel Z is -ve whilst Pos Z is increasing.

I misread the Log units for Nav Acc values, they're actually OK other than showing an excessive bias sometimes which would affect estimated Vel Z and Pos Z.

It might be useful to set debug_mode to VIBE and to check what posEstimator.imu.accWeightFactor is doing (VIBE debug 4) although looking at the log accVibe levels aren't significant so accWeightFactor probably remains at a value of 1 and therefore isn't affecting the estimate corrections. If that's the case problems with Nav acc values will just be caused by dubious posEstimator.imu.accelBias values being applied although it's hard to be sure given these values aren't logged. However, there are 3 unused VIBE debug fields so it would make sense to add the Nav accelBias x, y, z values here.