Dozens of channels stuck in TEARDOWN state, and continue decoding data

[Nokoa]

sdrtrunk Version
Fork of master-branch

Describe the bug
When running about 17 P25 sites with phase 2 voice, some channels go into TEARDOWN state and remain in that state until SDRTrunk is restarted. Channels quickly accumulate until SDRTrunk eventually runs out of memory. These channels remain present even after turning off every active channel in the playlist. It appears that SDRTrunk is actually processing these channels as I can see occasionally the radio ID for the slot appears and changes as if it's processing incoming calls on the frequency. It only stops processing when I disable the Tuner, but the TEARDOWN channels still remain.

Some of the TEARDOWN channels are of the same site and the same frequency.

This particular computer is not the fastest. If I load less sites, it doesn't happen, or not observed as quickly.

To Reproduce
Steps to reproduce the behavior:
Start a lot p25 sites from the same system.

Expected behavior
As channels go inactive they should be reallocated to the channel pool and stop processing.

• OS: Ubuntu 22.04 LTS
• CPU Cores: 4
• RAM: 16GB

[DSheirer]

Is this new behavior that you're now seeing with recent changes?

[Nokoa]

I don't believe that this is related to any new change. I just started using this older laptop, where it perhaps maxes out the CPU more. The performance of SDRTrunk looks and sounds good. It's just causing channels to get stuck on TEARDOWN.

[Nokoa]

I tested this some more. I can run 1 or 20 P25 trunked sites all at once. I ran 20 fine for more than a day, with no channels stuck in teardown.

However once I turn on in addition to the P25 Trunked site a DMR Cap+ with 2 frequencies which uses a different tuner, TEARDOWN channels from the P25 channels would start to accumulate.

[DSheirer]

Copy, It's going to be a little bit until I can dig into this some more.

[DSheirer]

@Nokoa

I just added a diagnostic report generation feature and merged it into master branch. It should be included in the nightly release, once GitLab builds the new release ... it's been running a little bit slow lately.

https://github.com/DSheirer/sdrtrunk/releases/tag/nightly

Can you please run the application to the point where you're starting to see channels pileup in the TEARDOWN state and then invoke the diagnostic report and either send to me or attach here please?

File >> Reports >> Processing Diagnostic Report

This should indicate if there are any threading deadlocks or channels that have been orphaned or that are in a bad state. I'm still not sure exactly what's causing these channels to stick around in the TEARDOWN state ... they should be automatically cleaned up. Hopefully the report will shed some light on the issue.

[Nokoa]

20240205_045512_sdrtrunk_processing_diagnostic_report.log
In this case there are 2 channels stuck on teardown. However when I issued the diagnostic report SDRtrunk had already ran out of memory.

I usually upload to RDIO-Scanner, I wasn't in this instance. I will run it again, and report back.

[Nokoa]

Here is another diagnostic report with several Teardown channels that accumulated in a 10 minutes period.

20240205_052830_sdrtrunk_processing_diagnostic_report.log

[DSheirer]

Thanks. That's exactly what I needed you to capture. The first report shows lots of blocked threads and something weird going on between the P25 decoder and the DMR traffic channel manager. The blocked threads would definitely account for why it's running out of memory on you.

[DSheirer]

Discovered threading dead-lock issue:

TunerChannelSource.stop() method was allowing the thread pool to interruptably stop the thread pool executor while the channel still held a lock in the traffic channel manager causing the thread to die while still holding the lock. This caused other channel threaded runnables to eventually block on the same lock and get stuck in the TEARDOWN state. Changed the Dispatcher.stop() to no longer perform runnable stop interruptibly and allow the stop to complete naturally. This may have some unintended side-effects as residual sample processing from this thread may cause additional messages to be decoded and acted upon. However, I updated the decoder state module with a new flag that gets set on shutodwn that should prevent any residually decoded messages to be thrown away and not enter the decoder state or the channel state.

[DSheirer]

Discovered second issue:

The channel processing manager distributes channel start/stop notifications to all channel processing chains so that the traffic channel managers can monitor startup/shutdown of their child traffic channels. The DMR traffic channel manager was not checking if the notification was for one of its child traffic channels and was responding to channel shutdown notifications for P25 channels and since the DMRTrafficChannelManager's lock was owned/orphaned (ie deadlocked), this was causing the pileup in blocked threads for P25 channels.

Updated DMR traffic channel manager to ignore all channel events except for those from child traffic channels.

Note: this issue would only manifest if the user is running an combination of P25 & DMR decoders as the same time.