Omnichannel widget fails to show new messages

[preeder]

Description:

In poor network conditions, the Omnichannel widget can get in a state where it does not show new messages, though it is able to send messages.

Steps to reproduce:

This was a difficult issue to reproduce and one we've seen sporadically in many versions of Rocket.Chat. In essence, we created two isolated networks, each NAT'd to the Internet. On one network was the client and on the other was Rocket.Chat. A software router was installed on both networks, providing connectivity between the two networks. The software router was given two IP addresses on each network, thereby allowing us to interrupt traffic in one direction only by removing the routing IP address from the router on a network, thus replicating things we would expect to see in the wild, such as with a mobile device on the edge of coverage.

We used the Hyper-V hypervisor on Windows 11. This could also be done with other virtualization technologies, though the specifics of the directions would be different.

SETUP:

1. In Hyper-V Manager, add two Internal class VLANs via the Virtual Switch Manager.
2. In the host machine's Network Connections, add the addresses 192.168.100.1/24 and 192.168.101.1/24 to the two networks, respectively.
3. Add the NATs by opening an Administrator Powershell terminal and executing the following:
   a. New-NetNat -Name Vm1NatNetwork -InternalIPInterfaceAddressPrefix 192.168.100.0/24
   b. New-NetNat -Name Vm2NatNetwork -InternalIPInterfaceAddressPrefix 192.168.101.0/24
4. Create a VM on the first network to host Rocket.Chat. Give it a static IP address of 192.168.100.10 with 192.168.100.1 as the default gateway and a low metric route for 192.168.101.0/24 through 192.168.100.254. Install Rocket.Chat however desired (e.g. docker compose). Configure Rocket.Chat.
5. Create a VM on the second network for the client visitor. Give it a static IP address of 192.168.101.10 with 192.168.101.1 as the default gateway and a low metric route for 192.168.100.0/24 through 192.168.101.255. Install the graphic OS of your choice with any mainstream browser.
6. Create VM with a virtual network adapter on both networks. Install the software firewall of your choice (e.g. VyOS). Give it IP addresses of 192.168.100.253 and 192.168.100.254 on the first network and 192.168.101.253 and 192.168.101.254 on the second network. Enable routing between the networks.
7. Add IIS to the Windows host. Create a test page named testchat.html on the Windows host with the Rocket.Chat widget snippet. Place this in the IIS root directory (typically C:\inetpub\wwwroot)
8. In the Windows host, log in to Rocket.Chat and set yourself available for Omnichannel chats.
9. In the visitor VM, open a browser to http://192.168.101.1/testchat.html. You should see the test page and be able to start a chat via the Rocket.Chat widget.
10. Ensure that communications are normal.

PROBLEM REPRODUCTION

1. Open the developer tools on the visitor client and go to the network tab
2. Refresh the test page and select the websocket connection to Rocket.Chat.
3. Start a test chat. Send messages in both directions.
4. In the router, remove the IP address 192.168.101.254. This will make network traffic from the client to Rocket.Chat be dropped but traffic from Rocket.Chat to the client will be sucessful.
5. Watch the traffic on the websocket to ensure that at least one ping is received by the client. NOTE: the response from the client will not successfully get to Rocket.Chat and Rocket.Chat will give up on the websocket connection. However, the client will still think the websocket connection is viable.
6. In the router, add back the IP address 192.168.101.254. Traffic will now be restored in both directions.
7. Send a message from the client. Notice that it is received by the Rocket.Chat user but is not shown in the visitor client.
8. Send a message from the Rocket.Chat user. Notice that it is also not shown to the visitor.

Expected behavior:

The Omnichannel client widget seems to lack a good mechanism to identify a defunct websocket connection. This test gets the client in a state where it believes to have a good connection, but that connection has been dropped by Rocket.Chat. We would expect the client to have some kind of mechanism to ensure this does not happen, either using standard traffic as a keepalive or perhaps employing bidirectional pings.

Actual behavior:

The client does not recognize that the websocket connection is effectively dead and its messaging subscriptions no longer working. This causes the client to not show any new messages in the widget, though new messages can be sent from the widget.

Server Setup Information:

• Version of Rocket.Chat Server: 7.5.1 (test), 6.13.0 (production)
• License Type: Starter (test), Enterprise (production)
• Number of Users: 2 (test), 123 (production)
• Operating System: Ubuntu 24.04 (test), Ubuntu 22.04 (production)
• Deployment Method: docker compose (test), custom pipeline (production)
• Number of Running Instances: 1 (test), 6 (production)
• DB Replicaset Oplog:
• NodeJS Version:
• MongoDB Version: 7.0.4 (production)

Client Setup Information

• Desktop App or Browser Version: Firefox 136.0.2
• Operating System: Endeavour OS Mercury Neo 2025.03.19

Additional context

Relevant logs: