Slow memory leak

[tinder-marissanovak]

Observed behavior

My pods that are running the NATS Java client are consuming more and more memory. I'm not using JetStream. There are slow consumers 1-2 times a week but they don't seem correlated with the memory usage increases since they don't happen at the same time. There are no errors or warnings logged by the Java client or on the NATS server.

Expected behavior

Memory usage stays constant

Server and client version

Client version 2.17.6
Server version 1.x

Host environment

No response

Steps to reproduce

I'm using default Options to create the Connection, other than maxReconnects is 300 and maxMessagesInOutgoingQueue is default * 2

[MauriceVanVeen]

Thanks for the report. Could you share what features of NATS Core you are using, pub/sub and/or request/reply?

Could you perhaps also create a small code snippet that shows your general interactions with the library? (For example making a connection and what pub/sub/request/reply actions are being performed)

And just checking; I assume the Y-axis is memory used in GBs?

[tinder-marissanovak]

Thanks for your quick reply! I'm using pub/sub (not using request/reply).

The Y-axis is the percentage of allocated memory used.

This is how I'm making a connection:

// Initialize connection to NATS server
  private Connection initConnection() {
    Options options = new Options.Builder()
        .maxReconnects(MAX_RECONNECT_ATTEMPTS)
        .server(natsConfig.getUrl())
        .maxMessagesInOutgoingQueue(MAX_MESSAGES_IN_OUTGOING_QUEUE)
        .errorListener(new PubSubErrorListener())
        .connectionListener(initConnectionListener())
        .build();

    try {
      return Nats.connect(options);
    } catch (IOException | InterruptedException e) {
      throw new RuntimeException("Failed to initialize connection to NATS server", e);
    }
  }

And I'm publishing like this:

  public void publish(String topic, byte[] protoBinary) {
    connection.publish(topic, protoBinary);
  }

There's a subscriber in another deployment using the Go client and there are no memory issues there.

[MauriceVanVeen]

It doesn't seem to reproduce when just calling initConnection() and having connection.publish(topic, protoBinary) in a loop.

However, if a reconnect to the server was performed the thread count would keep rising and not go down.

This issue is fixed starting from version 2.17.7. I'm assuming it was the Timer task that was used for PINGs to the server not being cleaned up after a reconnect, which was fixed in this PR: #1128.

Could you upgrade to latest 2.20.0 version and check if the issue is resolved?
(or otherwise tryout 2.17.7, but would recommend latest)

[tinder-marissanovak]

Interesting, I'll try that. Thanks!

[scottf]

@tinder-marissanovak 2.20.1 has been released, which might have additional fixes for reconnect. Thank you for your feedback and patience.

[tinder-marissanovak]

Thank you! The memory usage after a few days seems more stable in 2.20.0, and if I see more issues I'll try 2.20.1.