Memory leak in Sensor

[grograxa]

Describe the bug
There appears to be a memory leak in the Sensor component when triggering Argo Workflows. In a HA setup the leader memory usage continually increases until it is OOM killed. The standby memory usage remains constant.

To Reproduce
Steps to reproduce the behavior:

1. Deploy a Sensor that triggers an Argo Workflow. Example triggers definition:

triggers:
  - template:
      argoWorkflow:
        operation: submit
        parameters:
          - dest: spec.arguments.parameters.#(name=="message").value
            src:
              dataKey: body
              dependencyName: example
        source:
          resource:
            apiVersion: argoproj.io/v1alpha1
            kind: Workflow
            metadata:
              generateName: example-workflow-
            spec:
              arguments:
                parameters:
                  - name: message
                    value: Will be replaced with SQS payload
                  - name: env
                    value: dev
              workflowTemplateRef:
                name: example
      name: example

3. Send event that triggers the sensor
   Expected behavior
   Sensor memory usage is consistent and does not grow over time.

Screenshots

Environment (please complete the following information):

• Kubernetes: 1.32
• Argo: 3.6.7
• Argo Events: 1.9.6

Additional context
Add any other context about the problem here.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[tico24]

After some extensive testing, we have determined that this is caused by this: nats-io/nats.go#1856

main already has the patch on it, so a release is required to roll this fix out for Argo Events.

Hats off to @djanjic for finding the root cause.

[djanjic]

Here's a heap dump (in png) over time, where you can see nats.go StatusChanged method memory usage increase over time:

[tico24]

Please re-test in v1.9.7 and let us know.

[grograxa]

Thanks for fixing the issue.

Tested this in multiple environments and are no longer seeing the memory leak 😊