25.2.0 memory leak

[jonanon20]

We have noticed since the upgrade from 24.12.0 to 25.2.0 a significant increase in memory usage that has caused several jvm OOM errors. Our web3signers are responsible for 10s of thousands of keys and this issue appeared after onboarding a new series of ~200keys leading to the OOM crash and missed attestations.

You can see from the graph that with version 24.12.0 we didn’t have this memory issue, as soon as we deployed 25.2.0 on 19/02 then memory usage spiked considerably. As a temporary measure we have set our -XX:MaxRAMPercentage=50 and given our nodes more memory, we generally see memory usage plateau but the plateau will only last a day or 2 before it climbs again. This gradual climb continues until we are alerted and have to restart. Originally we were at -XX:MaxRAMPercentage=25 without any issues or memory spikes. Memory usage for the web3signer up until 25.2.0 has always been consistently low.

When we onboarded 200 new keys and we had missed attestations we noticed a considerable jvm garbage collection time as shown on the graph below across 2 of our 3 signers (jvm_gc_collection_seconds).

despite the long gc times we didnt see any reduction in memory for 2 of our signers during this period but this could be the result of memory pressure at the time.

We have noticed a worrying increase in memory and a gradual climb over several days. Where before 25.2.0 we were seeing a usage of ~1GB with no issues we now gradually climb beyond 6GB after 2 or 3 days.

[usmansaleem]

@jonanon20 do you use the /reload API call or restart web3signer instances to onboard new keys?

[jonanon20]

Yes that right, we use the /reload API

[usmansaleem]

@jonanon20 Can you provide some more details for us to debug, such as:

• How Web3Signer is deployed, via docker image or via binary
• How the keys are loaded, using keystore files or via AWS/Azure?
• What is the total memory allocated to Web3Signer instance? Also, system memory/CPU.

Thanks/ Uzi

[jonanon20]

Hello, we deploy to kubernetes using the docker image consensys/web3signer:25.2.0. We have a sidecar that writes the keys to disk before hitting the /reload endpoint. We set the env var WEB3SIGNER_KEY_STORE_PATH.

We are running the pod on an m5.2xlarge AWS instance. 8vCPU and 32GiB.
The web3signer container has the following resource requests:

resources:
  limits:
    memory: 12Gi
  requests:
    cpu: "4"
    memory: 9Gi

There is a pod anti affinity so multiple web3signers cannot be loaded onto the same node. There is no memory pressure on the node itself, only the pod after a couple of days. We still get jvm oom errors even after trippling our requests/limits, it just takes a few more days to fail.

We launch the container with the arg eth2 and we use the following WEB3SIGNER_OPTS:

- name: WEB3SIGNER_OPTS
  value: -Djava.util.concurrent.ForkJoinPool.common.parallelism=4 -XX:+DisableAttachMechanism -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dev/null -XX:MaxRAMPercentage=50.0

[usmansaleem]

@jonanon20 If possible, can you DM me on Discord (usmans.info) or on Consensys discord server Web3Signer channel (https://discord.gg/consensys) or on Telegram (@usmans_info). There are few questions about keystore parameters that I would like to ask directly. In addition, this will allow me to hook us up with our performance engineers as well. BR, Uzi.

[jonanon20]

Hi @usmansaleem , I have sent a couple of messages to you over discord, I wonder if you have had the chance to see them? We're still getting oom issues with this.

[aimxhaisse]

We also see a similar leak with /reload, I've shared details to Uzi on Disccord.

[ahamlat]

Hi @jonanon20,
Could you generate a heap dump, load it with Eclipse memory analyzer tool (https://eclipse.dev/mat/download/), and share the histogram view like below.

[ahamlat]

Any update on this issue ?

[usmansaleem]

@jonanon20 @aimxhaisse Can you try updating to latest W3S release and see if you still seeing OOM/increase memory issues: https://github.com/Consensys/web3signer/releases/tag/25.6.0 . We may still need a heapdump screenshot as mentioned by Ameziane above to analyse further if you still face memory issue.