InfluxDB OSS v2.7.10 unresponsive, eventually crashing with OOM

[dobrotaf]

Steps to reproduce:
List the minimal actions needed to reproduce the behaviour.

1. Workload with no deviation from long-term trends
2. No outside action (increased writes to buckets, read query with many datapoints in result executed etc.)

Expected behaviour:
No outage of Influx service seemingly without outside action

Actual behaviour:
Once every one or two weeks around 00:00 UTC most of write (Telegraf) and read (Grafana) requests ends with timeout error on the clients and memory consumption increases until the influxd process is reaped by oom killer. I'm suspecting some task around compaction, shard rotation or similar background task on the Telegraf bucket as the outages seems to have fixed period of appearance, traffic volume is not outside of long term trends that could cause increased usage of system resources and write / read requests to other buckets have normal (2xx) http status.

The Telegraf bucket i'm suspecting could cause the problem is reporting ~350G storage size (storage_shard_disk_size) and have "forever" retention policy, which means shard duration 7 days as i haven't configured custom shard duration.

Similar issue has been discussed in this thread #24406 I couldn't reopen the issue, but the problem seems very similar.

Environment info:

• Influx running in podman from image docker.io/library/influxdb:2.7 (amd64 2.7.10 version), no other container on the host is running
• Podman host is virtual machine with 16vcpu, 32G ram
• Custom environment variables:

INFLUXD_LOG_LEVEL=debug
INFLUXD_STORAGE_MAX_CONCURRENT_COMPACTIONS=2
INFLUXD_STORAGE_COMPACT_THROUGHPUT_BURST=10331648

Logs:
Heap images before the problem occurs, 10 minutes after and last minute before the oom killer reaped the process

00:01

00:10

00:26

Logs from Telegraf agent (write)

2025-06-16T00:00:18Z E! [outputs.influxdb_v2] When writing to [http://influxhost:8086/api/v2/write]: Post "http://influxhost:8086/api/v2/write?bucket=telegraf&org=custom_org": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2025-06-16T00:00:18Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
...
2025-06-16T00:28:03Z E! [outputs.influxdb_v2] When writing to [http://influxhost:8086/api/v2/write]: Post "http://influxhost:8086/api/v2/write?bucket=telegraf&org=custom_org": dial tcp influxhost:8086: connect: connection refused
2025-06-16T00:28:03Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)

Logs from Grafana (read)

2025-06-16T00:00:40.726475259Z logger=tsdb.influx_flux endpoint=queryData pluginId=influxdb dsName=influxdb-v2 dsUID=xxx uname=grafana_scheduler rule_uid=xxx org_id=1 t=2025-06-16T00:00:40.726475259Z level=warn msg="Flux query failed" err="Post \"http://influxhost:8086/api/v2/query?org=custom_org\": context deadline exceeded" query="from(bucket: \"telegraf\")..."
...
2025-06-16T00:28:06.297792863Z logger=tsdb.influx_flux endpoint=queryData pluginId=influxdb dsName=influxdb-v2 dsUID=xxx uname=grafana_scheduler rule_uid=xxx org_id=1 t=2025-06-16T00:28:06.297792863Z level=warn msg="Flux query failed" err="Post \"http://influxhost:8086/api/v2/query?org=custom_org\": dial tcp influxhost:8086: connect: connection refused" query="from(bucket: \"telegraf\")..."

Logs from Influx

ts=2025-06-16T00:00:13.147125Z lvl=info msg="Write failed" log_id=0wwmuKel000 service=storage-engine service=write shard=6443 error="engine: context canceled"
ts=2025-06-16T00:00:13.147342Z lvl=debug msg=Request log_id=0wwmuKel000 service=http method=POST host=influxhost:8086 path=/api/v2/write query="bucket=telegraf&org=custom_org" proto=HTTP/1.1 status_code=499 response_size=107 content_length=-1 referrer= remote=linuxserver:44892 user_agent=Telegraf authenticated_id=0b26d84447adf000 user_id=0ac7beb338afa000 took=10117.641ms error="internal error" error_code="internal error"
...
ts=2025-06-16T00:26:36.086156Z lvl=debug msg=Request log_id=0wwmuKel000 service=http method=POST host=influxhost:8086 path=/api/v2/write query="bucket=telegraf&org=custom_org" proto=HTTP/1.1 status_code=499 response_size=90 content_length=-1 referrer= remote=linuxhost:44006 user_agent=Telegraf authenticated_id=0b26d84447adf000 user_id=0ac7beb338afa000 took=10024.616ms error="internal error" error_code="internal error"
...
2025-06-16T00:26:36 systemd[1]: libpod-9d2217b610be3ef850f79a65a2d743487c9cf71d806c53c39320039fe05fd325.scope: A process of this unit has been killed by the OOM killer.

[philjb]

hi @dobrotaf - the heap profiles are helpful and more than most provide. Thank you. It appears that the write workload is growing in memory - the profiles show nodes handling incoming data with more and more data in memory. This suggests the DB is not writing to disk as fast as the data arrives. You said the write rate is the same so it must be something about getting to disk.

CPU profiles when memory usage grows from 8->20GB would be helpful to see what the DB is spending its time on. Are you seeing increased write request latency before the OOM kills?

[dobrotaf]

Are you seeing increased write request latency before the OOM kills?

Almost immediately after 02:00 all Telegraf agents started to report errors context deadline exceeded (Client.Timeout exceeded while awaiting headers) until new instance of Influx has been started after OOM ~02:30. Almost all of write traffic comes from Telegraf agents.

CPU profiles when memory usage grows from 8->20GB would be helpful to see what the DB is spending its time on.

Below is telemetry from Influx host time-wise around incident. I forgot to scroll legend on memory usage; red line is "used" memory.

• Heavy disk usage with high cpu iowait can be seen between 02:25 and 02:30 but i attribute it to swap usage.
• Low to non disk usage can be seen between 02:00 and 02:25
• It seems that "user" cpu load between 02:00 and 02:30 is lower than before the outage
• Incoming network volume on enp7s0 is increased between 02:00 and 02:30 but the increase started after the issue 02:00 so to me it seems like consequence rather than cause

From the look on the telemetry and your answer @philjb i get a feeling that around 02:00 something went wrong with Influx service writing data to disk, clients (mostly Telegraf services) started to time out during POST requests to api/v2/write, but Influx kept the incoming POST data in memory until it couldn't.

This suggests the DB is not writing to disk as fast as the data arrives. You said the write rate is the same so it must be something about getting to disk.

Maybe i haven't expressed myself correctly. What i meant was that there is no reason why there should be elevated traffic / write rate coming from Telegraf agents. Almost all of the write traffic to Influx we have from Telegraf agents.

There is elevated incoming traffic during the outage but combined with the fact that disk usage went down, i think it might be Grafana alerts with Influx datasource trying to recover from error state or maybe data in write requests from Telegraf agents due to buffering datapoints while getting error context deadline exceeded (Client.Timeout exceeded while awaiting headers). Around 300% increase of incoming traffic is strange nevertheless, for now i can't figure it out, but i think this is dead end road for determining root cause.

The only line in Influx log indicating something went wrong while writing to disk is this

ts=2025-06-16T00:00:13.095461Z lvl=info msg="Reindexing WAL data" log_id=0wwmuKel000 service=storage-engine engine=tsm1 db_shard_id=6443
...
ts=2025-06-16T00:00:13.147125Z lvl=info msg="Write failed" log_id=0wwmuKel000 service=storage-engine service=write shard=6443 error="engine: context canceled"

After these lines only errors come from service=http with status_code=499 took=10145.564ms error="internal error" error_code="internal error".

The shard id 6443 is also from different bucket than the one Telegraf agents are writing to. The bucket with shard id 6443 has only 90kB on fs.

[philjb]

You have an excellent monitoring setup - nice.

I reviewed the charts and your description quickly. I'll let it work in the back of my mind through the day.

A couple initial thoughts:

• do you have a chart of telegraf request success/error rates and/or request latencies? If you have a proxy, it might record this information. Telegraf itself has some metrics available: see the internal plugin, but of course those need to be written into influxdb too.
• You mentioned that the network volume increase is likely an effect and not a cause: How would an issue with influxdb cause more network traffic to it? I can imagine that happening if your telegraf agents are retrying the failed requests. Or i could imagine the increase in traffic being the cause of the issue. You can adjust the telegraf timeout (default is 5s I believe) - a larger number might help telegraf get through an influxdb issue. You can imagine that telegraf reaches its 5s limit, cancels its request while influxd is still working on it. Telegraf than tries again and now influxd has even more write traffic load to process. Your network in traffic has tripled, clearly making the issue worse and explaining the heap usage charts - more write traffic, more data in memory to write to disk. It is interesting that you have plenty of cpu capacity - ~10% baseline, which goes down slightly when the problem occurs.
• It is beginning to look like a deadlock or stall someplace (low cpu usage, doesn't self recover, disk write rate unchanged, cleared by process restart). Version 2.7.12 addressed a locking issue. Upgrading would be good - at least it would eliminate a variable.
• I don't recommend using disk swap with influxdb - it causes the kernel to increase disk io to page out memory and therefore takes disk bandwidth away from influxdb. Writing to disk (and reading) is generally the bottleneck so swap makes it worse under load. If it is needing to swap, influxd will almost certainly not recover on its own as that "memory" is now slow. I would recommend you let influxd oom earlier instead of giving it swap.
• I'm surprised there's not more interesting logging from around UTC 00:00 at debug level - nothing about shard creation?
• While you likely don't want to write to a new bucket, you could create a new bucket with a longer shard duration and configure telegraf agents to write to the new bucket (probably don't want to do this until the issue is resolved).
• You can try collecting lock/mutex profiles before and after UTC 00:00 whenever the next issue is expected (this weekend?).
• Lastly, I would change only one variable at a time. I would probably upgrade influx first.

[davidby-influx]

This may be fixed in 2.7.12. Please upgrade.

feat: TagValueIterator holds RLock for too long (#26369)