SNOW-2159000 TelemetryThreadPool not scaling beyond 1 thread

[matthagenbuch]

TelemetryThreadPool not scaling beyond 1 thread

Please answer these questions before submitting your issue.
In order to accurately debug the issue this information is required. Thanks!

1. What version of JDBC driver are you using?

   3.23.1

2. What operating system and processor architecture are you using?

   macOS 14.5.0 (darwin 24.5.0) - but this affects all platforms

3. What version of Java are you using?

   temurin 24

4. What did you do?

   Issue Description:
   The TelemetryThreadPool class uses a ThreadPoolExecutor with an unbounded LinkedBlockingQueue and expects it to scale up to the maximum pool size (10 threads) when there are queued telemetry tasks. However, due to how ThreadPoolExecutor works with unbounded queues, it never creates more than the core pool size threads.

   Current Configuration:

   uploader = new ThreadPoolExecutor(
       0, // core size
       10, // max size - EFFECTIVELY IGNORED
       1, // keep alive time
       TimeUnit.SECONDS,
       new LinkedBlockingQueue<>() // unbounded queue - CAUSES THE ISSUE
   );

   Root Cause:
   According to ThreadPoolExecutor documentation and confirmed by this Stack Overflow answer:

   "Using an unbounded queue (for example a LinkedBlockingQueue without a predefined capacity) will cause new tasks to wait in the queue when all corePoolSize threads are busy. Thus, no more than corePoolSize threads will ever be created. (And the value of the maximumPoolSize therefore doesn't have any effect.)"

   Reproduction:

   • Configure your test client to connect to a Snowflake instance in a region with higher latency (this eases reproduction of the issue, but the issue also impacts lower latency connections)
   • Enable debug logging for net.snowflake.client.jdbc.RestRequest
   • Initialize the driver and execute queries with increasing concurrency, ramping up over at least a few minutes
   • Observe from the debug logs that calls to /telemetry/send are all executed from 1 thread
   • Verify from a heap dump that the LinkedBlockingQueue is not empty

   Note that there is also a race case in the thread pool executor logic for spawning new threads when the pool is empty. If a high volume of new telemetry batch requests are generated concurrently while the pool is empty, the executor will spawn multiple threads randomly.

5. What did you expect to see?

   Expected Behavior:
   When telemetry tasks are queued and not being processed quickly enough, the thread pool should create additional threads up to the maximum pool size (10) to process the backlog in parallel.

   Actual Behavior:

   • Thread pool rarely scales beyond core pool size due to unbounded queue
   • Telemetry tasks queue up instead of being processed in parallel
   • Inconsistent thread count (1-10) due to race conditions and special handling of corePoolSize=0
   • Query performance degradation during high concurrency load

6. Can you set logging to DEBUG and collect the logs?

   I can't share our complete debug log but I can extract the relevant logs as described above if you think that would be helpful.

[sfc-gh-dszmolka]

hi - thanks for reporting this issue, we'll take a look

[sfc-gh-wfateem]

@matthagenbuch can you share more information on how you were able to link this issue with the following problem you stated?

Query performance degradation during high concurrency load

Can you first define what you mean by query performance here?

Are queries taking longer to run on Snowflake at the backend, or is this slowness something on the application's side? I'm assuming it's the latter.

Even if you have a bottleneck with processing telemetry data, the impact is basically a delay in sending that over to Snowflake. That shouldn't slow anything down that's happening in the other threads responsible for sending queries to Snowflake and for downloading results from the storage service.

Do you happen to have Java thread dumps from the JVM process during the slowdown you experienced during high load?

[matthagenbuch]

Hi @sfc-gh-wfateem,

Can you first define what you mean by query performance here?

We are using the JDBC driver inside the Trino snowflake connector. I've found that Trino query latency increases linearly after a certain number of concurrently executing queries is reached. This concurrency threshold differs depending on the latency from the Trino cluster to the Snowflake instance.

Are queries taking longer to run on Snowflake at the backend, or is this slowness something on the application's side?

The slowness is coming from Trino, which closes its JDBC connection before returning query results. Closing the connection triggers a synchronous telemetry client batch send, which is where the thread pool becomes a bottleneck.

I'll work on getting a thread dump to share with you.

[sfc-gh-wfateem]

Thanks @matthagenbuch for the information. Yes, I think a thread dump would be helpful. Ideally, a few of them with some interval between each capture to see how the threads are progressing (or not).

Are those threads in Trino responsible for running the concurrent queries, fetching connections from a connection pool, or do they create a new connection each time?

[matthagenbuch]

threaddump-20250618-160205.txt
threaddump-20250618-160053.txt
threaddump-20250618-155908.txt

These were taken a minute or two apart. You'll find the only thread executing TelemetryClient work is pool-31-thread-3

[matthagenbuch]

Are those threads in Trino responsible for running the concurrent queries, fetching connections from a connection pool, or do they create a new connection each time?

Our understanding is that Trino does not utilize connection pooling, and creates/closes a new connection for each query. A parallel effort is underway to investigate connection pooling for Trino in general.

[sfc-gh-dszmolka]

this should be fixed with #2231 and will be included in the next* release

* : 'next' in this particular case is 3.25.0 where the release is already in progress, we just hit some problems, so it's not yet on Maven. we're working on it - the release, that is. The fix is already in main, so can be used even right now if the driver is built from source.