SNOW-340077: UnsupportedOperationException sun.misc.Unsafe or java.nio.DirectByteBuffer when running on JDK16

[plavreshin]

Hi!

Snowflake-jdbc version: 3.13.2
JDK version: openjdk:16-slim

I am observing following issue while trying to fetch data from Snowflake used jdbc/JDK versions above:

Apr 09, 2021 12:35:31 PM net.snowflake.client.jdbc.SnowflakeChunkDownloader getNextChunkToConsume
SEVERE: downloader encountered error: Max retry reached for the download of #chunk0 (Total chunks: 5) retry=10, error=net.snowflake.client.jdbc.SnowflakeSQLLoggedException: JDBC driver internal error: Exception: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available.
at net.snowflake.client.jdbc.SnowflakeChunkDownloader$2.downloadAndParseChunk(SnowflakeChunkDownloader.java:895)
at net.snowflake.client.jdbc.SnowflakeChunkDownloader$2.call(SnowflakeChunkDownloader.java:950)
at net.snowflake.client.jdbc.SnowflakeChunkDownloader$2.call(SnowflakeChunkDownloader.java:766)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:831)
Caused by: java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
at net.snowflake.client.jdbc.internal.io.netty.util.internal.PlatformDependent.directBuffer(PlatformDependent.java:399)
at net.snowflake.client.jdbc.internal.io.netty.buffer.NettyArrowBuf.getDirectBuffer(NettyArrowBuf.java:243)
at net.snowflake.client.jdbc.internal.io.netty.buffer.NettyArrowBuf.nioBuffer(NettyArrowBuf.java:233)
at net.snowflake.client.jdbc.internal.io.netty.buffer.ArrowBuf.nioBuffer(ArrowBuf.java:247)
at net.snowflake.client.jdbc.internal.apache.arrow.vector.ipc.ReadChannel.readFully(ReadChannel.java:81)
at net.snowflake.client.jdbc.internal.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:696)
at net.snowflake.client.jdbc.internal.apache.arrow.vector.ipc.message.MessageChannelReader.readNext(MessageChannelReader.java:68)
at net.snowflake.client.jdbc.internal.apache.arrow.vector.ipc.ArrowStreamReader.loadNextBatch(ArrowStreamReader.java:106)
at net.snowflake.client.jdbc.ArrowResultChunk.readArrowStream(ArrowResultChunk.java:78)
at net.snowflake.client.jdbc.SnowflakeChunkDownloader$2.downloadAndParseChunk(SnowflakeChunkDownloader.java:879)
... 6 more

Could you please advice if there is any ETA to remove/replace sun.misc.Unsafe usage in favour Varhandle etc? Assuming new JDK LTS is scheduled for September 2021, it would good to get rid of Unsafe usages for good.

Thanks in advance!

[sfc-gh-mknister]

Hi, this error seems to be caused by the Arrow library that we use as a dependency. As a workaround for now, could you try setting -Dio.netty.tryReflectionSetAccessible=true? I see this in their documentation: "For java 9 or later, should set "-Dio.netty.tryReflectionSetAccessible=true". This fixes java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available. thrown by netty."

[anders-ahlgren]

Hi, I'm having the same problem (also 3.13.2, but with Java 15). I did try setting -Dio.netty.tryReflectionSetAccessible=true but that did not help.

[dzlab]

@sfc-gh-mknister any updates on this ticket? I'm having same issue when running on spark

I'm using:

• spark 3.1.1
• java 11 installed via java-11-amazon-corretto-devel-11.0.10.9-1.x86_64.rpm
• using those dependencies

"net.snowflake" % "snowflake-jdbc" % "3.12.10",
"net.snowflake" %% "spark-snowflake" % "2.6.0-spark_2.4"

I read data from Snowflake like this

val format = net.snowflake.spark.snowflake.Utils.SNOWFLAKE_SOURCE_NAME
val options = Map("sfURL" -> "", "sfUser" -> "", "sfPassword" -> "", "sfDatabase" -> "", "sfWarehouse" -> "", "dbtable" -> "")
sqlContext.read.format(format).options(options).load()

I'm passing the Dio.netty.tryReflectionSetAccessible=true java option to both driver and executor

spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:+PrintFlagsFinal -XX:+UseContainerSupport -Dio.netty.tryReflectionSetAccessible=true -javaagent:/opt/spark/jars/jmx_prometheus_javaagent.jar=9091:/opt/spark/conf/prometheus-config.yml  
spark.driver.memory=1G
spark.driver.memoryOverhead=385M
spark.driver.port=7077
spark.executor.cores=1
spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:+PrintFlagsFinal -XX:+UseContainerSupport -Dio.netty.tryReflectionSetAccessible=true 

here is full stack trace when trying to read from Snowflake table

ResultStage 25 (count at DatasetOutputNodeInfo.scala:26) failed in 0.065 s due to Job aborted due to stage failure: Task 0 in stage 25.0 failed 4 times, most recent failure: Lost task 0.3 in stage 25.0 (TID 95) (10.0.128.72 executor 2): java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
	at net.snowflake.client.jdbc.internal.io.netty.util.internal.PlatformDependent.directBuffer(PlatformDependent.java:399)
	at net.snowflake.client.jdbc.internal.io.netty.buffer.NettyArrowBuf.getDirectBuffer(NettyArrowBuf.java:243)
	at net.snowflake.client.jdbc.internal.io.netty.buffer.NettyArrowBuf.nioBuffer(NettyArrowBuf.java:233)
	at net.snowflake.client.jdbc.internal.io.netty.buffer.ArrowBuf.nioBuffer(ArrowBuf.java:247)
	at net.snowflake.client.jdbc.internal.apache.arrow.vector.ipc.ReadChannel.readFully(ReadChannel.java:81)
	at net.snowflake.client.jdbc.internal.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:696)
	at net.snowflake.client.jdbc.internal.apache.arrow.vector.ipc.message.MessageChannelReader.readNext(MessageChannelReader.java:68)
	at net.snowflake.client.jdbc.internal.apache.arrow.vector.ipc.ArrowStreamReader.loadNextBatch(ArrowStreamReader.java:106)
	at net.snowflake.client.jdbc.ArrowResultChunk.readArrowStream(ArrowResultChunk.java:78)
	at net.snowflake.client.core.SFArrowResultSet.buildFirstChunk(SFArrowResultSet.java:278)
	at net.snowflake.client.core.SFArrowResultSet.<init>(SFArrowResultSet.java:187)
	at net.snowflake.client.jdbc.SnowflakeResultSetSerializableV1.getResultSet(SnowflakeResultSetSerializableV1.java:933)
	at net.snowflake.spark.snowflake.io.ResultIterator.<init>(SnowflakeResultSetRDD.scala:69)
	at net.snowflake.spark.snowflake.io.SnowflakeResultSetRDD.compute(SnowflakeResultSetRDD.scala:35)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.$anonfun$getOrCompute$1(RDD.scala:386)
	at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1423)
	at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1350)
	at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1414)
	at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1237)
	at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:384)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:335)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.sql.execution.SQLExecutionRDD.$anonfun$compute$1(SQLExecutionRDD.scala:52)
	at org.apache.spark.sql.internal.SQLConf$.withExistingConf(SQLConf.scala:124)
	at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

[sfc-gh-dbryant]

I am using Flyway (7.14.1) with Snowflake JDBC driver 3.13.7 and OpenJDK 16 getting this error:
SQL State : XX000
Error Code : 200001
Message : JDBC driver internal error: Fail to retrieve row count for first arrow chunk: sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available.

Caused by: net.snowflake.client.jdbc.SnowflakeSQLLoggedException: JDBC driver internal error: Fail to retrieve row count for first arrow chunk: sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available.

I am passing -jdbcProperties.Dio.netty.tryReflectionSetAccessible=true per the suggestion above without success.

[IRus]

@sfc-gh-dbryant in JDK16 you should pass -Djdk.module.illegalAccess=permit
In JDK 17 --add-opens jdk.unsupported/sun.misc=ALL-UNNAMED (https://jdk.java.net/17/release-notes#JDK-8266851) but it doesn't work for me /shrug
Also https://issues.apache.org/jira/browse/ARROW-12747

[mdiskin]

Duplicate/related issue:
#533

[cloojure]

I got it to work for [net.snowflake/snowflake-jdbc "3.13.8"] using Java 11.

[IRus]

@cloojure driver works fine on jdk11, this ticket about jdk16-17 issues

[anders-ahlgren]

@IRus Except I have the same problem -- or at least a very, very similar problem -- on jdk 15.

[fprochazka]

Related #589

[apixandru]

If anyone else is having issues with this, you could add this to your startup script as a workaround until this is solved

export JDK_JAVA_OPTIONS="$JDK_JAVA_OPTIONS --add-opens=java.base/java.nio=ALL-UNNAMED"