KCL 2.0.5 running into failure while with exception: SEVERE: Failed to make request to https://kinesis.us-east-1.amazonaws.com/ java.lang.IllegalArgumentException: The first frame must be a headers frame. Was: DATA

[mmritesh]

I am following example from https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-kcl-java.html. But unfortunately its running into this issue as soon as I run the application:

`Nov 22, 2018 5:26:34 PM software.amazon.kinesis.coordinator.Scheduler initialize
INFO: Syncing Kinesis shard info
Nov 22, 2018 5:26:36 PM software.amazon.awssdk.http.nio.netty.internal.RunnableRequest handleFailure
SEVERE: Failed to make request to https://kinesis.us-east-1.amazonaws.com/
java.lang.IllegalArgumentException: The first frame must be a headers frame. Was: DATA
at io.netty.handler.codec.http2.ForkedHttp2MultiplexCodec$DefaultHttp2StreamChannel$Http2ChannelUnsafe.write(ForkedHttp2MultiplexCodec.java:1007)
at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1391)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
at io.netty.handler.timeout.WriteTimeoutHandler.write(WriteTimeoutHandler.java:113)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
at software.amazon.awssdk.http.nio.netty.internal.http2.HttpToHttp2OutboundAdapter.write(HttpToHttp2OutboundAdapter.java:94)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:801)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
at com.typesafe.netty.http.HttpStreamsHandler.completeBody(HttpStreamsHandler.java:298)
at com.typesafe.netty.http.HttpStreamsHandler.access$400(HttpStreamsHandler.java:14)
at com.typesafe.netty.http.HttpStreamsHandler$3$1.run(HttpStreamsHandler.java:276)
at com.typesafe.netty.http.HttpStreamsHandler.executeInEventLoop(HttpStreamsHandler.java:342)
at com.typesafe.netty.http.HttpStreamsHandler.access$300(HttpStreamsHandler.java:14)
at com.typesafe.netty.http.HttpStreamsHandler$3.complete(HttpStreamsHandler.java:273)
at com.typesafe.netty.HandlerSubscriber$3.operationComplete(HandlerSubscriber.java:244)
at com.typesafe.netty.HandlerSubscriber$3.operationComplete(HandlerSubscriber.java:241)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:162)
at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95)
at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30)
at com.typesafe.netty.HandlerSubscriber.onComplete(HandlerSubscriber.java:241)
at software.amazon.awssdk.http.nio.netty.internal.RunnableRequest$StreamedRequest$1.onNext(RunnableRequest.java:376)
at software.amazon.awssdk.http.nio.netty.internal.RunnableRequest$StreamedRequest$1.onNext(RunnableRequest.java:351)
at software.amazon.awssdk.core.internal.http.async.SimpleRequestProvider$SubscriptionImpl.request(SimpleRequestProvider.java:82)
at com.typesafe.netty.HandlerSubscriber.maybeRequestMore(HandlerSubscriber.java:271)
at com.typesafe.netty.HandlerSubscriber.maybeStart(HandlerSubscriber.java:207)
at com.typesafe.netty.HandlerSubscriber.provideSubscription(HandlerSubscriber.java:196)
at com.typesafe.netty.HandlerSubscriber.access$000(HandlerSubscriber.java:18)
at com.typesafe.netty.HandlerSubscriber$1.run(HandlerSubscriber.java:184)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)

Nov 22, 2018 5:26:36 PM software.amazon.kinesis.leases.ShardSyncTask call
SEVERE: Caught exception while sync'ing Kinesis shards and leases
java.lang.RuntimeException: software.amazon.awssdk.core.exception.SdkClientException
at software.amazon.kinesis.retrieval.AWSExceptionManager.apply(AWSExceptionManager.java:65)
at software.amazon.kinesis.leases.KinesisShardDetector.listShards(KinesisShardDetector.java:172)
at software.amazon.kinesis.leases.KinesisShardDetector.listShards(KinesisShardDetector.java:132)
at software.amazon.kinesis.leases.HierarchicalShardSyncer.getShardList(HierarchicalShardSyncer.java:248)
at software.amazon.kinesis.leases.HierarchicalShardSyncer.checkAndCreateLeaseForNewShards(HierarchicalShardSyncer.java:81)
at software.amazon.kinesis.leases.ShardSyncTask.call(ShardSyncTask.java:67)
at software.amazon.kinesis.metrics.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:53)
at software.amazon.kinesis.coordinator.Scheduler.initialize(Scheduler.java:247)
at software.amazon.kinesis.coordinator.Scheduler.run(Scheduler.java:213)
at java.lang.Thread.run(Thread.java:748)
Caused by: software.amazon.awssdk.core.exception.SdkClientException
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.handle(AsyncRetryableStage.java:143)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.lambda$execute$0(AsyncRetryableStage.java:121)
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$Completable.lambda$completeExceptionally$1(MakeAsyncHttpRequestStage.java:208)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: java.lang.IllegalArgumentException: The first frame must be a headers frame. Was: DATA
at io.netty.handler.codec.http2.ForkedHttp2MultiplexCodec$DefaultHttp2StreamChannel$Http2ChannelUnsafe.write(ForkedHttp2MultiplexCodec.java:1007)
at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1391)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
at io.netty.handler.timeout.WriteTimeoutHandler.write(WriteTimeoutHandler.java:113)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
at software.amazon.awssdk.http.nio.netty.internal.http2.HttpToHttp2OutboundAdapter.write(HttpToHttp2OutboundAdapter.java:94)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:801)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
at com.typesafe.netty.http.HttpStreamsHandler.completeBody(HttpStreamsHandler.java:298)
at com.typesafe.netty.http.HttpStreamsHandler.access$400(HttpStreamsHandler.java:14)
at com.typesafe.netty.http.HttpStreamsHandler$3$1.run(HttpStreamsHandler.java:276)
at com.typesafe.netty.http.HttpStreamsHandler.executeInEventLoop(HttpStreamsHandler.java:342)
at com.typesafe.netty.http.HttpStreamsHandler.access$300(HttpStreamsHandler.java:14)
at com.typesafe.netty.http.HttpStreamsHandler$3.complete(HttpStreamsHandler.java:273)
at com.typesafe.netty.HandlerSubscriber$3.operationComplete(HandlerSubscriber.java:244)
at com.typesafe.netty.HandlerSubscriber$3.operationComplete(HandlerSubscriber.java:241)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:162)
at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95)
at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30)
at com.typesafe.netty.HandlerSubscriber.onComplete(HandlerSubscriber.java:241)
at software.amazon.awssdk.http.nio.netty.internal.RunnableRequest$StreamedRequest$1.onNext(RunnableRequest.java:376)
at software.amazon.awssdk.http.nio.netty.internal.RunnableRequest$StreamedRequest$1.onNext(RunnableRequest.java:351)
at software.amazon.awssdk.core.internal.http.async.SimpleRequestProvider$SubscriptionImpl.request(SimpleRequestProvider.java:82)
at com.typesafe.netty.HandlerSubscriber.maybeRequestMore(HandlerSubscriber.java:271)
at com.typesafe.netty.HandlerSubscriber.maybeStart(HandlerSubscriber.java:207)
at com.typesafe.netty.HandlerSubscriber.provideSubscription(HandlerSubscriber.java:196)
at com.typesafe.netty.HandlerSubscriber.access$000(HandlerSubscriber.java:18)
at com.typesafe.netty.HandlerSubscriber$1.run(HandlerSubscriber.java:184)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
... 1 more

Nov 22, 2018 5:26:38 PM software.amazon.awssdk.http.nio.netty.internal.RunnableRequest handleFailure
SEVERE: Failed to make request to https://kinesis.us-east-1.amazonaws.com/
java.lang.IllegalArgumentException: The first frame must be a headers frame. Was: DATA
at io.netty.handler.codec.http2.ForkedHttp2MultiplexCodec$DefaultHttp2StreamChannel$Http2ChannelUnsafe.write(ForkedHttp2MultiplexCodec.java:1007)
at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1391)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
at io.netty.handler.timeout.WriteTimeoutHandler.write(WriteTimeoutHandler.java:113)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
at software.amazon.awssdk.http.nio.netty.internal.http2.HttpToHttp2OutboundAdapter.write(HttpToHttp2OutboundAdapter.java:94)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:801)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
at com.typesafe.netty.http.HttpStreamsHandler.completeBody(HttpStreamsHandler.java:298)
at com.typesafe.netty.http.HttpStreamsHandler.access$400(HttpStreamsHandler.java:14)
at com.typesafe.netty.http.HttpStreamsHandler$3$1.run(HttpStreamsHandler.java:276)
at com.typesafe.netty.http.HttpStreamsHandler.executeInEventLoop(HttpStreamsHandler.java:342)
at com.typesafe.netty.http.HttpStreamsHandler.access$300(HttpStreamsHandler.java:14)
at com.typesafe.netty.http.HttpStreamsHandler$3.complete(HttpStreamsHandler.java:273)
at com.typesafe.netty.HandlerSubscriber$3.operationComplete(HandlerSubscriber.java:244)
at com.typesafe.netty.HandlerSubscriber$3.operationComplete(HandlerSubscriber.java:241)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:162)
at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95)
at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30)
at com.typesafe.netty.HandlerSubscriber.onComplete(HandlerSubscriber.java:241)
at software.amazon.awssdk.http.nio.netty.internal.RunnableRequest$StreamedRequest$1.onNext(RunnableRequest.java:376)
at software.amazon.awssdk.http.nio.netty.internal.RunnableRequest$StreamedRequest$1.onNext(RunnableRequest.java:351)
at software.amazon.awssdk.core.internal.http.async.SimpleRequestProvider$SubscriptionImpl.request(SimpleRequestProvider.java:82)
at com.typesafe.netty.HandlerSubscriber.maybeRequestMore(HandlerSubscriber.java:271)
at com.typesafe.netty.HandlerSubscriber.maybeStart(HandlerSubscriber.java:207)
at com.typesafe.netty.HandlerSubscriber.provideSubscription(HandlerSubscriber.java:196)
at com.typesafe.netty.HandlerSubscriber.access$000(HandlerSubscriber.java:18)
at com.typesafe.netty.HandlerSubscriber$1.run(HandlerSubscriber.java:184)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)

`

[sahilpalvia]

Could you provide the configuration setup and how is the KinesisAsyncClient being created.

[oboudry]

I'm experiencing the exact same problem with KCL 2.0.5, same failed to make request to https://kinesis.us-east-1.amazonaws.com, with same "first frame must be a header frame" messge.
The way I came to this is by first implementing a working KCL 1.0 client using the following documentation: https://docs.aws.amazon.com/streams/latest/dev/learning-kinesis-module-one.html, and then converting it to KCL 2.0 using this documentation: https://docs.aws.amazon.com/streams/latest/dev/kcl-migration.html#client-configuration.
I put my code and config on BitBucket git@bitbucket.org:oboudry/stocktradesapplication.git it's and eclipse and Maven project.

[mwu]

I ran into this as well, it looks like there's a dependency conflict between com.amazonaws:aws-java-sdk-kinesisvideo and software.amazon.kinesis:amazon-kinesis-client over io.netty:netty-codec-http. The former brings in 4.1.17.Final directly (even up to the most recent release at time of writing (1.11.479)) and the latter brings in 4.1.26.Final through software.amazon.awssdk:netty-nio-client, which has a lot of other netty dependencies that don't work with netty-codec-http:4.1.17.Final.

So I believe the issue is really that aws-java-sdk-kinesisvideo from the java aws v1 sdk can't co-exist with kcl2. The code in @oboudry's repo seems to run with a dependency exclusion, for example (I'm not suggesting you do this, it's better to only pull the individual aws sdk dependencies you need rather than bring everything into the classpath via aws-java-sdk):

    <dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>aws-java-sdk</artifactId>
      <version>1.11.475</version>
      <scope>compile</scope>
      <exclusions>
        <exclusion>
          <groupId>com.amazonaws</groupId>
          <artifactId>aws-java-sdk-kinesisvideo</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

[oboudry]

Effectively importing just core and kinesis instead of the full sdk resolves the problem.
Java is not my primary programming language and I did not realize I was importing so many different modules.
Thanks !