netty-tcnative conflict, and using hadoop

[kenzshelley]

Hi,

I'm running into an issue with the cloud-bigtable-client that is conceptually similar to #1445.

Initially, I was using bigtable-hbase-1.2 at 0.9.5. I need to have netty-tcnative:2.0.5.Final in my project for other dependancies, which caused problems because bigtable-hbase-1.2 shades a version of netty that isn't compatible with netty-tcnative:2.0.5.Final.

Based on the discussion in #1445, I switched to bigtable-hbase-1.x:1.0.0-pre1 with netty excluded, and that solved my original issue. However, my project also uses hadoop, and this new artifact caused all hadoop jobs to fail due to a bunch of class not found exceptions.

Based on #1436, I tried using bigtable-hbase-1.x:-hadoop1.0.0-pre1. That fixed the issue with hadoop, but because that artifact shades netty I still had the original problem with my incompatible netty-tcnative version.

Is there any way I can make netty-tcnative:2.0.5.Final work with the bigtable client and hadoop?

[sduskis]

netty-tcnative:2.0.5.Final has been a bit of a pain... Sorry about that. As of now, we don't have an artifact that would work well in your case. We'd need a new hadoop oriented artifact that shades in newer io.netty:* versions that are compatible with netty-tcnative:2.0.5.Final.

Can you give a bit more background as to why you need netty-tcnative:2.0.5.Final? Is there another dependency you have that requires that version?

[kenzshelley]

Okay, I see. Thanks for the quick reply!

We wanted 2.0.5.Final because the latest version of grpc uses it. Just talked to some people and it sounds like there's some potential that 2.0.1.Final would work too, so I'll try that.

Alternatively, I might just try to re-shade the shaded netty classes to point to the unshaded version, which I think will effectively allow me to sub in my version of netty.

[kenzshelley]

Update: reverting to 2.0.1.Final didn't help

[igorbernstein2]

Hi,

The root of the issue is that grpc 1.4.0 updated its dependency on netty from version 4.1.8 to 4.1.11. During that transition, netty substantially changed netty-tcnative (they changed their package from tomcat to netty.internal). The cloud bigtable client has historically recommended for its users to explicitly add the tcnative dependency as opposed to transitively depending on it. This created a big issue for us: upgrading to newer netty would be a breaking change for our users, so we stayed with the older version of netty, which is shaded along in the -shaded & -hadoop jars. However since tcnative couldn't be shaded, it was kept as a transitive dependency, forcing our users to stay with the previous version of tcnative. So tcnative >= 2 will not work.

I'm working on a more permanent solution and also I think I have a temporary workaround for you. But I'd like to understand your situation a bit better first. Would you mind giving me a bit more detail as to:

1. what you are using grpc for? Main reason I ask is because grpc has guava, protobuf & netty dependencies that conflict with hadoop, which is why we are using shading for bigtable-hbase-1.x-hadoop.

2. Can you clarify your usage of hadoop?

3. Are you trying to use bigtable in a mapreduce job? or are you using hadoop client apis to manage your hadoop cluster from the outside? Mainly I'd like to understand what your classpath looks like and who controls it.

Thanks!

[kenzshelley]

Hi Igor,

I see, definitely a difficult situation.

Also: a little background, I'm an engineer at Square, working in one of our large java repos.

To answer your questions:

1. We use grpc for rpc calls between some of our java services. My best guess as to why using grpc hasn't caused hadoop problems in the past is because we probably don't directly use grpc in an app that uses hadoop elsewhere. We just try to pin dependency versions for 3rd party jars across this entire repo, so my app is being impacted by the fact that we want this higher tcnative version in a different app in the same repo.
2. We use hadoop to run a variety of map reduce jobs in one of our apps.
3. We use bigtable both in a mapreduce job (running on dataproc), and just normally in an application that also happens to run mapreduce jobs.

Let me know if I can clarify any of that!

Thanks,
Mackenzie

[igorbernstein2]

Thanks for the detailed information!
As a temporary solution, in the projects that require bigtable-hbase-1.x-hadoop, please override the version of netty-tcnative to version 1.1.33.Fork26. This will break grpc outside of the bigtable client, but as you mentioned, its not currently being used.

As a longer term solution we will shaded netty-tcnative into bigtable-hbase-1.x-hadoop. This is currently being worked on by grpc:
grpc/grpc-java#2485

[kenzshelley]

We try to avoid having app-specific dependancies, but yeah I think that would do the trick and I don't really see any other solutions. Thanks!

[igorbernstein2]

I'll close this ticket for now. But I'll make sure to update it when this workaround is no longer necessary. (ie. when netty-tcnative is shaded.

[kenzshelley]

Sounds good, thanks!

[igorbernstein2]

Please follow progress on #1487