HDFS data transfer encryption support

[mxk1235]

I was attempting to use this library against an hdfs cluster that has hadoop.rpc.protection setting in core-site.xml set to privacy, as well as dfs.encrypt.data.transfer which is true in hdfs-site.xml. i believe those apply to the protobuf rpc interface.

the error message i received was the following.
no available namenodes: SASL handshake: wrong Token ID. Expected 0504, was 6030

after some debugging i think it occurs here https://github.com/colinmarc/hdfs/blob/master/internal/rpc/kerberos.go#L67 . i suspect the namenode is replying with an encrypted message, while the doKerberosHandshake() expects otherwise.

on first look the library just sets the default value for dfs.encrypt.data.transfer property as false

hdfs/internal/protocol/hadoop_hdfs/hdfs.proto

Line 402 in f87e1d6

optional bool encryptDataTransfer = 6 [default = false];

and there is no way of creating a client with that property set to true.

hdfs/client.go

Line 122 in f87e1d6

func NewClient(options ClientOptions) (*Client, error) {

there is a fetchDefaults() function, that's only invoked by file_writer, but not by file_reader (e.g. Stat(), Readdir(), and Read() methods).

can you comment if i'm digging in the right place, and whether the encrypted part of the protocol applies to the read functionality?

here are relevant properties from core-site.xml

  <property>
    <name>hadoop.security.authentication</name>
    <value>kerberos</value>
  </property>
  <property>
    <name>hadoop.security.authorization</name>
    <value>true</value>
  </property>
  <property>
    <name>hadoop.rpc.protection</name>
    <value>privacy</value>
  </property>

and hdfs-site.xml

  <property>
    <name>dfs.encrypt.data.transfer.algorithm</name>
    <value>3des</value>
  </property>
  <property>
    <name>dfs.encrypt.data.transfer.cipher.suites</name>
    <value>AES/CTR/NoPadding</value>
  </property>
  <property>
    <name>dfs.encrypt.data.transfer.cipher.key.bitlength</name>
    <value>256</value>
  </property>
  <property>
    <name>dfs.namenode.acls.enabled</name>
    <value>true</value>
  </property>

[colinmarc]

Hi there,

Thanks for the report!

This looks like an accurate assessment of the missing functionality. I don't know how it works, but I'll try to do some research.

[mnotti]

Getting very similar issue at the same spot during the doKerberosHandshake():
Couldn't connect to namenode: no available namenodes: SASL handshake: wrong Token ID. Expected 0504, was 603f

However, I don't have either hadoop.rpc.protection in core-site.xml, or dfs.encrypt.data.transfer in hdfs-site.xml like mxk12345 above.

Unsure what to look for here. I get the same issue using the binary provided as well as the lib api itself.

[colinmarc]

@Shastick do you have any idea on this?

[Shastick]

Sorry for the delay. @mxk1235 you are digging in the correct place. However, we did not implement the "privacy" mode of operation (as well as any other work that relies on Kerberos to sign or encrypt payloads) when integrating gokrb5 into the client. This is mainly because we did not need it, and the additional work was non-trivial.

If you are interested in attempting to implement this, the RFC's mentionned in the source code (possible the one of gokrb5, I'm not sure anymore) will contain the relevant pointers.

Hope this helps

[mxk1235]

@Shastick i kept looking and i think the error occurs at this line

hdfs/internal/rpc/kerberos.go

Line 67 in f87e1d6

err = nnToken.Unmarshal(resp.GetToken(), true)

which i think is past communication with the KDC and rather talking to the namenode. every time i look i get a little deeper, but obviously this is a lot to implement.

[colinmarc]

@Shastick thanks. I know you've given a lot of your time already, but it would be immensely helpful if you could create issues for any missing kerberos things you can think of. I didn't even know wire encryption was a thing.

[Shastick]

@mxk1235 the line you are referring to corresponds to the deserialisation (and verification) of the token that was sent back by the namenode. If this token is anything else than a signed wrap token (e.g, a sealed wrap token) then it blows up at this point.

@colinmarc I'll peruse https://tools.ietf.org/html/rfc4121 a bit when I get some time and will try to remember what other modes we skipped over

[mxk1235]

that’s how i’m reading it as well. in the hadoop code, the payload is encrypted in AesCtrNoPadding (only supported cipher), and the spec mentions retrieving the key from the ticket. however, this only one instance of encryption in the protocol, and there is at least one other setting which is 3des.

[freshsun]

I get the same error today. "no available namenodes: SASL handshake: wrong Token ID. Expected 0504, was 6030"
I have no hadoop.rpc.protection and dfs.encrypt.data.transfer settings on the server.
I also tried native java hadoop client to list the files, I can see the command and return file list in the tcp bodies, so i guess it's not encrypted?
TCP packages from go hdfs and java hdfs command are different, java version has more packages before the package contains hex 6030 show up.

[colinmarc]

@freshsun that's really useful information. Can you post a hexdump of the packet with 0504, or email it to hi@colinmarc.com if you're not comfortable posting it?

[freshsun]

@mnotti check my workaround solution #144 (comment)

[jackyliang]

Hi @colinmarc!

My name is Jacky and I am a product manager at MemSQL (colleague of @mxk1235)

I would like to 👍 on implementing support for RPC authentication + HDFS wire encryption for this library -- it will solve a major need for our customers that run Hadoop clusters but want to ingest data into MemSQL in a secure fashion. At this point, all of them require auth + encryption due to the size of these companies.

My colleague's note are an accurate reflection on one of the things that this awesome library doesn't yet support (hadoop.rpc.protection set to privacy which enables RPC authentication, hdfs.encrypt.data.transfer set to true for HDFS encrypted data transfer)