how to do tls auth in grpc+grpc-gateway

[vtolstov]

I'm already have grpc server with my own letsencrypt cert behind Nginx:
nginx listens ssl, and proxy_pass to https grpc-gateway

grpc server listens on int_ip:7777, nginx listens on ext_ip: 443, hostname - external domain name on nginx, endpoint intenal listen addr - int_ip:7777

grpc code stuff:

certPool, err := certPool() // add fullchain.pem from letsencrypt
if err != nil {
  return nil, err
}
 opts = append(opts, grpc.Creds(credentials.NewClientTLSFromCert(certPool, hostname)))
}
opts = append(opts, grpc.StreamInterceptor(grpc_auth.StreamServerInterceptor(authHandlerFunc)))
opts = append(opts, grpc.UnaryInterceptor(grpc_auth.UnaryServerInterceptor(authHandlerFunc)))
grpcs := grpc.NewServer(opts...)

gw stuff:

   opts = append(opts, grpc.WithTimeout(2*time.Second), grpc.WithBlock())

        certPool, err := certPool()
        if err != nil {
                return nil, err
        }

                certificate, err := tls.LoadX509KeyPair(crt, key)
                if err != nil {
                        return nil, fmt.Errorf("could not load server key pair: %s", err)
                }
                creds := credentials.NewTLS(&tls.Config{
                        ServerName:   hostname,
                        Certificates: []tls.Certificate{certificate},
                        RootCAs:      certPool,
                })
                opts = append(opts, grpc.WithTransportCredentials(creds))
 conn, err := grpc.Dial(endpoint, opts...)

main stuff:

        certificate, err := tls.LoadX509KeyPair(crt, key)
        if err != nil {
                return nil, err
        }

        tlsConf := &tls.Config{
                Certificates: []tls.Certificate{certificate},
                Rand:         rand.Reader,
        }

ln, err := net.Listen("tcp", "127.0.0.1:7777")
tls.NewListener(ln, tcpConf)

       cm := cmux.New(ln)
        grpcLn := cm.MatchWithWriters(cmux.HTTP2MatchHeaderFieldPrefixSendSettings("content-type", "application/grpc"))
        //wsLn := cm.Match(cmux.HTTP1HeaderField("Upgrade", "websocket"))
        restLn := cm.Match(cmux.HTTP1Fast())

        grpcs, err := prepareGRPC(ctx)
        if err != nil {
                log.Fatal(err)
        }

        rests, err := prepareREST(ctx)
        if err != nil {
                log.Fatal(err)
        }

        go func() {
                if err = grpcs.Serve(grpcLn); err != cmux.ErrListenerClosed {
                        log.Fatal(err)
                }
        }()
        go func() {
                if err = rests.Serve(restLn); err != cmux.ErrListenerClosed {
                        log.Fatal(err)
                }
        }()

        if err := cm.Serve(); !strings.Contains(err.Error(), "use of closed network connection") {
                log.Fatal(err)
        }

when i'm try to start my app i have context deadline exceeded because grpc-gateway can't connect to grpc server

[vtolstov]

Without tls all works fine with cmux, i can connect without certs to grpc server and do some rest requests, but with tls enabled i'm always have timeouts

[vtolstov]

i think that me question is:
does it possible to use the same cert on client (gateway) and on server ?

[johanbrandhorst]

mTLS between the grpc-gateway client and the gRPC server should absolutely be possible. You have to turn off client authentication for your gateway though, unless you expect those clients to also present a certificate. https://github.com/gogo/grpc-example/blob/master/main.go shows an example of how to do server auth, and adding client auth on top shouldn't be much work.

[vtolstov]

I found the root of my issue - cmux.
I need to run both grpc and rest on the same port. In case of plain tcp - all fine, cmux works by query request headers. In case of crypt - i need to pass to grpc server plain tcp listener, but for rest i need to pass tls encrypted listener.

[vtolstov]

So my new question - does have somebody already knows how to do encrypted grpc+rest with cmux.

[vtolstov]

i'm solve. With plain mode i'm use cmux to get rest and grpc on the same port.
For tls mode i'm use another example from @philips

[achew22]

Could you post the code you ended up using in here? It might be useful to others in the future

[vtolstov]

        tcpl, err := net.Listen("tcp", *endpoint)
        if err != nil {
                log.Fatalf("%s", err)
        }
        defer tcpl.Close()

        if !secure {
                tcpm = cmux.New(tcpl)
                //              grpcl = tcpm.MatchWithWriters(cmux.HTTP2MatchHeaderFieldPrefixSendSettings("content-type", "application/
grpc"))
                grpcl = tcpm.Match(cmux.HTTP2())
                restl = tcpm.Match(cmux.HTTP1())
        } else {
                tlsl, err := tlsListener(tcpl)
                if err != nil {
                        log.Fatal("%s", err)
                }
                grpcl = tlsl
                restl = tlsl
        }

        grpcs, err := prepareGRPC(ctx)
        if err != nil {
                log.Fatal(err)
        }
        rests, err := prepareREST(ctx, grpcs)
        if err != nil {
                log.Fatal(err)
        }

        if !secure {
                go func() {
                        if err = grpcs.Serve(grpcl); err != cmux.ErrListenerClosed {
                                log.Fatal(err)
                        }
                }()
        }
        go func() {
                if err = rests.Serve(restl); err != cmux.ErrListenerClosed {
                        log.Fatal(err)
                }
        }()

        if !secure {
                if err := tcpm.Serve(); !strings.Contains(err.Error(), "use of closed network connection") {
                        log.Fatal(err)
                }
        } else {
                select {}
        }

[vtolstov]

main part is above. may be i create some repo to put all stuff to it and provide link.

[vtolstov]

no, i don't create repo

[dayadev]

@vtolstov Thanks!

[johanbrandhorst]

I've written about gRPC client authentication which might be useful: https://jbrandhorst.com/post/grpc-auth/

[azhuo-va]

@vtolstov Can you please also post the function prepareGRPC and prepareREST? I am wondering why prepareREST needs to accept grpcs as a parameter. Thanks!

[judavi]

@vtolstov thanks!! your example helped me to solve an issue!!