protoc-gen-grpc-gateway: can't generate gateway stubs for multiple packages in one go

[adambabik]

🐛 Bug Report

I would like to execute protoc with protoc-gen-grpc-gateway to generate stubs for all my packages in one go. Unfortunately, I am getting an error: --grpc-gateway_out: inconsistent package names: av1 bv1.

To Reproduce

1. Organize the packages like this:

proto
└── my-org
    ├── a
    │   └── v1
    │       └── a.proto
    └── b
        └── v1
            └── b.proto

where a.proto's package is av1 and b.proto's package name is bv1.

2. Execute protoc in the following way:

protoc \
    -I=/proto \
    -I=/third_party \
    --go_opt=paths=source_relative \
    --go_out=/gen \
    --go-grpc_opt=paths=source_relative \
    --go-grpc_out=/gen \
    --grpc-gateway_out=paths=source_relative,logtostderr=true:/gen \
    --validate_out=paths=source_relative,lang=go:/gen \
    $(find /proto -name '*.proto')

Expected behavior

I expect to have grpc-gateway stubs generated in corresponding directories.

Actual Behavior

I am getting an error --grpc-gateway_out: inconsistent package names: av1 bv1. However, when executing the protoc command package-by-package, it seems that the stubs are generated properly in the corresponding directories in accordance with paths=source_relative.

Your Environment

I execute this in a Docker container. It uses golang:1.14-stretch as a base image.

The version of protoc-gen-grpc-gateway is v1.14.7.

[johanbrandhorst]

This would be cool to support, but it's definitely a bigger piece of work. I'd love to hepl you bring this in if you have the spare cycles but we're not going to be able to add this unless it's as a community contribution. Note that protoc-gen-go doesn't support this (golang/protobuf#39), so you usually end up writing separate protoc invocations for each package anyway, but I can see this being a request for non-go users.

[adambabik]

@johanbrandhorst I do have spare cycles and actually started digging into the code. If you could give me some introduction how this feature can be added that would be awesome.

Regarding protoc-gen-go, I use google.golang.org/protobuf@v1.25.0 and it actually works fine. In the issue that you mentioned, the last comment claims it was fixed in 1.20.

[johanbrandhorst]

Ha, TIL, thanks for that! I think for something like this we'd be looking at doing it for v2 exclusively. I briefly looked into rewriting the generator using the new generator framework they have in the new go protobuf stack. If you'd be interested in taking that on, this might be something we could do at the same time. What do you think? Otherwise, you'd probably want to start looking at the generator entrypoint: https://github.com/grpc-ecosystem/grpc-gateway/blob/v2/protoc-gen-grpc-gateway/main.go. I'm not that familiar with this part of the code and it's overdue something of a rewrite.

[adambabik]

It looks like work on v2 is pretty advanced. I think it make sense, especially as there is this new generator framework. It also doesn't seem like a lot of people expects this feature in v1.

I will start digging into the generator entrypoint and the new framework.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[johanbrandhorst]

Is this done?

[adambabik]

I just checked this and it works with v2! In v1 I needed to write something like this:

CMD protoc \
    -I=/proto \
    -I=/third_party \
    --go_opt=paths=source_relative \
    --go_out=/gen \
    --go-grpc_opt=paths=source_relative \
    --go-grpc_out=/gen \
    --grpc-gateway_out=paths=source_relative,logtostderr=true:/gen \
    --validate_out=paths=source_relative,lang=go:/gen \
    $(find /proto -name '*.proto') && \
    # Can't make grpc-gateway to work for multiple package
    # in one go yet: https://github.com/grpc-ecosystem/grpc-gateway/issues/1612.
    for package in ${GRPC_GATEWAY_PACKAGES}; do \
        echo "generating gRPC Gateway stubs for $package: $(find $package -name '*.proto')"; \
        protoc \
            -I=/proto \
            -I=/third_party \
            --go_opt=paths=source_relative \
            --go_out=/gen \
            --go-grpc_opt=paths=source_relative \
            --go-grpc_out=/gen \
            --grpc-gateway_out=paths=source_relative,logtostderr=true:/gen \
            --validate_out=paths=source_relative,lang=go:/gen \
            $(find $package -name '*.proto'); \
    done

And now, with v2, I can get rid of the for loop and move --grpc-gateway_out=paths=source_relative,logtostderr=true:/gen to the first protoc call.