We want to be able to fetch images from PV (RGB) and VLC (tracking) cameras of the Hololens 2, and send them to SolAR mapping and relocalization services in the cloud.
This document focuses on how to send the data from the Hololens 2 to a server that will act as a proxy to call the SolAR cloud services, i.e. we will neither describe how to actually get the data from the Hololens sensors, nor will we describe how the C++ proxy server transfers these data to the SolAR cloud services (this is described in the documentation regarding SolAR remoting feature).
The approach is using gRPC because it addresses several requirements:
- being portable: we should reuse this feature with other devices and toolkit that allow to get camera frames. Being able to do it in C# allows us to theorically addresses the many devices Unity supports.
- Interact with C++: SolAR remoting is currently accessible mainly in C++. GRPC allows us be called from Hololens 2 Unity C# scripts and call the SolAR cloud services in C++ directly.
This is the architecture this document will help achieve: a Unity client running on a Hololens 2 device will use portable C# scripts to call SolAR cloud services via a proxy server written in C++ that will route the requests to the SolAR/remoting infrastructure. As explained later, due to lack of support in Unity, gRPC-Web must be used, and it required the presence of an HTTP proxy called Envoy.
This version is now the officially supported library for writing gRPC client for C# (nuget, github), as gRPC.Core is now in maintenance mode (source) This approach is also more interesting for us, since it is a C# only approach, where the previous implementation relied on the C library, which raises cross platform support concerns. Indeed, the only official support for ARM64 is ARM64/Linux (source), and we need ARM64/Windows to target Hololens 2. Using Grpc.Net.Client thus frees us from this constraint.
Unity does not support HTTP/2 for gRPC, gRPC-Web must be used. (source, source)
The following snippet shows how to configure a gRPC client that uses gRPC-Web, by configuring the channel to use a GrpcWebHandler.
var channel = GrpcChannel.ForAddress("https://localhost:5001", new GrpcChannelOptions
{
HttpHandler = new GrpcWebHandler(new HttpClientHandler())
});
var client = new Greeter.GreeterClient(channel);
var response = await client.SayHelloAsync(new HelloRequest { Name = ".NET" });If this does not work, maybe it's because the default HTTP version used is not 1.1, which can happen according to the target platform (source).
If so, you can explicitly set this version to be 1.1 by configuring the GrpcWebHandler as follows:
HttpHandler = new GrpcWebHandler(new HttpClientHandler())
{
HttpVersion = new Version("1.1")
}In order to be able to use Grpc.Net.Client package in your Unity scripts, you must have a Visual Studio C# project properly configured to declare the right refrences.
Unfortunately, the project cannot directly be configured in Visual Studio, because Unity won't be able to "see" these changes. An attempt to build the project from Unity will result in compilation errors whereas no errors will be visible in the properly configured Visual Studio project.
A way to solve this problem is to find a way to tell Unity how to add those references so it can configure the project properly.
When confronted to this kind of problem, a basic solution usually consists in adding the libraries directly in the Assets/Plugins directory of the Unity project. This make Unity aware of the libraries and they will be added to the C# Visual Studio project.
With the previous version of the gRPC library for C#, a (experimental?) Unity package was provided. This packaged contained all the required libraries, and once imported they all were placed in the Assets/Plugins directory.
For Grpc.Net.Client, to our knowledge, no such package has been released. An attempt to create it manually happened to be tedious (there is some dependencies, with different versions according to the target platform) and finally not successful.
Thankfully, a solution was found: NuGetForUnity. This project adds to Unity the ability to use the NuGet dependency manager used in Visual Studio to fetch Grpc.Net.Client and all its dependencies.
So to configure Unity to be able to use Gprc.Net.Client in the scripts, simply:
- install NuGetForUnity by imported the latest released package.
- open the newly created NuGet menu and select Manage NuGet packages
- install Grpc.Net.Client
- install Grpc.Net.Client.Web (for gRPC-Web support)
- install Google.Protobuf
These libraries should now appear in the Assets/Packages directory.
In the generated C# project, the gRPC client code snippet above should compile fine.
Visual Studio can generate stubs automatically with Grpc.Tools when a .proto file is specified in the Visual Studio project file, more details here.
As we saw, it is not possible to configure the Visual Studio project outside of Unity because the modification won't be taken into account when building the project. So it has been decided not to use Grpc.Tools, but instead generate the C# stubs manually with the protoc CLI and manually add the scripts to the project.
The gRPC CLI tools can be obtained (for Windows and Linux)
- by following the instructions of this tutorial to build them from source (select the right version tag and don't forget to specify an install directory to easily find the executables)
- by getting one of the Official gRPC Releases using your dependency manager of choice
- by downloading one of Daily Builds of
masterBranch on the same page.
Reference (for prerequisite):
https://grpc.io/docs/languages/cpp/quickstart/#build-and-install-grpc-and-protocol-buffers https://github.com/grpc/grpc/blob/v1.41.0/BUILDING.md
- Windows
set MY_INSTALL_DIR=C:\Users\jmhenaff\.local\gRPC-v1.41.0
cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=%MY_INSTALL_DIR% ..\..
cmake --build . --config Release
cmake --install . --config Release --prefix C:\Users\jmhenaff\.local\gRPC-v1.41.0- Linux
export MY_INSTALL_DIR=~/.local/grpc-v1.41.0/
cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=$MY_INSTALL_DIR ../..
make -j8
make installFinally you can add $MY_INSTALL_DIR/bin to your PATH to be able to invoke protoc.
The command line used to generate the C# stubs and the C++ gRPC service are:
protoc -I ./ --csharp_out=. greet.proto --grpc_out=. --plugin=protoc-gen-grpc=grpc_csharp_plugin.exe
protoc -I ./ --cpp_out=. greet.proto --grpc_out=. --plugin=protoc-gen-grpc=grpc_cpp_pluginThis will generate Greet.cs and GreetGrpc.cs that can now be added to the Unity Assets in order to be used by the other scripts.
This will also generate greet.pb. h, greet.pb.cc, greet.grpc.pb.h and greet.grpc.pb.h, that can be used in the C++ gRPC service.
To have a gRPC service, just implement the gRPC-generated class for this service with the desired logic.
This is an exemple from the HelloWorld) sample:
// Logic and data behind the server's behavior.
class GreeterServiceImpl final : public Greeter::Service {
Status SayHello(ServerContext* context, const HelloRequest* request,
HelloReply* reply) override {
std::string prefix("Hello ");
reply->set_message(prefix + request->name());
return Status::OK;
}
};
void RunServer() {
std::string server_address("0.0.0.0:50051");
GreeterServiceImpl service;
grpc::EnableDefaultHealthCheckService(true);
grpc::reflection::InitProtoReflectionServerBuilderPlugin();
ServerBuilder builder;
// Listen on the given address without any authentication mechanism.
builder.AddListeningPort(server_address, grpc::InsecureServerCredentials());
// Register "service" as the instance through which we'll communicate with
// clients. In this case it corresponds to an *synchronous* service.
builder.RegisterService(&service);
// Finally assemble the server.
std::unique_ptr<Server> server(builder.BuildAndStart());
std::cout << "Server listening on " << server_address << std::endl;
// Wait for the server to shutdown. Note that some other thread must be
// responsible for shutting down the server for this call to ever return.
server->Wait();
}
int main(int argc, char** argv) {
RunServer();
return 0;
}To able to receive gRPC-Web requests, an HTTP proxy must be used between the client and the server.
Currently the recommended proxy that is used for this task is Envoy.
The roadmap plans to remove this need in the future via In-process Proxies, but for now it seems like the way to go.
Envoy is available for many platforms, but I will describe the case where Envoy and the server resides on a WSL, because it allows to run an Envoy instance and the C++ gRPC service on the same physical machine (for Windows, Envoy is deployed via a Docker container).
As mentionned earlier, the installation is described for Linux as Windows installation is performed only via a Docker image, which adds some complexity and/or performance issues.
On Ubuntu Envoy can be installed simply via apt, as described in the official documentation.
Follow the instructions here to run Envoy with a default configuration and attempt to connect to it.
envoy -c envoy-demo.yamlcurl -v localhost:10000The idea is to configure Envoy with gRPC-Web filters and a listening port corresponding the the one the Hololens 2 client is connected to and a forward port corresponding the the one the gRPC services listens to.
The configuration file currently has been built by taking inspiration from the Envoy demo configuration file found here and the one used in the gRPC-Web example (here).
It listens on port 5001:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 5001Declares the gRPC-WEb filter
http_filters:
- name: envoy.filters.http.grpc_web
- name: envoy.filters.http.cors
- name: envoy.filters.http.routerAnd route to a the port to which the gRPC service listens
clusters:
- name: echo_service
connect_timeout: "0.25s"
type: LOGICAL_DNS
# Comment out the following line to test on v6 networks
dns_lookup_family: V4_ONLY
http2_protocol_options: {}
lb_policy: round_robin
load_assignment:
cluster_name: cluster_0
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 0.0.0.0
port_value: 5002The complete file:
admin:
access_log_path: /home/jmhenaff/work/tmp/admin_access.log
address:
socket_address: { address: 0.0.0.0, port_value: 9901 }
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 5001
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
access_log:
- name: envoy.access_loggers.stdout
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: echo_service
timeout: 0s
max_stream_duration:
grpc_timeout_header_max: 0s
cors:
allow_origin_string_match:
- prefix: "*"
allow_methods: GET, PUT, DELETE, POST, OPTIONS
allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
max_age: "1728000"
expose_headers: custom-header-1,grpc-status,grpc-message
http_filters:
- name: envoy.filters.http.grpc_web
- name: envoy.filters.http.cors
- name: envoy.filters.http.router
clusters:
- name: echo_service
connect_timeout: "0.25s"
type: LOGICAL_DNS
# Comment out the following line to test on v6 networks
dns_lookup_family: V4_ONLY
http2_protocol_options: {}
lb_policy: round_robin
load_assignment:
cluster_name: cluster_0
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 0.0.0.0
port_value: 5002If you can connect to the Envoy proxy running on WSL from a Unity app running on the same machine Windows OS but you cannot connect via the Hololens 2, you may need to do the following:
- create a rule in the Windows firewall for the desired port ((see instructions here
- Follow instructions here to forward port
netsh interface portproxy add v4tov4 listenport=<port> listenaddress=0.0.0.0 connectport=<port> connectaddress=<WSL IP>You should now be able to create a gRPC client configured to connect to the IP of the WSL and on the port the Envoy proxy is listening to to call the methods of the C++ proxy gRPC service.
We saw that this was in maintenance mode, but may still work for some time. There are some examples of using gRPC with Unity with it (here, and here). We saw they provide Unity packages.But as we saw, this stack relies on native code, and gRPC is not supported for ARM64/Windows (only ARM/Linux is supported)
Too old, replaced by WCF
Deprecated as well, no longer supported in .NET5, Windows only (source).
Cross platform, open source, it appeared as the alternative to using WCF, but it is not supported by Microsoft as they advise to now use gRPC (source).
Github repo: https://github.com/zeromq
The C# version is a wrapper around the native library (source). But this native library does not seem to be supported for ARM64/Windows (source).
Official site: https://github.com/microsoft/vs-streamjsonrpc Cited as a mean to replace C# remoting in official documentation. It is a cross-platform, .NET portable library that implements the JSON-RPC wire protocol that can use custom serialization, e.g. compact binary format via MessagePack.
UNet is deprecated but there is new set of API available called MLAPI. It can provide high level APIs dedicated to multiplayer gaming, but also lower level API.
A solution like this would probably require to develop a Unity server between the Hololens and the C++ proxy service, which might not be desirable performance-wise.