Skip to content

Commit ffc058a

Browse files
JoeWang1127JoeWang1127
authored
feat: add generate_library.sh without post processing (#1916)
* feat: add * generate gapic and proto folder * refactor utilities * add an action to verify * checkout googleapis-gen * setup repo name * add commit hash of googleapis-gen * change secret * change token * change to git clone * change user name * add input list * include resources folder in main * remove grpc version in `*ServiceGrpc.java` * change destination path * compare generation result with googleapis-gen * fix type in diff command * checkout repo using checkout action * checkout repos as nested repo * sparse checkout googleapis * Revert "sparse checkout googleapis" This reverts commit 3d612f8. * change library * change step name * add a readme * make grpc version optional * make protobuf version optional * checkout master branch, rather than a commit hash * allow snapshot version of generator * download snapshot of generator parent pom * update README * download generator and grpc using mvn * change error message * add maven central mirror * add comments in utilities * add comments * add an integration test * fail fast if no file is found * do not delete google/ * get protobuf version from WORKSPACE * add instructions on download `google/` from googleapis * add comments * update description of `destination_path` * update comments * download dependencies using `curl` * increase download time * remove comment * add samples directory in readme * remove prerequisite about `proto_path` * add explanation in prerequisite * add example to generate showcase * add a comment
1 parent 933cad5 commit ffc058a

File tree

6 files changed

+641
-0
lines changed

6 files changed

+641
-0
lines changed

‎.github/workflows/verify_library_generation.yaml

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
on:
2+
push:
3+
branches:
4+
- main
5+
pull_request:
6+
paths:
7+
- library_generation/**
8+
9+
workflow_dispatch:
10+
name: verify_library_generation_against_googleapis-gen
11+
jobs:
12+
verify_library_generation:
13+
runs-on: ubuntu-22.04
14+
strategy:
15+
matrix:
16+
java: [ 8 ]
17+
steps:
18+
- uses: actions/checkout@v3
19+
- uses: actions/setup-java@v3
20+
with:
21+
java-version: ${{ matrix.java }}
22+
distribution: temurin
23+
cache: maven
24+
- name: Run integration tests
25+
run: |
26+
set -x
27+
library_generation/generate_library_integration_test.sh \
28+
-p google/bigtable/v2 \
29+
-d google-cloud-bigtable-v2-java \
30+
--googleapis_gen_url https://cloud-java-bot:${{ secrets.CLOUD_JAVA_BOT_GITHUB_TOKEN }}@github.com/googleapis/googleapis-gen.git

‎library_generation/README.md

+135
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
# Generate GAPIC Client Library without post-processing
2+
3+
The script, `generate_library.sh`, allows you to generate a GAPIC client library from proto files.
4+
5+
## Environment
6+
7+
Use Linux environment and install java runtime environment (8 or above).
8+
9+
## Prerequisite
10+
Protos referenced by protos in `proto_path` (see `proto_path` below) should be copied to the current
11+
working directory (refers as `$cwd`, a directory contains `generate_library.sh`).
12+
The directory structure should be the same as import statements in protos.
13+
14+
For example, we want to generate from `folder1/folder2/protoA`, so `proto_path`
15+
should be set to `folder1/folder2` (a relative path from `$cwd`).
16+
protoA imports protoB as `folder3/folder4/protoB`, then there should
17+
be `folder3/folder4` (containing protoB) in `$cwd`.
18+
19+
In order to generate a GAPIC library, you need to pull `google/` from [googleapis](https://github.com/googleapis/googleapis)
20+
and put it into `$cwd` since protos in `google/` are likely referenced by
21+
protos from which the library are generated.
22+
23+
## Parameters to run `generate_library.sh`
24+
25+
You need to run the script with the following parameters.
26+
27+
### proto_path
28+
A directory in `$cwd` and copy proto files into it.
29+
The absolute path of `proto_path` is `$cwd/$proto_path`.
30+
31+
Use `-p` or `--proto_path` to specify the value.
32+
33+
### destination_path
34+
A directory within `$cwd`.
35+
This is the path in which the generated library will reside.
36+
The absolute path of `destination_path` is `$cwd/$destination_path`.
37+
38+
Use `-d` or `--destination_path` to specify the value.
39+
40+
Note that you do not need to create `$detination_path` beforehand.
41+
42+
The directory structure of the generated library is
43+
```
44+
$destination_path
45+
|_gapic-*
46+
| |_src
47+
| |_main
48+
| |_java
49+
| |_resources
50+
| |_test
51+
|_grpc-*
52+
| |_src
53+
| |_main
54+
| |_java
55+
|
56+
|_proto-*
57+
| |_src
58+
| |_main
59+
| |_java
60+
| |_proto
61+
|_samples
62+
|_snippets
63+
|_generated
64+
```
65+
You can't build the library as-is since it does not have `pom.xml` or `build.gradle`.
66+
To use the library, copy the generated files to the corresponding directory
67+
of a library repository, e.g., `google-cloud-java`.
68+
69+
### gapic_generator_version
70+
You can find the released version of gapic-generator-java in [maven central](https://repo1.maven.org/maven2/com/google/api/gapic-generator-java/).
71+
72+
Use `--gapic_generator_version` to specify the value.
73+
74+
Note that you can specify a `SNAPSHOT` version as long as you have build the SNAPSHOT of gapic-generator-java in your maven
75+
local repository.
76+
77+
### protobuf_version (optional)
78+
You can find the released version of protobuf in [GitHub](https://github.com/protocolbuffers/protobuf/releases/).
79+
The default value is defined in `gapic-generator-java-pom-parent/pom.xml`.
80+
81+
Use `--protobuf_version` to specify the value.
82+
83+
Note that if specified, the version should be compatible with gapic-generator-java.
84+
85+
### grpc_version (optional)
86+
You can find the released version of grpc in [maven central](https://repo1.maven.org/maven2/io/grpc/protoc-gen-grpc-java/).
87+
The default value is defined in `gapic-generator-java-pom-parent/pom.xml`.
88+
89+
Use `--grpc_version` to specify the value.
90+
91+
Note that if specified, the version should be compatible with gapic-generator-java.
92+
93+
### transport (optional)
94+
One of GAPIC options passed to the generator. The value is either `grpc` or `grpc+rest`.
95+
The default value is `grpc`.
96+
97+
Use `--transport` to specify the value.
98+
99+
### rest_numeric_enums (optional)
100+
One of GAPIC options passed to the generator. The value is either `true` or `false`.
101+
The default value is `true`.
102+
103+
Use `--rest_numeric_enums` to specify the value.
104+
105+
### include_samples (optional)
106+
Whether generates code samples. The value is either `true` or `false`.
107+
The default value is `true`.
108+
109+
Use `--include_samples` to specify the value.
110+
111+
## An example to generate a client library
112+
```
113+
library_generation/generate_library.sh \
114+
-p google/cloud/confidentialcomputing/v1 \
115+
-d google-cloud-confidentialcomputing-v1-java \
116+
--gapic_generator_version 2.24.0 \
117+
--protobuf_version 23.2 \
118+
--grpc_version 1.55.1 \
119+
--transport grpc+rest \
120+
--rest_numeric_enums true \
121+
--include_samples true
122+
```
123+
124+
## An example to generate showcase client
125+
```
126+
library_generation/generate_library.sh \
127+
-p schema/google/showcase/v1beta1 \ # google/ should be in library_generation/.
128+
-d output \
129+
--gapic_generator_version 2.24.0 \
130+
--protobuf_version 23.2 \
131+
--grpc_version 1.55.1 \
132+
--transport grpc+rest \
133+
--rest_numeric_enums false \
134+
--include_samples false
135+
```

‎library_generation/gapic-generator-java-wrapper

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#!/usr/bin/env bash
2+
3+
set -e
4+
5+
# Wrap gapic-generator-java.jar because protoc requires the plugin to be executable.
6+
working_directory=$(dirname "$(readlink -f "$0")")
7+
exec java -classpath "$working_directory/gapic-generator-java-$gapic_generator_version.jar" com.google.api.generator.Main

‎library_generation/generate_library.sh

+160
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
#!/usr/bin/env bash
2+
3+
set -eo pipefail
4+
5+
# parse input parameters
6+
while [[ $# -gt 0 ]]
7+
do
8+
key="$1"
9+
case $key in
10+
-p|--proto_path)
11+
proto_path="$2"
12+
shift
13+
;;
14+
-d|--destination_path)
15+
destination_path="$2"
16+
shift
17+
;;
18+
--gapic_generator_version)
19+
gapic_generator_version="$2"
20+
# export this variable so that it can be used in gapic-generator-java-wrapper.sh
21+
export gapic_generator_version
22+
shift
23+
;;
24+
--protobuf_version)
25+
protobuf_version="$2"
26+
shift
27+
;;
28+
--grpc_version)
29+
grpc_version="$2"
30+
shift
31+
;;
32+
--transport)
33+
transport="$2"
34+
shift
35+
;;
36+
--rest_numeric_enums)
37+
rest_numeric_enums="$2"
38+
shift
39+
;;
40+
--include_samples)
41+
include_samples="$2"
42+
shift
43+
;;
44+
*)
45+
echo "Invalid option: [$1]"
46+
exit 1
47+
;;
48+
esac
49+
shift # past argument or value
50+
done
51+
52+
working_directory=$(dirname "$(readlink -f "$0")")
53+
# source utility functions
54+
cd "$working_directory"
55+
source ./utilities.sh
56+
57+
if [ -z "$protobuf_version" ]; then
58+
protobuf_version=$(get_protobuf_version "$gapic_generator_version")
59+
fi
60+
61+
if [ -z "$grpc_version" ]; then
62+
grpc_version=$(get_grpc_version "$gapic_generator_version")
63+
fi
64+
65+
if [ -z "$transport" ]; then
66+
transport="grpc"
67+
fi
68+
69+
if [ -z "$rest_numeric_enums" ]; then
70+
rest_numeric_enums="true"
71+
fi
72+
73+
if [ -z "$include_samples" ]; then
74+
include_samples="true"
75+
fi
76+
77+
cd "$working_directory"
78+
mkdir -p "$destination_path"
79+
destination_path="$working_directory/$destination_path"
80+
##################### Section 0 #####################
81+
# prepare tooling
82+
#####################################################
83+
cd "$working_directory"
84+
# the order of services entries in gapic_metadata.json is relevant to the
85+
# order of proto file, sort the proto files with respect to their name to
86+
# get a fixed order.
87+
proto_files=$(find "$proto_path" -type f -name "*.proto" | sort)
88+
folder_name=$(extract_folder_name "$destination_path")
89+
# download gapic-generator-java, protobuf and grpc plugin.
90+
download_tools "$gapic_generator_version" "$protobuf_version" "$grpc_version"
91+
##################### Section 1 #####################
92+
# generate grpc-*/
93+
#####################################################
94+
cd "$working_directory"
95+
"$protoc_path"/protoc "--plugin=protoc-gen-rpc-plugin=$working_directory/protoc-gen-grpc-java-$grpc_version-linux-x86_64.exe" \
96+
"--rpc-plugin_out=:$destination_path/java_grpc.jar" \
97+
$proto_files
98+
# unzip java_grpc.jar to grpc-*/src/main/java
99+
unzip_src_files "grpc"
100+
# remove empty files in grpc-*/src/main/java
101+
remove_empty_files "grpc"
102+
# remove grpc version in *ServiceGrpc.java file so the content is identical with bazel build.
103+
remove_grpc_version
104+
###################### Section 2 #####################
105+
## generate gapic-*/, part of proto-*/, samples/
106+
######################################################
107+
cd "$working_directory"
108+
"$protoc_path"/protoc --experimental_allow_proto3_optional \
109+
"--plugin=protoc-gen-java_gapic=$working_directory/gapic-generator-java-wrapper" \
110+
"--java_gapic_out=metadata:$destination_path/java_gapic_srcjar_raw.srcjar.zip" \
111+
"--java_gapic_opt=$(get_gapic_opts)" \
112+
${proto_files} $(search_additional_protos)
113+
114+
unzip -o -q "$destination_path/java_gapic_srcjar_raw.srcjar.zip" -d "$destination_path"
115+
# Sync'\''d to the output file name in Writer.java.
116+
unzip -o -q "$destination_path/temp-codegen.srcjar" -d "$destination_path/java_gapic_srcjar"
117+
# Resource name source files.
118+
proto_dir=$destination_path/java_gapic_srcjar/proto/src/main/java
119+
if [ ! -d "$proto_dir" ]; then
120+
# Some APIs don'\''t have resource name helpers, like BigQuery v2.
121+
# Create an empty file so we can finish building. Gating the resource name rule definition
122+
# on file existences go against Bazel'\''s design patterns, so we'\''ll simply delete all empty
123+
# files during the final packaging process (see java_gapic_pkg.bzl)
124+
mkdir -p "$proto_dir"
125+
touch "$proto_dir"/PlaceholderFile.java
126+
fi
127+
128+
cd "$working_directory"
129+
# move java_gapic_srcjar/src/main to gapic-*/src.
130+
mv_src_files "gapic" "main"
131+
# remove empty files in gapic-*/src/main/java
132+
remove_empty_files "gapic"
133+
# move java_gapic_srcjar/src/test to gapic-*/src
134+
mv_src_files "gapic" "test"
135+
if [ "$include_samples" == "true" ]; then
136+
# move java_gapic_srcjar/samples/snippets to samples/snippets
137+
mv_src_files "samples" "main"
138+
fi
139+
##################### Section 3 #####################
140+
# generate proto-*/
141+
#####################################################
142+
cd "$working_directory"
143+
"$protoc_path"/protoc "--java_out=$destination_path/java_proto.jar" $proto_files
144+
# move java_gapic_srcjar/proto/src/main/java (generated resource name helper class)
145+
# to proto-*/src/main
146+
mv_src_files "proto" "main"
147+
# unzip java_proto.jar to proto-*/src/main/java
148+
unzip_src_files "proto"
149+
# remove empty files in proto-*/src/main/java
150+
remove_empty_files "proto"
151+
# copy proto files to proto-*/src/main/proto
152+
for proto_src in $proto_files; do
153+
mkdir -p "$destination_path/proto-$folder_name/src/main/proto"
154+
cp -f --parents "$proto_src" "$destination_path/proto-$folder_name/src/main/proto"
155+
done
156+
##################### Section 4 #####################
157+
# rm tar files
158+
#####################################################
159+
cd "$destination_path"
160+
rm -rf java_gapic_srcjar java_gapic_srcjar_raw.srcjar.zip java_grpc.jar java_proto.jar temp-codegen.srcjar

0 commit comments

Comments
 (0)