build: publish 'hedera-protobuf-java-api' from services repo

[jjohannes]

Description:

To put and publish the code generated by protoc, we introduce a new module that replaces the Maven build from https://github.com/hashgraph/hedera-protobufs-java. For this we introduce this structure in the hapi folder that follows conventions:

├── hapi
│   ├── hapi                       // "HAPI" sources and configuration of the "HAPI" project moved here one level up
│   │   └── src/main/proto         // services internal 'proto' files
│   └── hedera-protobuf-java-api   // module published as "hedera-protobuf-java-api"
│       └── src/main/proto         // The protobuf files of "hedera-protobuf-java-api"

After this PR, the "protoc" generation runs twice in this repo:

• To produce hedera-protobuf-java-api.jar
• For testing protoc vs pbj in HAPI. As part of feat: move Google protobuf generated code to test fixtures scope #14026 the second "protoc" run in HAPI should be removed and the testing should use what "hedera-protobuf-java-api" produces directly

Related issue(s):

Fixes #17425
Follow up: #14026

Notes for reviewer:

• Is the new structure fine?

• You can test the new publishing as follws

Run:

:hedera-protobuf-java-api:publishToMavenLocal --no-configuration-cache

The you can compare the published .jar with the one published locally

• https://repo1.maven.org/maven2/com/hedera/hashgraph/hedera-protobuf-java-api/0.59.0/
• ~/.m2/repository/com/hedera/hashgraph/hedera-protobuf-java-api/0.60.0-SNAPSHOT/

The general structure is the same. But there are some details different. It looks to me like the "proto" files in this repo and in "hedera-protobuf-java-api" are not completely in sync.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.92%. Comparing base (039f859) to head (1ce9273).

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #17737   +/-   ##
=========================================
  Coverage     68.92%   68.92%           
  Complexity    22986    22986           
=========================================
  Files          2647     2647           
  Lines         99445    99445           
  Branches      10285    10285           
=========================================
+ Hits          68544    68546    +2     
+ Misses        27023    27021    -2     
  Partials       3878     3878           

Files with missing lines	Coverage Δ
.../src/main/java/com/hedera/hapi/util/HapiUtils.java	59.66% <ø> (ø)
...m/hedera/hapi/util/UnknownHederaFunctionality.java	100.00% <ø> (ø)

... and 8 files with indirect coverage changes

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.00% (target: -1.00%)	✅ ∅

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (039f859)	99228	72271	72.83%
Head commit (1ce9273)	99228 (+0)	72273 (+2)	72.84% (+0.00%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#17737)	0	0	∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

[thomas-swirlds-labs]

The general structure is the same. But there are some details different. It looks to me like the "proto" files in this repo and in "hedera-protobuf-java-api" are not completely in sync.

Yes this is correct. So in the current process of deploying hedera-protobufs-java to Maven Central the release manager performs a couple of extra manual steps beforehand :

1. Sync'ing up the .proto files in hedera-protobufs with those in hedera-services for that release
2. Comparing and sync'ing .proto files in hedera-protobufs with those in hedera-protobuf-java-api

[andrewb1269hg]

platform-ci files look good!

[thomas-swirlds-labs]

LGTM. Thanks @jjohannes

[anthony-swirldslabs]

Tools and Libraries don't really own any of the .proto files, but the changes look harmless to me, so approving.

I'd like @jsync-swirlds to take a look as he's done a lot of work on these .proto models in the past.

[andrewb1269hg]

Approve build and settings gradle files.

[jsync-swirlds]

This structure seems off at first glance (not least because we end up with a path like hapi/hapi/src/...).

Could we rename the second hapi to proto? At least that avoids the duplicated folder name.

As to the published outputs, I think we need to consider this a bit more broadly in terms of making hiero-consensus-node the proper single-source for the proto files that form the formal Hiero API specification.

To this end, a few points worth discussing:

• The protocol buffers from this repo are used for a lot more than a Java API. Every SDK, every communicating project, everyone uses these protocol buffer definitions. The Java API is, at best, used by the Java SDK and a couple other projects, but not, by any means, all of them. Right now the majority of usage is via the hedera-protobufs repo. If we truly want to make hiero-consensus-node the sole source of the API definitions, then we should probably ensure they are readily consumed by many projects without having to clone the whole consensus node codebase and run a complex custom process to extract the API definitions.
• What we probably need is two or three published outputs.
  • One to publish the internal Java API (PBJ generated) used by the consensus node itself
  • Another to publish the proto files uncompiled as a jar or similar consumable archive, suitable for use by the many SDKs and other projects (e.g. Mirror Node, Block Node, etc...)
  • Possibly a third output, the protoc generated java files for use by projects that currently depend on the jar produced by hedera-protobuf-java-api (which always seems to be more out of sync than we might prefer). If we can deprecate and remove this legacy API copy process, that would be even better.

[jjohannes]

Thank you for the detailed feedback @jsync-swirlds

Regarding the discussion point (source of truth for protobufs)

I would propose to regard this PR as a solution to replace the publishing of Generated Protobuf Java Code which currently lives in https://github.com/hashgraph/hedera-protobufs-java (note the -java in the end of the name). It contains itself a copy of the protobuf files. This is published as hedera-protobuf-java-api.jar to Maven Central. I am not aware that we use this ourselves. Even the Java SDK is generating protobuf code itself for the protos they are interested in instead of using that Jar.
What to do about the "source of truth" for ptotobufs, which are currently modified here and then synced back to https://github.com/hashgraph/hedera-protobufs (without -java) should be a separate discussion point outside of this PR.
But yes, for that I think it would be an interesting consideration to publish only the raw proto files and let the SDKs consume that. If it is sufficient for the SDKs to use published versions. Right now we have:

1. hapi.jar (contains PBJ generated code + proto files)
2. hedera-protobuf-java-api.jar (contains Protobuf generated code + proto files)

With this PR, this will not change (only the publishing of (2) is moved here instead of maintaining it in a separate repo). So this PR is hopefully a good base for further adjustments in this area.

Regarding naming

I left the second level folder names (names of the Modules) as they are to not change the publishing coordinates and not create naming confusion. I would prefer to leave them as they are. We may consider renames later, if we should change publishing coordinates with the Hiero rename anyway.

• hapi/hapi
  published to: https://repo1.maven.org/maven2/com/hedera/hashgraph/hapi/
• hapi/hedera-protobuf-java-api
  published to: https://repo1.maven.org/maven2/com/hedera/hashgraph/hedera-protobuf-java-api/

[andrewb1269hg]

Review and approve build and settings gradle files.