Xdrgen: Proposal to change where language generators are maintained

[leighmcculloch]

Hi all,

I propose we make a change to how language generators are maintained in xdrgen, moving generators out of the xdrgen repo, and maintaining them in closer proximity to the Stellar SDKs that use them.

The move out of generators from xdrgen does not need to happen immediately, or ever, for all generators. But I think we should move to a model where existing generators are considered unmaintained and the next time significant changes are needed to them, they get moved out at that time. New proposed generators can be directed to use the new pattern below.

Why: Generators built-in to xdrgen are not tested in that repo, they're usually tested in their respective SDKs. SDF maintains xdrgen the tool, but not most of the generators within. Generators that are tightly coupled to their SDKs require multi-repo coordination to change. Most generators have a single use case.

This proposal will result in xdrgen itself rarely changing and remaining a stable foundation to build generators on, as the majority of the changes over time to xdrgen have been to the generators not to the core xdrgen library/cli itself.

In the immediate term I propose:

1️⃣ Adding support for "custom generators" to xdrgen. This is a very small change that allows using xdrgen as a library to power any generator written in any other code base:

• Add support for custom XDR generators xdrgen#220

2️⃣ Moving out the Rust generator. The Rust generator will be moved into the rs-stellar-xdr repo. It will serve as an example for how to do a move out, and it will benefit greatly from the move because it is tightly coupled with that repo. The tight coupling has meant we frequently need to coordinate changes to both repos together, and that multi-repo dance will no longer be required:

• Move xdrgen Rust generator into this repository rs-stellar-xdr#449
• Remove Rust generator xdrgen#221

Check out the first Rust PR above to see how a custom generator comes together with xdrgen being used as a library.

---

Backround and context:

Xdrgen is a tool that takes XDR files (e.g. stellar-xdr) and generates code for encoding/decoding that XDR. Xdrgen today has built-in generators for Ruby, Go, JavaScript, Java, Elixir, C#, Rust, and most recently Python, and there are also contributions for Elixir and Scala in open PRs.

None of the generators are tested within the xdrgen repo and they're largely assumed to work.

For the most part how xdrgen gets used is Stellar SDK authors write and maintain their SDK, and write and maintain a generator. The SDK is in a repo they maintain. The XDR generator is in the xdrgen repo. This has the unfortunate bottleneck on SDF engineers reviewing and validating changes to generators or new generators being added, but SDF engineers do not use those generators, the SDK maintainers do. Some generators are tightly coupled to their respective SDKs and so there's little benefit of xdrgen containing the generator for more generic uses.

For generators that get frequent contributions, changes must be coordinated, first making changes to xdrgen, then applying them in the respective SDK. In often cases with the changes not truly being tested until they get applied to the SDK. This process is cumbersome.

---

Feedback appreciated!

cc @overcat (py-stellar-base, java-stellar-sdk)
cc @christian-rogobete (stellar-ios-mac-sdk, stellar_flutter_sdk, stellar-php-sdk)
cc @nebolsin (ruby-stellar-sdk)
cc @cuongph87 @jopmiddelkamp (dotnet-stellar-sdk)
cc @einerzg @elisa6 (stellar-base)
cc @stellar/platform-eng (Go stellar XDR package, JS stellar base module)
cc @stellar/contract-committers (rs-stellar-xdr)

[Shaptic]

Nice idea, I think this makes a lot of sense, particularly since it reduces the absolute craziness of repos you have to follow for the TypeScript case:

• xdrgen creates code compatible with js-xdr
• our fork of dts-xdr runs on top of the generated code to make TypeScript definitions
• stellar-base is the one actually running both of the above pieces
• it then uses said generated code (both JS for itself and TS defs for stellar-sdk)

This would allow us to, if I understand it correctly, move the generator code js-xdr where it belongs: so the generator lives alongside the schema itself (as it should!). I also agree with all of the maintenance points: SDK authors maintain their generators separately, decoupling the need for SDF to review+merge them and adding the ability to actually test them.

My only mild concern is the fact that there's now freaky-deaky Ruby code across all of the SDK repos which might confuse people, but I don't think there's any way around that.

TL;DR: +1

[FrankSzendzielarz]

You could replace it with freaky-deaky C# which is less freaky-deaky than Ruby lol :-D (see my comment below)

[leighmcculloch]

I see there's upvotes and an emoji showing support. Thanks all. I've merged the above changes to implement this pattern with the Rust generator. None of the other existing generators will be removed from xdrgen for now unless their respective maintainers open PRs doing so and moving the implementations into their respective projects.

If we find out this is not a good pattern to use for all generators, it's easy to change direction and keep a generator in xdrgen also.

[FrankSzendzielarz]

Yeah this is the approach I took with the cross platform SDK, by integrating an XDR generator into the project https://github.com/FrankSzendzielarz/SorobanRPCSDK/tree/main/Generator
It really helped because I use the generator to output different variants depending on if the target build is Unity, Native, .NET etc., also extra stuff to support Protobuf wrapping. Also easier for me to modify and iterate on as C# is closer to home.

What might be worth considering if not already mentioned above is a more standard grammar. I use ANTLR.

https://github.com/FrankSzendzielarz/SorobanRPCSDK/tree/main/Generator/XDR/Grammar

Also (the following is not an up to date version) a standard set of XDR serialisation test data could be useful for all SDKs to share, such as something like this (which is only for basic round trip serialise/deserialise tests for all stellar entities) https://github.com/FrankSzendzielarz/SorobanRPCSDK/blob/main/SDKTest/XDR_TEST_CASES.txt

[leighmcculloch]

+1 This is also a good point that folks do not need to build generators with xdrgen and that tech stack. There's a lot of options.