Future of schema generators

[domoritz]

This is a discussion about the future of Typescript JSON schema generators.

TL;DR

• Use https://github.com/vega/ts-json-schema-generator/ if you want a robust but opinionated schema generator. It has good support for aliases and it continuously updated. This is probably the right option for most people.
• Use https://github.com/YousefED/typescript-json-schema if you want a less robust but hackable schema generator.

[domoritz]

From @HoldYourWaffle

I have created issues for most of the problems I encountered in YousefED/typescript-json-schema. I had to switch to that module because conditional types (particularily Omit) are not supported yet.

But there's something else I want to discuss. There are currently 3 modules that accomplish basically the same goal (at least that I know of), and they all have a lot of issues.
vega/ts-json-schema-generator seems to be the best overall solution, featuring clean(er) code and proper type alias support. However, it doesn't support conditional types (#100), which means Omit, Pick, Exclude etc. arent' supported either (#71, #93). This makes this module completely unusable for me because I heavily use these language features. YousefED/typescript-json-schema does support these constructs, but it contains a lot of bugs (I have reported 8 so far). There's also xiag-ag/typescript-to-json-schema, but I have no idea what the difference is between this module and it's 'ancestor' (the README only says that this is an 'extended' version).

I see 3 possible solutions for this forkmania (though there could be more):

Fix YousefED/typescript-json-schema. This can sortof be seen as the 'default' option, as it would leave the current situation mostly untouched. This isn't my preferred solution because the architecture of vega/ts-json-schema-generator looks a lot better and it doesn't really solve the forkmania.

Write a new generator from scratch, using the knowledge and experience from the other three. If we were to do this we'd of course have the amazing power of hindsight and shared knowledge, which would allow us to write a clean, well-designed and future proof generator. However, this would be very time-consuming and since vega/ts-json-schema-generator is already very well designed I don't see much reason to do this.

Merge all the good parts into vega/ts-json-schema-generator and 'opening it up' for general usage. I think this would be the best option because it will actually fix the fork issue without consuming a lot of time and effort. I think this module currently has the best/cleanest code and I don't see a reason for using one of the others if we increased flexibility and supported more use cases than what vega is doing.

If you're interested I can write up a more detailed overview in a couple of hours. No matter what option you/we choose I'd love to help/fix/develop/maintain.

[HoldYourWaffle]

Good to see there's interest in my proposal! Perhaps it would be good to pin this issue so more people will see this and hopefully voice their opinion on the matter?

[domoritz]

Thank you for the comments. I will use this issue to explain some of the different philosophies behind the different libraries.

The goal of all of these libraries is to convert Typescript to JSON schema. There are different approaches to achieving this and also different interpretations that you can follow. One fundamental issue is that JSON schema and Types are not equivalent. Some things are not expressible in the other.

YousefED/typescript-json-schema

This was the original schema generator that I picked for my main use case, which is Vega-Lite. The philosophy of this library was to be flexible and configurable for different use cases. This means that some configurations work better than others because optimizing all of them is complex.

It worked fairly well once I extended it a bit to deal with some of the more complicated types we use. However, I constantly ran into issues with getting meaningful properties that correspond to aliased types. The fundamental problem is that this library uses the type hierarchy, and not the AST to create the JSON schema. Therefore I went on to look for a different library, which was xiag-ag/typescript-to-json-schema.

I stopped active development but occasionally review PRs and make releases. I consider this library to be in maintenance mode.

xiag-ag/typescript-to-json-schema

This library was mostly written by @mrix and uses the AST to create the schema. It is also much more modular. Rather than having one big file, there are separate modules for parsing and code generation for all different types of AST nodes and types. I had to extend it significantly to make it work for Vega-Lite. That's what vega/ts-json-schema-generator is.

vega/ts-json-schema-generator

This is the extended version of xiag-ag/typescript-to-json-schema. Besides extending the supported AST nodes, I also changed some of the behavior because it was not correct for some of my use cases. Some of these improvements have since been ported back into the original library but not all.

Overall, this library is much more robust than YousefED/typescript-json-schema and produces generally better schemas. It is also much more opinionated. I don't want to have options but instead, make particular design decisions and bake them into the code.

Even though this library still has some missing functionalities, it works in production for Vega-Lite, which is a complex piece of TypeScript. I keep fixing this library if it is necessary for Vega-Lite but otherwise don't have the cycles to do anything else.

I am happy to review PRs if they have sufficient tests, don't introduce options, and provide useful additions (I will reject support for functions because there is no clean mapping to JSON schemas).

I consider this library to be in active development and would be more than thrilled to have people help with it.

Conclusion

I have considered writing a new generator based on the things I have learned. Ideally, it would be a bit leaner than vega/ts-json-schema-generator and not have as many files. However, I don't have the cycles to do it and I'm not convinced that it will be much cleaner. Overall, I think investing resources into vega/ts-json-schema-generator would be the best way forward. I am happy to review PRs and make timely releases for this library. However, my number one priority is to support Vega-Lite and anything that is in conflict with that goal will be rejected (I don't see this as an issue, though).

Let me know what you think.

[HoldYourWaffle]

Since your comment is pretty big I'm just going to respond per section.

One fundamental issue is that JSON schema and Types are not equivalent. Some things are not expressible in the other.

This is of course true. I think the best way to handle this would be to have a section in the README that clearly lists all constructs that are not or only partially supported. It's annoying to discover something you need is unsupported, but discovering it after you've already started using an automated generator setup is way worse (I can unfortunately speak from experience).

Maybe we should also look into a way to manually override generated the generated schema in a fluent way. This way any shortcomings of the library can be manually filled in or corrected. In my own projects I've been using a script that manually changes the generated JSON object but this is very inflexible and error prone. I haven't really thought about how to implement such a mechanism, perhaps a JSdoc annotation with a JSON pointer could be something? I'll think about it.

This means that some configurations work better than others because optimizing all of them is complex.

I'm not sure I understand what you mean by this. Are you trying to say that more options → more complexity → hard to get working correctly? I agree that having more options might increase complexity, but there should always be a sensible default behavior. Having more options to override common sense because it's assumptions doesn't match with your use case is a good thing in my opinion.

It is also much more opinionated. I don't want to have options but instead, make particular design decisions and bake them into the code.

I'm not sure I agree with you on this. Having sensible defaults is always a good thing, and making some assumptions when designing something like this is necessary, but I don't see how this would hinder adding options. Could you give an example of an option you've rejected/don't want to add? I'm probably just misunderstanding what you're saying.

I keep fixing this library if it is necessary for Vega-Lite but otherwise don't have the cycles to do anything else.
I am happy to review PRs if they have sufficient tests, don't introduce options, and provide useful additions (I will reject support for functions because there is no clean mapping to JSON schemas).
I consider this library to be in active development and would be more than thrilled to have people help with it.

I'd love to help you maintain this project if you don't have the time for it! Again I'm not sure why you'd want to reject new options, could you clarify what you mean by this?
And out of curiosity: why do people want to send functions over JSON? There's no function type in the JSON spec, nor can I think of a usecase where one would want to. If you really wanted to put functions in your JSON you can use Function.toString with eval on the other side, but this is pretty unsafe in almost all cases.

Ideally, it would be a bit leaner than vega/ts-json-schema-generator and not have as many files. However, I don't have the cycles to do it and I'm not convinced that it will be much cleaner.

This is the main reason why I think a new generator isn't the best option. Is there a reason why we can't make vega/ts-json-schema-generator leaner without rewriting the whole thing? Also, what's wrong with having more files? It makes it a lot easier to find what you're looking for, as well as leaving less room for weird global state anti-patterns.

Overall, I think investing resources into vega/ts-json-schema-generator would be the best way forward. I am happy to review PRs and make timely releases for this library. However, my number one priority is to support Vega-Lite and anything that is in conflict with that goal will be rejected (I don't see this as an issue, though).

I also think this is be the best way forward. I don't see a reason why there would be conflicts with Vega, since the goal of a general purpose library is to support most (if not all) usecases, Vega included of course.

If we decide to adopt this strategy there's one more issue remaining: uniting the modules/forks to fix the current forkmania. I think YousefED/typescript-json-schema could just be deprecated with a nice forward to this module (as soon as it's ready of course, mainly looking at conditional types).

xiag-ag/typescript-to-json-schema is a different story though. I found this PR by you that aims to merge the 2 repositories together, but as you already know there hasn't been any response in 2 years. It seems like @mrix has disappeared from the community, which of course doesn't help our case.

The main reason why I want the forks to be united is that the current situation is really confusing. Last week I basically went like this:

• typescript-json-schema, has the most downloads and looks pretty solid
• typescript-to-json-schema, mentions that it uses a superior approach in the README which apparently fixes type aliases? Seems like this is the best choice then
• ts-json-schema-generator, is a fork of the previous superior one? What's the difference? There's like 200+ additional commits so I think it's better? Why are all these names so similar?!
  And it only got worse when I discovered that you have contributed to both and recommend both to users of the other.

Removing YousefED/typescript-json-schema from the equation would help a lot. We may never get a response from @mrix, but we can remove this repository's forked status or add a clear explanation in the README on what the differences are and why you should probably use this module instead of the upstream one.

On a completely different note, maybe it's a good idea to create an issue in YousefED/typescript-json-schema referencing this issue and pin it there too. Since that module has more users we're probably going to get more responses then.

[domoritz]

One fundamental issue is that JSON schema and Types are not equivalent. Some things are not expressible in the other.

This is of course true. I think the best way to handle this would be to have a section in the README that clearly lists all constructs that are not or only partially supported. It's annoying to discover something you need is unsupported, but discovering it after you've already started using an automated generator setup is way worse (I can unfortunately speak from experience).

The issue here really is JSON schema and not Typescript. I have found good ways around missing things in typescript such as maxLength but the other way around it much messier. For example, JSON schema neither supports union types or inheritance properly.

This means that some configurations work better than others because optimizing all of them is complex.

I'm not sure I understand what you mean by this. Are you trying to say that more options → more complexity → hard to get working correctly? I agree that having more options might increase complexity, but there should always be a sensible default behavior. Having more options to override common sense because it's assumptions doesn't match with your use case is a good thing in my opinion.

More options increase the number of paths through the code and make it harder to get right. I am definitely against adding more code paths just to support another use case. Instead, the defaults should be good.

As an example, including or not including aliases or not using a top-level reference have implications on many parts of the code. I can speak from experience that not having the config options avoided a bunch of headaches.

I am not against the ability to configure things that have only implications on localized pieces of the code.

And out of curiosity: why do people want to send functions over JSON? There's no function type in the JSON spec, nor can I think of a usecase where one would want to.

I agree and still there we got PRs and issues for it: https://github.com/YousefED/typescript-json-schema/issues?utf8=%E2%9C%93&q=functions+

Is there a reason why we can't make vega/ts-json-schema-generator leaner without rewriting the whole thing?

No. I think the current design is good enough and I don't see any reason to rewrite.

I think YousefED/typescript-json-schema could just be deprecated with a nice forward to this module

It's already in maintenance mode and I think that's what it should be. A forward link sounds good to me but then I want support with issues that people report ;-)

xiag-ag/typescript-to-json-schema has a few features that we should get working in this fork. See #63

On a completely different note, maybe it's a good idea to create an issue in YousefED/typescript-json-schema referencing this issue and pin it there too.

Go ahead. I will pin it. I already added a note to https://github.com/YousefED/typescript-json-schema#background. Before we can deprecate the other library, we need support for conditionals here.

[HoldYourWaffle]

The issue here really is JSON schema and not Typescript. I have found good ways around missing things in typescript such as maxLength but the other way around it much messier. For example, JSON schema neither supports union types or inheritance properly.

You really did a good job expressing stuff like minLength! It's very fluent and it just makes a lot of sense. It's also self-documenting by definition, which is always a good thing.
I'm not sure why union types would be a problem, isn't that just oneOf? Inheritance is a common issue with JSON schema, but since this is an automated tool it's probably not that bad to have some duplication in the output schema (perhaps adding a description field with the original information would be good for clarity/readability?).

It should also be noted that without additionalProperties: false it's perfectly possible to express inheritance using allOf, so if we really wanted to support a form of inheritance preservance this could be an option, but I think this will get needlessly complex very quickly. Since this is an issue with the JSON schema spec itself and not with this module I think a clear explanation on why this isn't possible in the README would suffice for now.

More options increase the number of paths through the code and make it harder to get right. I am definitely against adding more code paths just to support another use case. Instead, the defaults should be good.
As an example, including or not including aliases or not using a top-level reference have implications on many parts of the code. I can speak from experience that not having the config options avoided a bunch of headaches.
I am not against the ability to configure things that have only implications on localized pieces of the code.

That makes a lot of sense. So something like --strictTuples (is it the default yet?) or --strictNullChecks (however misleading it may be) would be fine? I agree that configuring the overall shape of the schema will get messy very quickly, but as far as I can see there's no reason why options for individual constructs should be rejected.

I think YousefED/typescript-json-schema could just be deprecated with a nice forward to this module

It's already in maintenance mode and I think that's what it should be. A forward link sounds good to me but then I want support with issues that people report ;-)

That makes sense, but is there really a reason why someone would want to use the other module once we include all missing feature here? It's always good to keep providing support for something, but is it really worth the effort if this were to be a (practically) drop-in replacement?

xiag-ag/typescript-to-json-schema has a few features that we should get working in this fork. See #63

I'd love to help, but I can't figure out what new features we're missing since there are so many changes in the vega version. If you can give me some kind of list I'd be more than happy to take a look.

Go ahead. I will pin it.

Done. I also created an issue in @XriM's repository in case there are more lost souls like I was.

[domoritz]

I'm not sure why union types would be a problem, isn't that just oneOf

I meant intersection types. allOf does not work because of additionalProperties: false as you noted in inheritance as well.

So something like --strictTuples (is it the default yet?) or --strictNullChecks (however misleading it may be) would be fine?

Yep. I think my request to keep the number of code paths low is reasonable and I think you agree.

That makes sense, but is there really a reason why someone would want to use the other module once we include all missing feature here?

In the future, yes.

I'd love to help, but I can't figure out what new features we're missing since there are so many changes in the vega version. If you can give me some kind of list I'd be more than happy to take a look.

See master...xiag-ag:master.

[HoldYourWaffle]

Yep. I think my request to keep the number of code paths low is reasonable and I think you agree.

Of course! I was just wondering where your "line" was on what's too complicated, glad to hear it's in a very reasonable place.

See master...xiag-ag:master.

I tried to look through it, but I fear I'm just not well versed enough in the codebase to know what changed, what hasn't been done here already and what is even applicable to our version. Maybe we could copy over the tests that were added, see which ones fail and go from there? I'd love to try it but I'll have to figure out the test infrastructure first, which is of course going to take some time.

I meant intersection types. allOf does not work because of additionalProperties: false as you noted in inheritance as well.

The more I use JSON schema the more I think "How have they not solved this yet?". I think from a spec perspective there are 2 logical solutions:

1. Ignore additionalProperties in an allOf clause. I can't think of a use-case where this would be actual useful behavior.
2. Allow additional keys on a $ref "schema" to supplement/override the reference.

These "ideas" aren't very useful from a generator perspective of course since no validator supports them. The only solution I see is to duplicate & merge the inherited and inheriting schemas but this would probably get really messy really quickly (both in the code and in the output). Maybe adding a description to the frankensteined output would help? I feel like there should be a better solution though...
How is this currently handled?