output multiple files, in place?

[srsudar]

I'm working on a big project. It compiles using TSC to .d.ts files (see #2215).

However, when trying to run the ts-json-schema-generator on the .d.ts files, I am regularly getting errors that I seem to have hit the max string size:

<snip>/ts-json-schema-generator/ts-json-schema-generator.ts:82
    const schemaString = (config.minify ? stringify(schema) : stringify(schema, null, 2)) as string;
                                                              ^


RangeError: Invalid string length
    at stringify (<anonymous>)
    at stableStringify (<snip>/ts-json-schema-generator/ts-json-schema-generator.ts:82:63)
    at Object.<anonymous> (<snip>/ts-json-schema-generator/ts-json-schema-generator.ts:108:1)
    at Module._compile (node:internal/modules/cjs/loader:1364:14)
    at Object.transformer (<snip>/ts-json-schema-generator/node_modules/tsx/dist/register-DCnOAxY2.cjs:2:1186)
    at Module.load (node:internal/modules/cjs/loader:1203:32)
    at Module._load (node:internal/modules/cjs/loader:1019:12)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:203:29)
    at ModuleJob.run (node:internal/modules/esm/module_job:195:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:337:24)

Is there a way to instead of putting the schema in a single large file, instead have it generate smaller schema files? If I had Foo.d.ts and Bar.d.ts, eg, I'd be happy to have Foo.schema.json and Bar.schema.json, and have them reference each other using relative paths.

Apologies if this is already supported somehow in a way that I'm missing.

[domoritz]

Whoa, I have not had that size. The biggest I worked on is Mosaic which is 8mb of JSON but apparently not big enough to trigger this 😅 . Anyway, haven't seen this before and have no fix. Happy to review a pull request if you find a way to improve this.

[srsudar]

The same thing happens in this library if I don't turn off stable sorting:

BridgeAR/safe-stable-stringify#12

I think the underlying problem is likely that some of our our types are poorly defined, but I'm struggling to figure out which it is. I'll let you know if I come up with anything.

[domoritz]

We could not stable stringify and just use JSON.stringify. The reason for stable stringify for me is that I wanted to have more reproducible schema generation with minimal diffs.

Also, Hi. We overlapped during our PhDs at UW.

[srsudar]

If I use --unstable I get the first error. Without that flag I get it in the other lib. That just made me think it's likely a hard limit with the approach (and our weird types).

And hello! lol I definitely recognize you now. What a small world!

[BridgeAR]

@srsudar strings have a hard limit hardcoded in V8. You may report it there, since the limit is not defined by the spec. It would likely require a lot of changes to improve it and it will only happen in case enough people report it to V8.

It could theoretically be cut off when reaching that point, but that will likely prevent the output from being JSON compatible. An alternative is to cut off huge strings. That is possible, if you as a user want to define these.

https://github.com/BridgeAR/safe-stable-stringify also provides options to limit the amount of properties to do something similar.

[domoritz]

Could you use bun which use jscore instead?

[BridgeAR]

Without looking into it further: streaming the JSON might be an option: https://www.npmjs.com/package/stream-json

It does feel like a workaround though and another format is likely what would be more suitable.

[srsudar]

I definitely think whatever my codebase is doing to generate strings that big is something I don't want to be doing. I'm going to try and resolve this in another way and will report back!