Skip to content

Less test generation #2273

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

Less test generation #2273

wants to merge 5 commits into from

Conversation

jneem
Copy link
Member

@jneem jneem commented Jun 6, 2025

This switches our path-based tests from test_generator's compile-time generation to runtime generation using libtest_mimic.

This has a couple of benefits:

  • new test files are automatically picked up without having to trigger a rebuild some other way (for example, in cargo test; echo '{' > cli/tests/snapshot/inputs/pretty/asdf.ncl; cargo test the second test succeeds because the new snapshot doesn't get picked up
  • modifying existing test files doesn't trigger any recompilation

It also has a couple of downsides:

  • it's different from the usual test registration story, and the custom runner doesn't automatically recognize #[test]-annotated tests.
  • it doesn't capture stdout by default. That's apparently unfixable because of some black magic in the rust standard library. Workarounds include cargo nextest, or just living with messy test output.

While making these changes, I discovered some core integration tests that weren't actually being run, so I moved them to a place where they'd be picked up automatically.

@jneem jneem requested a review from yannham June 6, 2025 11:49
@yannham
Copy link
Member

yannham commented Jun 10, 2025

it's different from the usual test registration story, and the custom runner doesn't automatically recognize #[test]-annotated tests.

What does this mean in practice?

@jneem
Copy link
Member Author

jneem commented Jun 10, 2025

So for example in lsp/nls/tests/main.rs we were running a bunch of file-based snapshot tests but also a couple of special-cased tests. After switching to libtest-mimic, those special-cased tests were no longer registered by the #[test] macro, so I had to either register them manually or move them to another test file that uses the default harness (which is what I did).

@yannham
Copy link
Member

yannham commented Jun 10, 2025

I see. This is a potential footgun, so I suppose it must be properly documented that one should not mix custom-harness-based tests and normal tests. The stdout part isn't a showstopper, but it is a degradation of the quality of life.

@jneem
Copy link
Member Author

jneem commented Jun 10, 2025

Yeah, overall I'm not totally sold on this approach. It looks like there's some interest upstream in improving the testing situation, so maybe we can just wait for that.

@jneem
Copy link
Member Author

jneem commented Jun 17, 2025

I'll close this for now and revisit if the built-in test situation improves.

@jneem jneem closed this Jun 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants