Skip to content

7904061: Enable JMH integration tests to be executed against other impls #163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

galderz
Copy link
Contributor

@galderz galderz commented Jul 8, 2025

NOTE: This PR should be marked as draft because not all tests have been migrated to this setup. I've migrated the ones in the root test package to get a feel on how things would work and get feedback on whether this approach is fine for the JMH community.

This is the last in a series of PRs extending JMH to benchmark Java code running as GraalVM native images more easily.

In this PR:

  • We enable jmh-core-it test jar to be created so that JMH extensions can consume it.
  • We create a test RunnerFactory interface with a method called createRunner that by default creates JMH's Runner instance. JMH extensions that want to run the same test with a different runner would apply changes like these. Here the JMH extension extends the default RunnerFactory interface, and extends individual test classes to make them implement the extended runner factory interface.

The result of both these changes enables JMH tests to be executed with the JMH extension:

[INFO] --- failsafe:3.5.2:integration-test (default) @ fibula-jmh-it ---
[INFO] Using auto detected provider org.apache.maven.surefire.junit4.JUnit4Provider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.mendrugo.fibula.jmh.it.SingleShotTest
# JMH version: fibula:999-SNAPSHOT
# VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1
# *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong.
# VM invoker: target/benchmarks
...
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.536 s -- in org.mendrugo.fibula.jmh.it.SingleShotTest
[INFO] Running org.mendrugo.fibula.jmh.it.IterationCountCmdTest
# JMH version: fibula:999-SNAPSHOT
# VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1
# *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong.
# VM invoker: target/benchmarks
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.712 s -- in org.mendrugo.fibula.jmh.it.IterationCountCmdTest
[INFO] Running org.mendrugo.fibula.jmh.it.IterationCountAnnTest
# JMH version: fibula:999-SNAPSHOT
# VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1
# *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong.
# VM invoker: target/benchmarks
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.345 s -- in org.mendrugo.fibula.jmh.it.IterationCountAnnTest
[INFO] Running org.mendrugo.fibula.jmh.it.WarmupIterationCountAnnTest
# JMH version: fibula:999-SNAPSHOT
# VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1
# *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong.
# VM invoker: target/benchmarks
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.751 s -- in org.mendrugo.fibula.jmh.it.WarmupIterationCountAnnTest
[INFO] Running org.mendrugo.fibula.jmh.it.WarmupIterationCountCmdTest
# JMH version: fibula:999-SNAPSHOT
# VM version: JDK 24.0.1, Substrate VM, GraalVM CE 24.0.1+9.1
# *** WARNING: This VM is not supported by JMH. The produced benchmark data can be completely wrong.
# VM invoker: target/benchmarks
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.756 s -- in org.mendrugo.fibula.jmh.it.WarmupIterationCountCmdTest
[INFO] Running org.mendrugo.fibula.jmh.it.StackTraceInThrowableTest
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 s -- in org.mendrugo.fibula.jmh.it.StackTraceInThrowableTest
[INFO] 
[INFO] Results:
[INFO] 
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 1

Note that above we can observe that one of the JMH tests is disabled. It could happen that several JMH tests are not applicable to native image, e.g. perfasm tests. In the usage changes above we can observe how JUnit's @Ignore annotation can be easily used to disable particular tests as needed. In the particular case of PrintFlags, the JMH extension could filter this flag and not pass on (there are already such cases) so with that modification the test could pass but I left it as is so that test exclusion can be demonstrated for this PR.

Here is the PR list for reference:

  1. Make OutputFormatAdapter public
  2. Enable BenchmarkParams construction to be overriden
  3. Enable alternative Runner instantiation and Main reuse
  4. Enable profiler classes to be reused outside of JMH
  5. Enable JMH integration tests to be executed against other impls

Progress

  • Change must not contain extraneous whitespace
  • Change must be properly reviewed (1 review required, with at least 1 Committer)

Issue

  • CODETOOLS-7904061: Enable JMH integration tests to be executed against other impls (Enhancement - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jmh.git pull/163/head:pull/163
$ git checkout pull/163

Update a local copy of the PR:
$ git checkout pull/163
$ git pull https://git.openjdk.org/jmh.git pull/163/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 163

View PR using the GUI difftool:
$ git pr show -t 163

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jmh/pull/163.diff

Using Webrev

Link to Webrev Comment

@bridgekeeper
Copy link

bridgekeeper bot commented Jul 8, 2025

👋 Welcome back galder! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk
Copy link

openjdk bot commented Jul 8, 2025

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

@mlbridge
Copy link

mlbridge bot commented Jul 8, 2025

Webrevs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
rfr Pull request is ready for review
Development

Successfully merging this pull request may close these issues.

1 participant