Docker Support for White box Testing

[naveensabavath]

hi @arcuri82 ,

Subject: Inquiry About Docker Support for White-Box Testing in EvoMaster

Hi @arcuri82 ,

I came across the EvoMaster documentation, which states that running white-box tests from Docker is currently not supported and is discouraged due to hostname and ephemeral port resolution issues.

I wanted to ask if there has been any progress on this, or if support for white-box testing in Docker is planned for future development.

Statement in Documentation :

IMPORTANT

For white-box testing, running from Docker is currently not supported. As such, it is discouraged. You can try it, but likely you will get major issues with hostname and ephemeral port resolutions, as well as issues with external dependency mocks.

[arcuri82]

Docker support for white-box testing will be provided in the future, but it is not a major priority now. there are too many other tasks on our backlog that have higher priority :(

[naveensabavath]

Thank You for your response @arcuri82

After ..Either black box or white box testing generates test classes ...Should we validate or Execute them If yes how ?

[arcuri82]

EM generates test cases, with the goal of maximing code coverage when executed and finding faults. how you use these test cases is up to you

[naveensabavath]

hi @arcuri82 , is it needed that EmbeddedEvomasterController class should be written in only src/test/java folder ?

while maven building this EmbeddedEvomasterController.java class is not being added on fat jar

But it did when i added that class in main folder ..for ex: src/main/java

In intelli J it is running fine without errors but while running cmd line please check below

command :

java -Djdk.attach.allowAttachSelf=true --add-opens java.base/java.util.regex=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED -jar target/test-jar.jar

but while running with command line getting below error

15:18:27.892 [main] INFO org.testcontainers.utility.ImageNameSubstitutor -- Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
WARNING: A Java agent has been loaded dynamically (/tmp/javaagent.org.evomaster.client.java.instrumentation.InstrumentingAgent17940319427532506921.jar)
WARNING: If a serviceability tool is in use, please run with -XX:+EnableDynamicAgentLoading to hide this warning
WARNING: If a serviceability tool is not in use, please run with -Djdk.instrument.traceUsage for more information
WARNING: Dynamic loading of agents will be disallowed by default in a future release
Exception in thread "Attach Listener" java.lang.ClassNotFoundException: org.evomaster.client.java.instrumentation.InstrumentingAgent
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:504)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:582)
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message Outstanding error when calling method in invokeJavaAgentMainMethod at ./src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 627
*** java.lang.instrument ASSERTION FAILED ***: "success" with message invokeJavaAgentMainMethod failed at ./src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 466
*** java.lang.instrument ASSERTION FAILED ***: "success" with message startJavaAgent failed at ./src/java.instrument/share/native/libinstrument/InvocationAdapter.c line: 444
Agent failed to start!
Exception in thread "main" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:118)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:95)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: java.lang.RuntimeException:
Failed to apply bytecode instrumentation with JavaAgent.
If you are using JDK 11 or above, are you sure you added the following JVM option?
-Djdk.attach.allowAttachSelf=true
Also, if you are using JDK 17 or above, you also need the following:
--add-opens java.base/java.util.regex=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED
This can also be set globally with the JDK_JAVA_OPTIONS environment variable.
See documentation at https://github.com/EMResearch/EvoMaster/blob/master/docs/jdks.md
at org.evomaster.client.java.controller.InstrumentedSutStarter.loadAgent(InstrumentedSutStarter.java:48)
at org.evomaster.client.java.controller.InstrumentedSutStarter.(InstrumentedSutStarter.java:71)
at com.aidtaas.mobius.content.services.EmbeddedEvoMasterController.main(EmbeddedEvoMasterController.java:177)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
... 5 more
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at shaded.com.ea.agentloader.AgentLoader$1.loadAgent(AgentLoader.java:244)
at shaded.com.ea.agentloader.AgentLoader.loadAgent(AgentLoader.java:77)
at shaded.com.ea.agentloader.AgentLoader.loadAgentClass(AgentLoader.java:148)
at shaded.com.ea.agentloader.AgentLoader.loadAgentClass(AgentLoader.java:102)
at org.evomaster.client.java.controller.InstrumentedSutStarter.loadAgent(InstrumentedSutStarter.java:46)
... 8 more
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:118)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at shaded.com.ea.agentloader.AgentLoader$1.loadAgent(AgentLoader.java:240)
... 12 more
Caused by: java.lang.RuntimeException: Can't attach to this jvm. Add -javaagent:/tmp/javaagent.org.evomaster.client.java.instrumentation.InstrumentingAgent17940319427532506921.jar to the commandline
at shaded.com.ea.agentloader.AgentLoaderHotSpot.loadAgent(AgentLoaderHotSpot.java:74)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
... 14 more
Caused by: com.sun.tools.attach.AgentInitializationException: Agent JAR loaded but agent failed to initialize
at jdk.attach/sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:170)
at shaded.com.ea.agentloader.AgentLoaderHotSpot.loadAgent(AgentLoaderHotSpot.java:65)
... 15 more

[arcuri82]

@naveensabavath what you are asking is not really directly related to EM, but to how to setup JUnit tests in a Maven/Java project. That is independent of whether they are created by EM or written manually.
still I guess it could be useful to provide some updated video-tutorial covering all aspects of EM, including how the generated JUnit tests could be used in a Maven/Gradle project

[arcuri82]

thx. we keep "feature requests" open until we implement them, or decide that for sure we would not