-
Notifications
You must be signed in to change notification settings - Fork 94
Only one testcase was generated for 1 API (possible issue with string constraints) #1203
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
hi @ytfrank , thanks for reporting this issue. I think I know what is the problem. Probably it is related to:
we can handle constraints on length, and constraints on patterns, but not both at same time :( |
Unfortunately, still only 1 testcase was generated even after removing all the constraints about length and patterns. Here is the new api doc:
|
@ytfrank thanks for answering the survey removing the constraints on schema would make things worse, as I assume that the API is still validating those constraints internally. regarding regex handling, there is currently a bias in short strings... that might struggle to reach a 16-char long string (the handling of regex in EM should be improved). I would suggest to put back all the constraints, and try to make sure all length-related constraints are expressed in the pattern, eg, |
Still only saved 1 test, although many tests were evaluated. Is it expected that only one testcase is generated when the response is similar? Here is the info: 2025-03-24 17:27:59 [http-nio-8080-exec-2] INFO c.e.demo.controller.DemoController - ------Return resp:{}{"data":"Udq+Pxj63RniizY9hJ+jiHmkUbo5fxygW9JTWuLdwPGOlMX3lflxXNoKg9O8DiPl","key":"bJAhDO2ZBtIW1oe0vBvMCV2BESMqnejfXBhGEkiSylpkLq3NhWXFZRIlWJAY2i+Na9yTBQRSReGpXPZP48VdxF9M58Te5OD6ep0rlfKADitj7IEJh/oTkGRX+08QfDmcdDnbr2po4odJUW7NRxn3rckuOWM42GFLb5GXSNpyoVc4laOJRASuM1ObxdyPQF1cTgp7Xn6kEzWWKv2a9PU8RZ13ynTOJ/AEPYKdK3Z6itEM2SJtRFeQKlcH5IFSwFFZ19c4BhRNbVWIJvAXundvsWQR6HN3PwEGGIQkbOJuWJK9q+jeJ15FGuRBSNgdCunSwHi/w1KE26nQNEG9kp6Szw==","sign":"BP4OHeOEHshegDft+JFp9PMG+Z1hXqYXT8bUHks1oa3zHUiOWrFxojERf+/Fdmtvq0/K5YvaQ+B3D6rWShyKuDJfU0ltNHebHmrFyZB5nnGzEI4kZOXYZnykVxroh9/4KbXJT4nuvpaH07qo0fIaC0WPFXfJr8cFVN5jiZnhqS8A8qrkfBE3ezo0EazKz51HTL3xE9jzQSH+kV1oquig/hUazL7alGJODLcn+u5n2sJYqxsRJoeczbDLBBV3lMne7bzs+cAjOmsABcbVoXmAm8IhRV1mbqVMBqcWrZB1XXTSYm8YFsIys+vRQRN7Eoh6GdSEIixufCcb5bsCPO1t2w=="} (2) openapi doc
(3) the saved testcase
|
hi, it is weird. I would need to look at the source code of the API to be able to tell what is going on. |
You could refer the source code in EvoMaster/e2e-tests/spring-rest-rsa/src/main/java/com/example/demo. I just ran the testing against it, and also 1 test was saved.
|
2 tests were saved after adding a valid signature in the openapi doc:
It seems only one testcase is generated when the coverage is identical and the response is similar. I think it's OK and we just need to cover more targets. |
Although running more than 1 million tests, only 1 testcase was generated for 1 restful API.
Here is the command, using the default config. Is there any problem? I remember that it used to be possible to
generate multiple testcases.
And, below is the api doc:
The text was updated successfully, but these errors were encountered: