Only one testcase was generated for 1 API (possible issue with string constraints)

[ytfrank]

Although running more than 1 million tests, only 1 testcase was generated for 1 restful API.

Here is the command, using the default config. Is there any problem? I remember that it used to be possible to
generate multiple testcases.

java -jar ~/Downloads/evomaster.jar  --maxTime 7200s --outputFolder "/Users/apple/gitrepo/EvoMaster/whitebox-demo/src/main/java/long9"
*
 _____          ___  ___          _
|  ___|         |  \/  |         | |
| |____   _____ | .  . | __ _ ___| |_ ___ _ __
|  __\ \ / / _ \| |\/| |/ _` / __| __/ _ \ '__|
| |___\ V / (_) | |  | | (_| \__ \ ||  __/ |
\____/ \_/ \___/\_|  |_/\__,_|___/\__\___|_|


* EvoMaster version: 3.4.0
* Loading configuration file from: /Users/apple/gitrepo/em.yaml
* Initializing...
00:28:14.270 [main] WARN  o.t.u.TestcontainersConfiguration - Attempted to read Testcontainers configuration file at file:/Users/apple/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /Users/apple/.testcontainers.properties (No such file or directory)
* There is only 1 usable RESTful API endpoint defined in the schema configuration
* Starting to generate test cases
* Consumed search budget: 74.601%
* Consumed search budget: 99.994%
* Covered targets: 22; time per test: 4.0ms (1.0 actions); since last improvement: 4806s
* Starting to apply minimization phase
* Recomputing full coverage for 2 tests
02:28:15.179 [main] WARN  o.e.core.search.service.Minimizer - Different status code 500 returned from original null, ie, null != 500 for endpoint: POST:/api/bind_card_apply
* Recomputing coverage did lose many targets, more than the threshold 20.0%: from 22 to 7, i.e., lost 15
* Missing targets:
* Line_at_com.example.demo.controller.DemoController_00034
* Success_Call_at_com.example.demo.controller.DemoController_00034_0
* Success_Call_at_com.example.demo.controller.DemoController_00034_1
* Success_Call_at_com.example.demo.controller.DemoController_00034_2
* Success_Call_at_com.example.demo.controller.DemoController_00034_3
* Success_Call_at_com.example.demo.controller.DemoController_00034_4
* Success_Call_at_com.example.demo.controller.DemoController_00034_5
* Line_at_com.example.demo.controller.DemoController_00036
* Success_Call_at_com.example.demo.controller.DemoController_00036_0
* Line_at_com.example.demo.controller.DemoController_00037
* Line_at_com.example.demo.controller.DemoController_00038
* Success_Call_at_com.example.demo.controller.DemoController_00038_0
* Success_Call_at_com.example.demo.controller.DemoController_00038_1
* -1:POST:/api/bind_card_apply
* RESPONSE_BODY_PAYLOAD_POST/api/bind_card_apply_null
* No test to minimize
* Minimization phase took 0 seconds
* Evaluated tests: 1188138
* Evaluated actions: 1307316
* Needed budget: 34%
* Passed time (seconds): 7200
* Execution time per test (ms): Avg=4.77 , min=2.00 , max=35173.00
* Execution time per action (ms): Avg=4.42 , min=2.00 , max=35173.00
* Computation overhead between tests (ms): Avg=1.29 , min=0.00 , max=250.00
* Computation overhead of resetting the SUT (ms): Avg=0.99 , min=0.00 , max=28983.00
* Computation overhead of fetching test results, per test, subset of targets (ms): Avg=0.56 , min=0.00 , max=5131.00
* Going to save 1 test to /Users/apple/gitrepo/EvoMaster/whitebox-demo/src/main/java/long9
* TCP timeouts: 2
* Potential faults: 2
* Covered targets (lines, branches, faults, etc.): 10
* Bytecode line coverage: 2% (1 out of 42 in 1 units/classes)
* Successfully executed (HTTP code 2xx) 0 endpoints out of 1 (0%)
* EvoMaster process has completed successfully

And, below is the api doc:

openapi: 3.1.0
info:
  title: OpenAPI definition
  version: v0
servers:
- url: http://localhost:8000
  description: Generated server url
paths:
  /api/bind_card_apply:
    post:
      tags:
      - demo-controller
      operationId: bindCardApply
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/CommonReqBindCardReq'
        required: true
      responses:
        "200":
          description: OK
          content:
            '*/*':
              schema:
                $ref: '#/components/schemas/CommonRespBindCardResp'
components:
  schemas:
    BindCardReq:
      required:
      - bankCardNo
      - bankCardPhoneNumber
      - idCardNo
      - loanPersonName
      type: object
      properties:
        idCardNo:
          maxLength: 18
          minLength: 18
          pattern: "^[0-9]{17}[0-9Xx]$"
          type: string
        loanPersonName:
          maxLength: 20
          minLength: 0
          type: string
        bankCardNo:
          maxLength: 19
          minLength: 16
          pattern: "^[0-9]+$"
          type: string
        bankCardPhoneNumber:
          maxLength: 11
          minLength: 11
          pattern: "^[1][3-9][0-9]{9}$"
          type: string
    CommonReqBindCardReq:
      type: object
      properties:
        appId:
          type: string
        data:
          type: string
        requestId:
          type: string
        timestamp:
          type: string
        key:
          type: string
        sign:
          type: string
        bizData:
          $ref: '#/components/schemas/BindCardReq'
    BindCardResp:
      type: object
      properties:
        code:
          type: string
        msg:
          type: string
        sessionId:
          type: string
    CommonRespBindCardResp:
      type: object
      properties:
        data:
          type: string
        key:
          type: string
        sign:
          type: string
        bizData:
          $ref: '#/components/schemas/BindCardResp'

[arcuri82]

hi @ytfrank ,

thanks for reporting this issue. I think I know what is the problem. Probably it is related to:

         bankCardNo:
          maxLength: 19
          minLength: 16
          pattern: "^[0-9]+$"

we can handle constraints on length, and constraints on patterns, but not both at same time :(
The pattern one would prevail.
This needs to be fixed in EM.
For the time being, can you try to use pattern: "^[0-9]{16,19}$" and see if it helps?

[arcuri82]

Dear @ytfrank ,

thanks for trying out EvoMaster and opening an issue here. If you have 10 minutes, we would be grateful if you could fill this survey on test naming strategies #1206. We will use such information to improve EvoMaster. Thanks!

[ytfrank]

Unfortunately, still only 1 testcase was generated even after removing all the constraints about length and patterns.
By the way, I have submitted the survey.

Here is the new api doc:

openapi: 3.1.0
info:
  title: OpenAPI definition
  version: v0
servers:
- url: http://localhost:8000
  description: Generated server url
paths:
  /api/bind_card_apply:
    post:
      tags:
      - demo-controller
      operationId: bindCardApply
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/CommonReqBindCardReq'
        required: true
      responses:
        "200":
          description: OK
          content:
            '*/*':
              schema:
                $ref: '#/components/schemas/CommonRespBindCardResp'
components:
  schemas:
    BindCardReq:
      required:
      - bankCardNo
      - bankCardPhoneNumber
      - idCardNo
      - loanPersonName
      type: object
      properties:
        idCardNo:
          type: string
        loanPersonName:
          type: string
        bankCardNo:
          type: string
        bankCardPhoneNumber:
          type: string
    CommonReqBindCardReq:
      type: object
      properties:
        appId:
          type: string
        data:
          type: string
        requestId:
          type: string
        timestamp:
          type: string
        key:
          type: string
        sign:
          type: string
        bizData:
          $ref: '#/components/schemas/BindCardReq'
    BindCardResp:
      type: object
      properties:
        code:
          type: string
        msg:
          type: string
        sessionId:
          type: string
    CommonRespBindCardResp:
      type: object
      properties:
        data:
          type: string
        key:
          type: string
        sign:
          type: string
        bizData:
          $ref: '#/components/schemas/BindCardResp'

[arcuri82]

@ytfrank thanks for answering the survey

removing the constraints on schema would make things worse, as I assume that the API is still validating those constraints internally.

regarding regex handling, there is currently a bias in short strings... that might struggle to reach a 16-char long string (the handling of regex in EM should be improved). I would suggest to put back all the constraints, and try to make sure all length-related constraints are expressed in the pattern, eg, pattern: "^[0-9]{16,19}$" to represent min=16 and max=19

[ytfrank]

Still only saved 1 test, although many tests were evaluated. Is it expected that only one testcase is generated when the response is similar?

Here is the info:
(1) part of SUT log
2025-03-24 17:27:59 [http-nio-8080-exec-10] INFO c.e.demo.controller.DemoController - ------Return resp:{}{"data":"S6geXrvlQqX+XXHCX9PSMVQeO9C8QfJXJFBSS2Ef2RbVKf5mQSH0hgoU7eBdy470","key":"fwfMybMfkb4uOT3eeuS4O3jhDWSyik95mb9hlLkkLDOgNgd7X71lVB0qJWzhCn5bEeKWlm1hu3gBLXM45AX44d3I4msMfYV0Cqb+ZzAExDyZ5L7GR4XzChV3f+xJFskHJUb2quv+Tr9TuDBTzWFJUKK+U634VUNA1Ogkkl+TddmSTBfap3rmcO+e+L42G+avjpl9xCd825inCUG8C5WA9ZTa2RpJRcTom76lNEf9He7gK+0pvtM3JQy7BQYXlTwLDuW5L/Wza/33GzXrBNHvGmgawVX3SutBPPhyekjrIlGlEvjsCLRdbUcHpsl2stXwQJan2iI4GK50Eb2HZyPC/Q==","sign":"aXZ3p7DAxubnVfC7ernXA+QBK/JcqX8a8FLBDgfPA17eis6jHIoQdSR3ThWr4gUDJRMv6exCRVdwNU9WVhFkZC4lPDeu9HMrDZrzDZatr0q8tHnK+t6PpF8LUZpJ3MGbZ5ViyL21wqcqtxlndz0emYUoGD+TEykNH7y8roKyJ1PdfWWkrjEVBdlOv5RnSX1OZV/QPjh7a12z+vvF+pFQlugawcMePsMpfi4kBWchfTAXbZjCxSOwaMEKzTg9F6n1ydwKwKAWBdYfNjtsE8J4kw2c7ifRwNCvq5V9ZzlUOdGYBpO++ZhaBdWsKVCqbzZ/rZVYcQeUL781MKaX6Pr2dA=="}
2025-03-24 17:27:59 [qtp1089504328-19] INFO ks.long5.DemoEmbeddedSutController - Resetting state of SUT...
2025-03-24 17:27:59 [http-nio-8080-exec-1] INFO c.e.demo.controller.DemoController - ------Received req:{}{"appId":"KqV","bizData":{"bank_card_no":"682697863096554076","bank_card_phone_number":"17001810112","id_card_no":"239582327441924811","loan_person_name":"130"},"data":"IVxqEl","key":"4c","requestId":"b7QJg","sign":"EM_2391_XYZ","timestamp":"qBkndYxXDOF"}
2025-03-24 17:27:59 [http-nio-8080-exec-1] INFO c.e.demo.controller.DemoController - ------Return resp:{}{"data":"W85yJaLI2Vb3rVZIAK/OM1eVmpsBEiqGSuc3HcIhVgs51pMQpN2occ+ahelT0TMV","key":"dojeVzx8OGNF4TcSXtJqSc1Cm6y35K+GR39yWXFG9Pv0nT47O+KOb+antpwWz6e6/e5oylGpO52FA65DTxogg2XcIvVeR/yH9TzllqMZ6eLyiwme99koM51pmsgV3Nd/RguMNbyV99voYcFbYO/qoT/bvnpOxJIPHCbsgnVVcmoPQWaflxMdrz8ytCRShqqWhKm4iOhtSjomp3AhQmdn9JRI31D7Qqf93ZVrFfakkEgiHfJ1udsSkRjvCw/ZV8fJei0SzujZqVZG6LWycEtDgR252mrNj/RyPg1lab/SazLWfkcgFuIaldst3s4D2pfhCE4KfL3QkrYzm76pwMvS/Q==","sign":"P6W3nI8DnUAgeSO8QC00Xki6wyxJhPuEZlVV9pFRXUz/cWykrycu0+YjuxtLet/KrjwRvwtb0za8wBxvNHhc8xs+2XRsW+xv1qNfzTuyWfAytY5mnZCsh278ndM2+lEVkyVYw3HikMv/Cp/nWoEekRhC509qINMITjoYyIrdlGH2MN5uuQ1aYYhlL5sB7GL7lxLK7jJoAvBcmd7kF4sYbZ90+g8WBAp47V4J+S5nCFM9A19ZQCYranZej+qqvGAy6djYj3VpZTEOPxXCj+3kuThJbozbZQudzVT4BGts6YZsjl3wktfXRQYtICIwHjbWKVtRzLzwZZsoKzYFR52YhA=="}
2025-03-24 17:27:59 [qtp1089504328-18] INFO ks.long5.DemoEmbeddedSutController - Resetting state of SUT...
2025-03-24 17:27:59 [http-nio-8080-exec-2] INFO c.e.demo.controller.DemoController - ------Received req:{}{"appId":"EM_2390_XYZ","bizData":{"bank_card_no":"682697863096554076","bank_card_phone_number":"17001810112","id_card_no":"239582327441924811","loan_person_name":"130"},"data":"JVxqEl","key":"4d","requestId":"b7QJg","sign":"EM_2391_XYZ","timestamp":"qBkndYxXDOF"}

2025-03-24 17:27:59 [http-nio-8080-exec-2] INFO c.e.demo.controller.DemoController - ------Return resp:{}{"data":"Udq+Pxj63RniizY9hJ+jiHmkUbo5fxygW9JTWuLdwPGOlMX3lflxXNoKg9O8DiPl","key":"bJAhDO2ZBtIW1oe0vBvMCV2BESMqnejfXBhGEkiSylpkLq3NhWXFZRIlWJAY2i+Na9yTBQRSReGpXPZP48VdxF9M58Te5OD6ep0rlfKADitj7IEJh/oTkGRX+08QfDmcdDnbr2po4odJUW7NRxn3rckuOWM42GFLb5GXSNpyoVc4laOJRASuM1ObxdyPQF1cTgp7Xn6kEzWWKv2a9PU8RZ13ynTOJ/AEPYKdK3Z6itEM2SJtRFeQKlcH5IFSwFFZ19c4BhRNbVWIJvAXundvsWQR6HN3PwEGGIQkbOJuWJK9q+jeJ15FGuRBSNgdCunSwHi/w1KE26nQNEG9kp6Szw==","sign":"BP4OHeOEHshegDft+JFp9PMG+Z1hXqYXT8bUHks1oa3zHUiOWrFxojERf+/Fdmtvq0/K5YvaQ+B3D6rWShyKuDJfU0ltNHebHmrFyZB5nnGzEI4kZOXYZnykVxroh9/4KbXJT4nuvpaH07qo0fIaC0WPFXfJr8cFVN5jiZnhqS8A8qrkfBE3ezo0EazKz51HTL3xE9jzQSH+kV1oquig/hUazL7alGJODLcn+u5n2sJYqxsRJoeczbDLBBV3lMne7bzs+cAjOmsABcbVoXmAm8IhRV1mbqVMBqcWrZB1XXTSYm8YFsIys+vRQRN7Eoh6GdSEIixufCcb5bsCPO1t2w=="}
2025-03-24 17:27:59 [qtp1089504328-19] INFO ks.long5.DemoEmbeddedSutController - Resetting state of SUT...
2025-03-24 17:27:59 [http-nio-8080-exec-3] INFO c.e.demo.controller.DemoController - ------Received req:{}{"appId":"EM_2390_XYZ","bizData":{"bank_card_no":"682697863096554076","bank_card_phone_number":"17001810112","id_card_no":"239582327441924811","loan_person_name":"130"},"data":"IVxqEl","key":"4c","requestId":"b7QJg","sign":"EM_2391_XYZ","timestamp":"qBkndYxXDOFM"}

(2) openapi doc
openapi: 3.1.0
info:
title: OpenAPI definition
version: v0
servers:

• url: http://localhost:8000
  description: Generated server url
  paths:
  /api/bind_card_apply:
  post:
  tags:
  - demo-controller
  operationId: bindCardApply
  requestBody:
  content:
  application/json:
  schema:
  $ref: '#/components/schemas/CommonReqBindCardReq'
  required: true
  responses:
  "200":
  description: OK
  content:
  '/':
  schema:
  $ref: '#/components/schemas/CommonRespBindCardResp'
  components:
  schemas:
  BindCardReq:
  required:
  - bankCardNo
  - bankCardPhoneNumber
  - idCardNo
  - loanPersonName
  type: object
  properties:
  idCardNo:
  pattern: "^[0-9]{17}[0-9Xx]$"
  type: string
  loanPersonName:
  pattern: "^[0-9]{1,20}$"
  type: string
  bankCardNo:
  pattern: "^[0-9]{16,19}$"
  type: string
  bankCardPhoneNumber:
  pattern: "^[1][3-9][0-9]{9}$"
  type: string
  CommonReqBindCardReq:
  type: object
  properties:
  appId:
  type: string
  data:
  type: string
  requestId:
  type: string
  timestamp:
  type: string
  key:
  type: string
  sign:
  type: string
  bizData:
  $ref: '#/components/schemas/BindCardReq'
  BindCardResp:
  type: object
  properties:
  code:
  type: string
  msg:
  type: string
  sessionId:
  type: string
  CommonRespBindCardResp:
  type: object
  properties:
  data:
  type: string
  key:
  type: string
  sign:
  type: string
  bizData:
  $ref: '#/components/schemas/BindCardResp'

(3) the saved testcase
@test(timeout = 60000)
public void test_0() throws Exception {

    given().accept("*/*")
            .header("x-EMextraHeader123", "")
            .contentType("application/json")
            .body(" { " + 
                " \"appId\": \"_EM_0_XYZ_\", " + 
                " \"data\": \"j66xS1uTWaMNar01\", " + 
                " \"requestId\": \"_EM_1_XYZ_\", " + 
                " \"timestamp\": \"_EM_2_XYZ_\", " + 
                " \"key\": \"Kon_BuCkGdO8a\", " + 
                " \"sign\": \"_EM_3_XYZ_\", " + 
                " \"bizData\": { " + 
                " \"idCardNo\": \"810842383106002679\", " + 
                " \"loanPersonName\": \"5\", " + 
                " \"bankCardNo\": \"9572223310436983\", " + 
                " \"bankCardPhoneNumber\": \"13894612629\" " + 
                " } " + 
                " } ")
            .post(baseUrlOfSut + "/api/bind_card_apply?EMextraParam123=_EM_4_XYZ_")
            .then()
            .statusCode(200)
            .assertThat()
            .contentType("application/json")
            .body("'data'", containsString("eYy2Vo8ShMKBXBW90MnEWULcAXAYHV5hGPs37+eRgHRmqe8Ma6OafcCpt0dQqQjQ"))
            .body("'key'", containsString("TgKXdcHxXjW5pTgx8w8aUK24GZJFrx4UkK3WWDav/FDh7/c3rR8aB1xTbQRgKAHI9zszaE0IY+5XxjyPVT8pa9xr4xJKLN8U2/wyDv4FUA8lemwUymEtK5DaBecGww+YI+izTQl9Wu2/e5TE2J5LJTpRu/ENzBkwqCnZTv4CfuAN6cISMouOXmL0+mlCx6qRpEUz1jEXWgv/Nki4nReParUG/6SPQgIEudonqHA4m8t4rVbigoSsp7g/E8dkuyFN6pFw/fPerGE1pzJrUUHZcI5h9qpI7HUJYbNQ6+F42KgRqJBoZqPN+4CvPDNG6pADIEiE1cZK4NS9DG5MXOqcYw=="))
            .body("'sign'", containsString("T6bhqZArJ4X8/nZa/ETNDL352J28lf/Dxbdewc2k57RV2QFK4o/WsEdNHx0SA0wg4ejZzZpTEGeaZvmDT9QEM/dIGM4/5xlgbXyIPtOogiLXz9cUZ3zLaP7ygBajikvVMH7hRVOh51PQPHAWbSqO8uSkvzFEbHxjI9tMC55D3agYBzCyEjDIS77GhKwX1j7xF6LhFEfv5IG2DqSXr5Rsaj/uydXH1iCAhDgqO2aWXO1vnBmPmkwP5hHG+bUPbYTxn0OCZz058ZaeyTvsAkRDm1xoK57/Sz0gu3h/vyb4OpfpbcMird+BDjrTRFx5hphzZVUd/xxP0JcU0BgKOYcIMw=="))
            .body("'bizData'", nullValue());
}

[arcuri82]

hi,

it is weird. I would need to look at the source code of the API to be able to tell what is going on.

[ytfrank]

You could refer the source code in EvoMaster/e2e-tests/spring-rest-rsa/src/main/java/com/example/demo. I just ran the testing against it, and also 1 test was saved.

• Starting to generate test cases
• Consumed search budget: 99.463%
• Covered targets: 13; time per test: 6.3ms (1.0 actions); since last improvement: 88s
• Starting to apply minimization phase
• Recomputing full coverage for 1 tests
• Recomputing coverage did lose many targets, more than the threshold 20.0%: from 13 to 7, i.e., lost 6
• Missing targets:
• Line_at_com.example.demo.controller.DemoController_00031
• Success_Call_at_com.example.demo.controller.DemoController_00031_0
• Line_at_com.example.demo.controller.DemoController_00032
• Line_at_com.example.demo.controller.DemoController_00033
• Success_Call_at_com.example.demo.controller.DemoController_00033_0
• Success_Call_at_com.example.demo.controller.DemoController_00033_1
• No test to minimize
• Minimization phase took 0 seconds
• Evaluated tests: 6549
• Evaluated actions: 6950
• Needed budget: 0%
• Passed time (seconds): 90
• Execution time per test (ms): Avg=10.43 , min=4.00 , max=1333.00
• Execution time per action (ms): Avg=9.77 , min=4.00 , max=1333.00
• Computation overhead between tests (ms): Avg=3.29 , min=1.00 , max=56.00
• Computation overhead of resetting the SUT (ms): Avg=1.76 , min=0.00 , max=202.00
• Computation overhead of fetching test results, per test, subset of targets (ms): Avg=1.21 , min=0.00 , max=29.00
• Going to save 1 test to /Users/apple/gitrepo/EvoMaster/whitebox-deom/src/main/java/ks/long9
• Potential faults: 2
• Covered targets (lines, branches, faults, etc.): 10
• Bytecode line coverage: 2% (1 out of 41 in 1 units/classes)
• Successfully executed (HTTP code 2xx) 0 endpoints out of 1 (0%)

[ytfrank]

2 tests were saved after adding a valid signature in the openapi doc:
sign:
type: string

•      enum:
  

•        - +"Svlk4DIoAO0Q2cAU5ZLS0uESjKI4fz9TE7yzyKAZSRlVyJWMyaX7ZxzqUMAWusWV7+W8jLk2lNgP+Fzg8yf/OzKpz8R45ih94sBanvhClOdL6m9J8iY2+hxINaHme9YIdGnHAU5Sg47BAqlljrvKMEWcEsEDa25OMVzMeHPCig24PjurnfGHWmwEGJ/p9TZvcAOQPkmaAeRK2IfH/3aNzJNYJDNOGtETxGFbg0A54ry8+GGtdPtiY8yr5JLkVT6nGaRpZN2VQBgcbE9Uxlrx80RiHRbtR87i8rT95kKv1OnrlxTG/U+j1smuBNDqWlH65n829LvO2yIefM7RKiLt+Q=="
  

It seems only one testcase is generated when the coverage is identical and the response is similar. I think it's OK and we just need to cover more targets.