test note of iOS 17 with Xcode 15

[KazuCocoa]

The list of tests we may need to check for ios 17 as this repository

npm run e2e-test:base

• Need to check the faceID test app side

  4) XCUITestDriver - faceID -
       faceID enrollment functional tests applied to FaceID sample app
         should accept matching fingerprint if faceID is enrolled:
     Error: Element located by '-ios class chain:**/XCUIElementTypeStaticText[`label == "Face ID"`]' done not exist after 5000ms timeout
      at waitUntilExist (test/functional/basic/face-id-e2e-specs.js:64:15)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Context.<anonymous> (test/functional/basic/face-id-e2e-specs.js:96:9)

  5) XCUITestDriver - faceID -
       faceID enrollment functional tests applied to FaceID sample app
         should reject not matching fingerprint if faceID is enrolled:
     Error: Element located by '-ios class chain:**/XCUIElementTypeStaticText[`label == "Face ID"`]' done not exist after 5000ms timeout
      at waitUntilExist (test/functional/basic/face-id-e2e-specs.js:64:15)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Context.<anonymous> (test/functional/basic/face-id-e2e-specs.js:112:9)

  6) XCUITestDriver - faceID -
       faceID enrollment functional tests applied to FaceID sample app
         should enroll faceID and accept matching fingerprints then unenroll faceID and not be supported:
     Error: Element located by '-ios class chain:**/XCUIElementTypeStaticText[`label == "Face ID"`]' done not exist after 5000ms timeout
      at waitUntilExist (test/functional/basic/face-id-e2e-specs.js:64:15)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Context.<anonymous> (test/functional/basic/face-id-e2e-specs.js:137:9)

  8) XCUITestDriver - touchID -
       touchID enrollment functional tests applied to TouchId sample app
         "before each" hook for "should not support touchID if not enrolled":
     Error: Failed to create session.
An unknown server-side error occurred while processing the command. Original error: Could not create simulator with name 'appiumTest-C864A8C2-1155-41C1-83C4-8AAF6332483C-iPhone 8', device type id 'iPhone 8', with runtime ids 'com.apple.CoreSimulator.SimRuntime.iOS-17-0'
      at startWebDriverSession (file:///Users/kazu/GitHub/appium-xcuitest-driver/node_modules/webdriver/build/utils.js:68:15)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Function.newSession (file:///Users/kazu/GitHub/appium-xcuitest-driver/node_modules/webdriver/build/index.js:26:45)
      at async remote (file:///Users/kazu/GitHub/appium-xcuitest-driver/node_modules/webdriverio/build/index.js:48:22)
      at async initSession (test/functional/helpers/session.js:14:14)
      at async Context.<anonymous> (test/functional/basic/touch-id-e2e-specs.js:73:18)

npm run e2e-test:device

      1) should return recorded trace file on stop
2023-06-12T02:44:24.979Z INFO webdriver: COMMAND deleteSession()
2023-06-12T02:44:24.980Z INFO webdriver: [DELETE] http://127.0.0.1:4567/session/e1069e7e-9bbe-4091-b632-1f771f660ece
2023-06-12T02:44:25.003Z WARN webdriver: Request failed with status 404 due to A session is either terminated or not started
2023-06-12T02:44:25.003Z INFO webdriver: Retrying 1/1
2023-06-12T02:44:25.003Z INFO webdriver: [DELETE] http://127.0.0.1:4567/session/e1069e7e-9bbe-4091-b632-1f771f660ece
2023-06-12T02:44:25.007Z ERROR webdriver: Request failed with status 404 due to invalid session id: A session is either terminated or not started

e2e-test:driver

  1) XCUITestDriver
       should start and stop a session:
     Error: Timeout of 240000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/kazu/GitHub/appium-xcuitest-driver/test/functional/driver/driver-e2e-specs.js)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7)

  2) XCUITestDriver
       reset
         with udid: uses sim and resets afterwards if resetOnSessionStartOnly is false:
     Error: Timeout of 240000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/kazu/GitHub/appium-xcuitest-driver/test/functional/driver/driver-e2e-specs.js)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7)

  3) XCUITestDriver
       reset
         with udid booted: uses sim and leaves it afterwards:
     Error: Timeout of 240000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/kazu/GitHub/appium-xcuitest-driver/test/functional/driver/driver-e2e-specs.js)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7)

  4) XCUITestDriver
       reset
         with noReset set to true: leaves sim booted:
     Error: Timeout of 240000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/kazu/GitHub/appium-xcuitest-driver/test/functional/driver/driver-e2e-specs.js)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7)

e2e-test:long

ok

e2e-test:parallel

e2e-test:web

maybe need to update test code

  1) Safari - basics -
       init
         should start a session with custom init:

      AssertionError: expected 'Appium/welcome' to equal 'I am a page title'
      + expected - actual

      -Appium/welcome
      +I am a page title

      at Context.<anonymous> (test/functional/web/safari-basic-e2e-specs.js:45:26)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)

  2) iPhone Xs -
       should be able to tap on an element after scrolling:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:67:19
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at wrapped (node_modules/asyncbox/lib/asyncbox.js:95:13)
      at retry (node_modules/asyncbox/lib/asyncbox.js:68:13)
      at retryInterval (node_modules/asyncbox/lib/asyncbox.js:105:10)
      at spinTitleEquals (test/functional/web/helpers.js:64:5)
      at Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:134:21)

  3) iPhone Xs -
       should be able to tap on a button:
     AssertionError: expected '<html><head>\n  <title>I am a page ti…' to include 'Your comments: Hello'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/safari-nativewebtap-e2e-specs.js:145:43
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at wrapped (node_modules/asyncbox/lib/asyncbox.js:95:13)
      at retry (node_modules/asyncbox/lib/asyncbox.js:68:13)
      at retryInterval (node_modules/asyncbox/lib/asyncbox.js:105:10)
      at Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:143:21)

  4) iPad (10th generation) -
       should be able to bypass measuring the offset of banner:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:67:19
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at wrapped (node_modules/asyncbox/lib/asyncbox.js:95:13)
      at retry (node_modules/asyncbox/lib/asyncbox.js:68:13)
      at retryInterval (node_modules/asyncbox/lib/asyncbox.js:105:10)
      at spinTitleEquals (test/functional/web/helpers.js:64:5)
      at Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:124:21)

  5) iPad (10th generation) -
       should be able to tap on an element after scrolling:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:67:19
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at wrapped (node_modules/asyncbox/lib/asyncbox.js:95:13)
      at retry (node_modules/asyncbox/lib/asyncbox.js:68:13)
      at retryInterval (node_modules/asyncbox/lib/asyncbox.js:105:10)
      at spinTitleEquals (test/functional/web/helpers.js:64:5)
      at Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:134:21)

  6) iPad (10th generation) -
       with tabs -
         should be able to tap on an element after scrolling:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:67:19
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at wrapped (node_modules/asyncbox/lib/asyncbox.js:95:13)
      at retry (node_modules/asyncbox/lib/asyncbox.js:68:13)
      at retryInterval (node_modules/asyncbox/lib/asyncbox.js:105:10)
      at spinTitleEquals (test/functional/web/helpers.js:64:5)
      at Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:189:25)

  7) iPad (10th generation) -
       with tabs -
         should be able to tap on an element after scrolling, when the url bar is present:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:67:19
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at wrapped (node_modules/asyncbox/lib/asyncbox.js:95:13)
      at retry (node_modules/asyncbox/lib/asyncbox.js:68:13)
      at retryInterval (node_modules/asyncbox/lib/asyncbox.js:105:10)
      at spinTitleEquals (test/functional/web/helpers.js:64:5)
      at Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:212:25)

e2e-test:native-web-tap

Test code needs to be updated

  1) iPhone Xs -
       should be able to tap on an element after scrolling:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:53:13
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async wrapped (node_modules/asyncbox/build/lib/lib/asyncbox.js:95:13)
      at async retry (node_modules/asyncbox/build/lib/lib/asyncbox.js:68:13)
      at async retryInterval (node_modules/asyncbox/build/lib/lib/asyncbox.js:105:10)
      at async spinTitleEquals (test/functional/web/helpers.js:50:3)
      at async Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:145:11)

  2) iPhone Xs -
       should be able to tap on a button:
     AssertionError: expected '<html><head>\n  <title>I am a page ti…' to include 'Your comments: Hello'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/safari-nativewebtap-e2e-specs.js:162:31
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async wrapped (node_modules/asyncbox/build/lib/lib/asyncbox.js:95:13)
      at async retry (node_modules/asyncbox/build/lib/lib/asyncbox.js:68:13)
      at async retryInterval (node_modules/asyncbox/build/lib/lib/asyncbox.js:105:10)
      at async Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:160:11)

  3) iPad (10th generation) -
       should be able to tap on an element after scrolling:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:53:13
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async wrapped (node_modules/asyncbox/build/lib/lib/asyncbox.js:95:13)
      at async retry (node_modules/asyncbox/build/lib/lib/asyncbox.js:68:13)
      at async retryInterval (node_modules/asyncbox/build/lib/lib/asyncbox.js:105:10)
      at async spinTitleEquals (test/functional/web/helpers.js:50:3)
      at async Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:145:11)

  4) iPad (10th generation) -
       with tabs -
         should be able to tap on an element after scrolling:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:53:13
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async wrapped (node_modules/asyncbox/build/lib/lib/asyncbox.js:95:13)
      at async retry (node_modules/asyncbox/build/lib/lib/asyncbox.js:68:13)
      at async retryInterval (node_modules/asyncbox/build/lib/lib/asyncbox.js:105:10)
      at async spinTitleEquals (test/functional/web/helpers.js:50:3)
      at async Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:223:13)

  5) iPad (10th generation) -
       with tabs -
         should be able to tap on an element after scrolling, when the url bar is present:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:53:13
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async wrapped (node_modules/asyncbox/build/lib/lib/asyncbox.js:95:13)
      at async retry (node_modules/asyncbox/build/lib/lib/asyncbox.js:68:13)
      at async retryInterval (node_modules/asyncbox/build/lib/lib/asyncbox.js:105:10)
      at async spinTitleEquals (test/functional/web/helpers.js:50:3)
      at async Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:251:13)

[KazuCocoa]

Real device set location should be via com.apple.instruments.server.services.LocationSimulation instead of com.apple.dt.simulatelocation over the remote server.

[mwakizaka]

I'll try updating the tests if no problem

[Xiaotian0726]

Real device set location should be via com.apple.instruments.server.services.LocationSimulation instead of com.apple.dt.simulatelocation over the remote server.

Is this rule for all iOS real devices or just iOS 17?

[KazuCocoa]

it is for iOS 17+. Please use https://appium.github.io/appium-xcuitest-driver/latest/reference/execute-methods/#mobile-setsimulatedlocation . XCUITest driver calls it internally to set location for real devices for location command as well.

[Xiaotian0726]

it is for iOS 17+. Please use https://appium.github.io/appium-xcuitest-driver/latest/reference/execute-methods/#mobile-setsimulatedlocation . XCUITest driver calls it internally to set location for real devices for location command as well.

Thx, I got it.