Skip to content

Ensure the CI is marked as failed when Maestro test is failing #4700

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
May 15, 2025
Merged

Ensure the CI is marked as failed when Maestro test is failing #4700

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
4439f89
Ensure the CI is marked as failed when Maestro test is failing
bmarty May 9, 2025
8de0cd1
Fix typo in String to disable bookmark in ChromeCustomTab
bmarty May 9, 2025
fc0bf25
Fix Maestro test now that matrix.org is using MAS to authenticate.
bmarty May 9, 2025
99f00e3
Fix Maestro test - other issues due to recent changes.
bmarty May 9, 2025
6dbdf29
Maestro: add test on Element Call
bmarty May 9, 2025
5aab0d8
Run through Chrome on boarding step.
bmarty May 9, 2025
d400644
Try suggestion from https://github.com/mobile-dev-inc/Maestro/issues/…
bmarty May 9, 2025
5c09a3c
Revert "Try suggestion from https://github.com/mobile-dev-inc/Maestro…
bmarty May 12, 2025
f7240ca
Add comment on Maestro flow.
bmarty May 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion .github/workflows/maestro-local.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
maestro-cloud:
name: Maestro test suite
runs-on: ubuntu-latest
needs: [build-apk]
needs: [ build-apk ]
# Allow one per PR.
concurrency:
group: ${{ format('maestro-{0}', github.ref) }}
Expand All @@ -80,6 +80,7 @@ jobs:
- name: Install maestro
run: curl -fsSL "https://get.maestro.mobile.dev" | bash
- name: Run Maestro tests in emulator
id: maestro_test
uses: reactivecircus/android-emulator-runner@v2
continue-on-error: true
env:
Expand Down Expand Up @@ -109,3 +110,8 @@ jobs:
retention-days: 5
overwrite: true
if-no-files-found: error
- name: Fail the workflow in case of error in test
if: steps.maestro_test.outcome != 'success'
run: |
echo "Maestro tests failed. Please check the logs."
exit 1
20 changes: 11 additions & 9 deletions .maestro/tests/account/login.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,22 +7,24 @@ appId: ${MAESTRO_APP_ID}
- runFlow: ../assertions/assertLoginDisplayed.yaml
- tapOn:
id: "login-continue"
## MAS page
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we maybe add some comments here to remind us this is not working at the moment?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

f7240ca

## Conditional workflow to pass the Chrome first launch welcome page.
- runFlow:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a comment to specify this is a workaround for Chrome's onboarding screen.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

f7240ca

when:
visible: 'Use without an account'
commands:
- tapOn: "Use without an account"
## Working when running Maestro locally, but not on the CI yet.
- tapOn:
id: "login-email_username"
id: "form-1"
- inputText: ${MAESTRO_USERNAME}
- pressKey: Enter
- tapOn:
id: "login-password"
- inputText: "wrong-password"
- pressKey: Enter
- tapOn: "Continue"
- tapOn: "OK"
- tapOn:
id: "login-password"
- eraseText: 20
id: "form-3"
- inputText: ${MAESTRO_PASSWORD}
- pressKey: Enter
- tapOn: "Continue"
## Back to native world
- runFlow: ../assertions/assertSessionVerificationDisplayed.yaml
- runFlow: ./verifySession.yaml
- runFlow: ../assertions/assertAnalyticsDisplayed.yaml
Expand Down
2 changes: 1 addition & 1 deletion .maestro/tests/account/verifySession.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@ appId: ${MAESTRO_APP_ID}
- hideKeyboard
- tapOn: "Continue"
- extendedWaitUntil:
visible: "Device verified"
visible: "Verification complete"
timeout: 30000
- tapOn: "Continue"
3 changes: 2 additions & 1 deletion .maestro/tests/roomList/createAndDeleteDM.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@ appId: ${MAESTRO_APP_ID}
- tapOn:
text: ${MAESTRO_INVITEE1_MXID}
index: 1
- tapOn: "Send invite"
- takeScreenshot: build/maestro/330-createAndDeleteDM
- tapOn: "maestroelement2"
- scroll
- tapOn: "Leave conversation"
- tapOn: "Leave room"
- tapOn: "Leave"
13 changes: 13 additions & 0 deletions .maestro/tests/roomList/timeline/call/call.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
appId: ${MAESTRO_APP_ID}
---
- tapOn: "Start a call"
- takeScreenshot: build/maestro/700-Call
- extendedWaitUntil:
visible: "maestroelement"
timeout: 10000
- takeScreenshot: build/maestro/710-Call
# Hangup
- tapOn: "End call"
- extendedWaitUntil:
visible: "MyRoom"
timeout: 10000
1 change: 1 addition & 0 deletions .maestro/tests/roomList/timeline/timeline.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,6 @@ appId: ${MAESTRO_APP_ID}
- runFlow: messages/text.yaml
- runFlow: messages/location.yaml
- runFlow: messages/poll.yaml
- runFlow: call/call.yaml
- back
- runFlow: ../../assertions/assertHomeDisplayed.yaml
2 changes: 1 addition & 1 deletion ...tils/src/main/kotlin/io/element/android/libraries/androidutils/browser/ChromeCustomTab.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ fun Activity.openUrlInChromeCustomTab(
// Disable download button
intent.putExtra("org.chromium.chrome.browser.customtabs.EXTRA_DISABLE_DOWNLOAD_BUTTON", true)
// Disable bookmark button
intent.putExtra("org.chromium.chrome.browser.customtabs.EXTRA_DISABLE_START_BUTTON", true)
intent.putExtra("org.chromium.chrome.browser.customtabs.EXTRA_DISABLE_STAR_BUTTON", true)
Copy link
Member Author

@bmarty bmarty May 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also spot this typo while looking if we could disable the option to minify the ChromeTab during Oidc login, but it appears that it's not possible.

image

Minimized tab does not make any sense here:

image

EDIT: EXTRA_DISABLE_STAR_BUTTON does not seem to have any effect... But not a big deal.

intent.putExtra(Browser.EXTRA_HEADERS, Bundle().apply {
putString("Accept-Language", Locale.getDefault().toLanguageTag())
})
Expand Down
Loading