Ensure the CI is marked as failed when Maestro test is failing #4700

bmarty · 2025-05-09T12:30:09Z

Content

Ensure that the CI will complain that Maestro fails.
Fix existing issues on Maestro tests, not detected due to the first point
Add test on Element Call, to ensure that starting a call in a room will not crash the application.

Motivation and context

Fix Maestro, add more test, and secure releases

Screenshots / GIFs

Tests

Happy CI
Run maestro locally should work too

Tested devices

Physical
Emulator
OS version(s):

Checklist

Changes have been tested on an Android device or Android emulator with API 24
UI change has been tested on both light and dark themes
Accessibility has been taken into account. See https://github.com/element-hq/element-x-android/blob/develop/CONTRIBUTING.md#accessibility
Pull request is based on the develop branch
Pull request title will be used in the release note, it clearly define what will change for the user
Pull request includes screenshots or videos if containing UI changes
You've made a self review of your PR

github-actions · 2025-05-09T12:35:06Z

📱 Scan the QR code below to install the build (arm64 only) for this PR.

If you can't scan the QR code you can install the build via this link: https://i.diawi.com/uTsUyo

codecov · 2025-05-09T12:35:44Z

Codecov Report

Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 80.11%. Comparing base (5ed2e92) to head (f7240ca).
Report is 30 commits behind head on develop.

Files with missing lines	Patch %	Lines
.../libraries/androidutils/browser/ChromeCustomTab.kt	0.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #4700   +/-   ##
========================================
  Coverage    80.11%   80.11%           
========================================
  Files         2126     2126           
  Lines        56303    56303           
  Branches      7021     7021           
========================================
  Hits         45107    45107           
  Misses        8780     8780           
  Partials      2416     2416

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

bmarty · 2025-05-09T13:27:47Z

...idutils/src/main/kotlin/io/element/android/libraries/androidutils/browser/ChromeCustomTab.kt

@@ -53,7 +53,7 @@ fun Activity.openUrlInChromeCustomTab(
                // Disable download button
                intent.putExtra("org.chromium.chrome.browser.customtabs.EXTRA_DISABLE_DOWNLOAD_BUTTON", true)
                // Disable bookmark button
-                intent.putExtra("org.chromium.chrome.browser.customtabs.EXTRA_DISABLE_START_BUTTON", true)
+                intent.putExtra("org.chromium.chrome.browser.customtabs.EXTRA_DISABLE_STAR_BUTTON", true)


I also spot this typo while looking if we could disable the option to minify the ChromeTab during Oidc login, but it appears that it's not possible.

Minimized tab does not make any sense here:

EDIT: EXTRA_DISABLE_STAR_BUTTON does not seem to have any effect... But not a big deal.

bmarty · 2025-05-09T16:07:37Z

With the latest commit we arrive to this step:

But the test does not go further on the CI.

bmarty · 2025-05-12T08:17:10Z

With d400644 the emulator is stuck on:

Reverting this commit.

This reverts commit d400644.

bmarty · 2025-05-15T12:12:28Z

@jmartinesp I think we can (review and) merge this PR and let the action Maestro (local) / Maestro test suite (pull_request) not be mandatory temporarily. At least the PR make the action to fail and fix the issues in case we want to run Maestro locally, that we may want to do before each release until the CI is fixed.

I'll try in another PR to use our action runner to see if it's working better.

WDYT?

jmartinesp · 2025-05-15T12:19:25Z

Since Maestro is already broken I don't see any harm on merging this: even if it doesn't solve the problem with the webview in the CI, it's a good first step. If you think it's ready to review I can take a look 👍 .

jmartinesp · 2025-05-15T13:42:03Z

I was trying to test this locally and it failed at some random point (probably a fluke) but when I restarted the flow the browser still had the MAS credentials so the 'enter credentials' sub-flow was no longer working. I think we may want to either take that into account with another when condition or maybe also crear the browser's data if possible.

jmartinesp

I was able to complete the flow locally, so I only have a couple of suggestions. Thanks!

jmartinesp · 2025-05-15T12:24:29Z

.maestro/tests/account/login.yaml

@@ -7,22 +7,17 @@ appId: ${MAESTRO_APP_ID}
 - runFlow: ../assertions/assertLoginDisplayed.yaml
 - tapOn:
    id: "login-continue"
+## MAS page


Should we maybe add some comments here to remind us this is not working at the moment?

jmartinesp · 2025-05-15T12:25:35Z

.maestro/tests/account/login.yaml

@@ -8,6 +8,11 @@ appId: ${MAESTRO_APP_ID}
 - tapOn:
    id: "login-continue"
 ## MAS page
+- runFlow:


Maybe add a comment to specify this is a workaround for Chrome's onboarding screen.

bmarty · 2025-05-15T15:12:21Z

I was trying to test this locally and it failed at some random point (probably a fluke) but when I restarted the flow the browser still had the MAS credentials so the 'enter credentials' sub-flow was no longer working. I think we may want to either take that into account with another when condition or maybe also crear the browser's data if possible.

I also faced this problem. You need to sign out (manually) on the MAS page during the flow. This is not ideal, but should be OK on the CI with a flushed emulator...

sonarqubecloud · 2025-05-15T15:29:30Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Ensure the CI is marked as failed when Maestro test is failing

4439f89

bmarty added 4 commits May 9, 2025 14:38

Fix typo in String to disable bookmark in ChromeCustomTab

8de0cd1

Fix Maestro test now that matrix.org is using MAS to authenticate.

fc0bf25

Fix Maestro test - other issues due to recent changes.

99f00e3

Maestro: add test on Element Call

6dbdf29

bmarty added the PR-Build For changes related to build, tools, CI/CD label May 9, 2025

bmarty marked this pull request as ready for review May 9, 2025 13:25

bmarty requested a review from a team as a code owner May 9, 2025 13:25

bmarty requested review from jmartinesp and removed request for a team May 9, 2025 13:25

bmarty commented May 9, 2025

View reviewed changes

Run through Chrome on boarding step.

5aab0d8

Try suggestion from mobile-dev-inc/Maestro#1126 (comment)

d400644

Revert "Try suggestion from mobile-dev-inc/Maestro#1126 (comment)"

5c09a3c

This reverts commit d400644.

jmartinesp approved these changes May 15, 2025

View reviewed changes

Add comment on Maestro flow.

f7240ca

bmarty merged commit 38ad415 into develop May 15, 2025
24 checks passed

bmarty deleted the feature/bma/maestro branch May 15, 2025 15:14

Ensure the CI is marked as failed when Maestro test is failing #4700

Ensure the CI is marked as failed when Maestro test is failing #4700

Uh oh!

Conversation

bmarty commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Content

Motivation and context

Screenshots / GIFs

Tests

Tested devices

Checklist

Uh oh!

github-actions bot commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bmarty May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bmarty commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bmarty commented May 12, 2025

Uh oh!

bmarty commented May 15, 2025

Uh oh!

jmartinesp commented May 15, 2025

Uh oh!

jmartinesp commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmartinesp left a comment

Choose a reason for hiding this comment

Uh oh!

jmartinesp May 15, 2025

Choose a reason for hiding this comment

Uh oh!

bmarty May 15, 2025

Choose a reason for hiding this comment

Uh oh!

jmartinesp May 15, 2025

Choose a reason for hiding this comment

Uh oh!

bmarty May 15, 2025

Choose a reason for hiding this comment

Uh oh!

bmarty commented May 15, 2025

Uh oh!

Uh oh!

sonarqubecloud bot commented May 15, 2025

Quality Gate passed

Uh oh!

Uh oh!

bmarty commented May 9, 2025 •

edited

Loading

github-actions bot commented May 9, 2025 •

edited

Loading

codecov bot commented May 9, 2025 •

edited

Loading

bmarty May 9, 2025 •

edited

Loading

bmarty commented May 9, 2025 •

edited

Loading

jmartinesp commented May 15, 2025 •

edited

Loading