Introduce code coverage collection to Cypress runs #9084

AMoo-Miki · 2024-12-20T08:50:34Z

Description

Introduce code coverage collection to Cypress runs
Add coverage reporting to Cypress CI
Enable Cypress tests to detect features of the OSD instance they are running against
Turn on test isolation when security is disabled
Add comprehensive testing based on features during CI
Add auto-login to Cypress tests when security is enabled

Changelog

test: Introduce code coverage collection to Cypress runs
test: Enable Cypress tests to detect features of the OSD instance they are running against
test: Turn on test isolation when security is disabled
infra: Add coverage reporting to Cypress CI
infra: Add comprehensive testing based on features during CI
test: Add auto-login to Cypress tests when security is enabled

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

codecov · 2024-12-20T09:11:07Z

Codecov Report

Attention: Patch coverage is 75.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 57.62%. Comparing base (9b5ce68) to head (08bc81f).

Files with missing lines	Patch %	Lines
packages/osd-babel-preset/common_preset.js	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9084      +/-   ##
==========================================
- Coverage   61.02%   57.62%   -3.41%     
==========================================
  Files        3813     4187     +374     
  Lines       91401   114337   +22936     
  Branches    14443    24825   +10382     
==========================================
+ Hits        55782    65887   +10105     
- Misses      32056    45284   +13228     
+ Partials     3563     3166     -397

Flag	Coverage Δ
Linux_1	`29.08% <50.00%> (+<0.01%)`	⬆️
Linux_2	`56.45% <50.00%> (-0.01%)`	⬇️
Linux_3	`38.02% <50.00%> (?)`
Linux_4	`29.02% <0.00%> (-0.01%)`	⬇️
Windows_1	`29.09% <50.00%> (-0.02%)`	⬇️
Windows_2	`56.40% <50.00%> (-0.01%)`	⬇️
Windows_3	`38.03% <50.00%> (+<0.01%)`	⬆️
Windows_4	`29.02% <0.00%> (-0.01%)`	⬇️
cypress:core-opensearch-dashboards:1	`20.97% <100.00%> (?)`
cypress:root:1	`28.93% <100.00%> (?)`
cypress:root:2	`31.97% <100.00%> (?)`
cypress:root:3	`25.25% <100.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

AMoo-Miki · 2025-01-13T21:19:23Z

Talking to @ananzh, cypress:core-opensearch-dashboards:3 is expected to fail due to some data already existing; the tests have to clean up before they start.

Also: * Add coverage reporting to Cypress CI

…running against Also: * Turn on test isolation when security is disabled Signed-off-by: Miki <[email protected]>

Signed-off-by: Miki <[email protected]>

Also: * Add auto-login Signed-off-by: Miki <[email protected]>

Signed-off-by: Miki <[email protected]>

…hen needed Signed-off-by: Miki <[email protected]>

Signed-off-by: Miki <[email protected]>

…issing a `url` Signed-off-by: Miki <[email protected]>

Signed-off-by: Miki <[email protected]>

angle943 · 2025-02-21T19:56:42Z

There's a lot of good stuff in this PR. Unfortunately due to the age of the PR and the tremendous changes we made to the testing files past month, it cannot be merged as is. I will pick the good things that are relevant from this PR and make a fresh PR

angle943 · 2025-02-21T21:37:26Z

package.json

@@ -57,6 +57,8 @@
    "test:ftr": "scripts/use_node scripts/functional_tests",
    "test:ftr:server": "scripts/use_node scripts/functional_tests_server",
    "test:ftr:runner": "scripts/use_node scripts/functional_test_runner",
+    "test:cypress": "env TZ=America/Los_Angeles NO_COLOR=1 yarn cypress run",
+    "test:cypress:open": "env TZ=America/Los_Angeles NO_COLOR=1 yarn cypress open",


we recently changed the timezone to be run on UTC: https://github.com/opensearch-project/OpenSearch-Dashboards/pull/9358/files#diff-1a2b52d55ed1bf51fa9d61ccc1c27d91bc7d0af28b49ec0905966edf07e57c18R20

angle943 · 2025-02-21T21:55:52Z

cypress/utils/commands.js

+ * @param {Partial<Cypress.RequestOptions>} requestOptions
+ */
+
+Cypress.Commands.add('sendOpenSearch', (requestOptions) => {


i think requestOpenSearch is better to conform to cypress's naming of cy.request()

angle943 · 2025-02-21T22:31:16Z

cypress/utils/commands.js

+      'osd-xsrf': true,
+    },
+  });
+});


lets add this change once we have SECURITY_ENABLED tests

angle943 · 2025-02-22T00:19:00Z

cypress/support/auto_login.js

+
+  beforeEach(() => {
+    cy.login();
+  });


we've had discussions as a team about how we want to remove the usage of beforeEach whenever possible, due to the following:

The tests were taking too long

The crashes that were happening due to the duration of the tests

Would a future implementation where this is replaced with before work?

angle943 · 2025-02-26T06:27:23Z

.github/workflows/cypress_tests_workflow.yml

+            cypress/screenshots
+            cypress/videos
+            coverage
+          retention-days: 1


can code coverage work with split test buckets? We currently got all of our query-enhancements test split into 5 buckets due to how long they take

angle943 · 2025-03-03T21:15:09Z

We have recently broken up our tests into five separate buckets due to how long they take (i think total will be around 8 hours). Will cypress test coverage still work without having one test runner run the entire tests?

github-actions bot added the distinguished-contributor label Dec 20, 2024

opensearch-changeset-bot bot added a commit to AMoo-Miki/OpenSearch-Dashboards that referenced this pull request Dec 20, 2024

Changeset file for PR opensearch-project#9084 created/updated

4220384

LDrago27 previously approved these changes Dec 20, 2024

View reviewed changes

AMoo-Miki pushed a commit to AMoo-Miki/OpenSearch-Dashboards that referenced this pull request Dec 20, 2024

Changeset file for PR opensearch-project#9084 created/updated

ec8accc

ananzh previously approved these changes Dec 27, 2024

View reviewed changes

AMoo-Miki dismissed ananzh’s stale review via 1dfcc23 January 6, 2025 19:47

AMoo-Miki force-pushed the cc-2 branch from 280294b to cacbc9c Compare January 12, 2025 05:55

AMoo-Miki pushed a commit to AMoo-Miki/OpenSearch-Dashboards that referenced this pull request Jan 12, 2025

Changeset file for PR opensearch-project#9084 created/updated

b65017b

AMoo-Miki force-pushed the cc-2 branch from cacbc9c to b1f0f96 Compare January 12, 2025 06:42

AMoo-Miki pushed a commit to AMoo-Miki/OpenSearch-Dashboards that referenced this pull request Jan 12, 2025

Changeset file for PR opensearch-project#9084 created/updated

5547451

AMoo-Miki force-pushed the cc-2 branch from dc8c797 to 4cdcf93 Compare January 12, 2025 07:42

AMoo-Miki pushed a commit to AMoo-Miki/OpenSearch-Dashboards that referenced this pull request Jan 12, 2025

Changeset file for PR opensearch-project#9084 created/updated

ffde004

AMoo-Miki force-pushed the cc-2 branch from 726c774 to eb72ad4 Compare January 12, 2025 09:23

AMoo-Miki and others added 12 commits January 13, 2025 18:02

Introduce code coverage collection to Cypress runs

f2af088

Also: * Add coverage reporting to Cypress CI

Enable Cypress tests to detect features of the OSD instance they are …

358a6cf

…running against Also: * Turn on test isolation when security is disabled Signed-off-by: Miki <[email protected]>

Add comprehensive testing based on features during CI

f76be71

Signed-off-by: Miki <[email protected]>

Allow tests to use an array of conditionals

8145838

Also: * Add auto-login Signed-off-by: Miki <[email protected]>

Apply test preconditions

1090630

Signed-off-by: Miki <[email protected]>

Add sendOpenSearch command to Cypress to automatically authenticate w…

6d1c983

…hen needed Signed-off-by: Miki <[email protected]>

Finalize Cypress CI workflow

aba4021

Signed-off-by: Miki <[email protected]>

Use osd-launcher to configure OSD

4a92d68

Signed-off-by: Miki <[email protected]>

Changeset file for PR opensearch-project#9084 created/updated

3d2f10f

Default to openSearchUrl when ENGINE and SECONDARY_ENGINE are m…

f3d7816

…issing a `url` Signed-off-by: Miki <[email protected]>

Fix mapping loader in TestFixtureHandler

d1b8946

Signed-off-by: Miki <[email protected]>

Prepare tests for isolation

08bc81f

Signed-off-by: Miki <[email protected]>

AMoo-Miki force-pushed the cc-2 branch from 6b3ffa0 to 08bc81f Compare January 14, 2025 02:02

angle943 self-assigned this Feb 21, 2025

angle943 reviewed Feb 21, 2025

View reviewed changes

cypress/utils/commands.js

'osd-xsrf': true,

},

});

});

Copy link

Collaborator

angle943 Feb 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets add this change once we have SECURITY_ENABLED tests

angle943 reviewed Feb 22, 2025

View reviewed changes

angle943 mentioned this pull request Feb 22, 2025

Code coverage cypress #9437

Closed

7 tasks

angle943 reviewed Feb 26, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce code coverage collection to Cypress runs #9084

Introduce code coverage collection to Cypress runs #9084

AMoo-Miki commented Dec 20, 2024 •

edited

Loading

codecov bot commented Dec 20, 2024 •

edited

Loading

AMoo-Miki commented Jan 13, 2025

angle943 commented Feb 21, 2025

angle943 Feb 21, 2025

angle943 Feb 21, 2025

angle943 Feb 21, 2025

angle943 Feb 22, 2025

angle943 Feb 26, 2025

angle943 commented Mar 3, 2025

Introduce code coverage collection to Cypress runs #9084

Are you sure you want to change the base?

Introduce code coverage collection to Cypress runs #9084

Conversation

AMoo-Miki commented Dec 20, 2024 • edited Loading

Description

Changelog

Check List

codecov bot commented Dec 20, 2024 • edited Loading

Codecov Report

AMoo-Miki commented Jan 13, 2025

angle943 commented Feb 21, 2025

angle943 Feb 21, 2025

Choose a reason for hiding this comment

angle943 Feb 21, 2025

Choose a reason for hiding this comment

angle943 Feb 21, 2025

Choose a reason for hiding this comment

angle943 Feb 22, 2025

Choose a reason for hiding this comment

angle943 Feb 26, 2025

Choose a reason for hiding this comment

angle943 commented Mar 3, 2025

AMoo-Miki commented Dec 20, 2024 •

edited

Loading

codecov bot commented Dec 20, 2024 •

edited

Loading