-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Introduce code coverage collection to Cypress runs #9084
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
AMoo-Miki
wants to merge
12
commits into
opensearch-project:main
Choose a base branch
from
AMoo-Miki:cc-2
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
f2af088
Introduce code coverage collection to Cypress runs
358a6cf
Enable Cypress tests to detect features of the OSD instance they are …
f76be71
Add comprehensive testing based on features during CI
8145838
Allow tests to use an array of conditionals
1090630
Apply test preconditions
6d1c983
Add sendOpenSearch command to Cypress to automatically authenticate w…
aba4021
Finalize Cypress CI workflow
4a92d68
Use osd-launcher to configure OSD
3d2f10f
Changeset file for PR #9084 created/updated
opensearch-changeset-bot[bot] f3d7816
Default to `openSearchUrl` when `ENGINE` and `SECONDARY_ENGINE` are m…
d1b8946
Fix mapping loader in TestFixtureHandler
08bc81f
Prepare tests for isolation
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,297 @@ | ||
|
||
name: Cypress 12 experimental tests | ||
|
||
# trigger on every commit push and PR for all branches except pushes for backport branches | ||
on: | ||
push: | ||
branches: ['main', '[0-9]+\.x', '[0-9]+\.[0-9]+'] # Run the functional test on push for only release branches | ||
paths-ignore: | ||
- '**/*.md' | ||
- 'docs/**' | ||
- '.lycheeignore' | ||
- 'CODEOWNERS' | ||
- 'changelogs/fragments/**' | ||
pull_request: | ||
branches: ['**'] | ||
paths-ignore: | ||
- '**/*.md' | ||
- 'docs/**' | ||
- '.lycheeignore' | ||
- 'CODEOWNERS' | ||
- 'changelogs/fragments/**' | ||
|
||
env: | ||
CYPRESS_BROWSER: 'chromium' | ||
CYPRESS_username: 'admin' | ||
NODE_OPTIONS: '--max-old-space-size=6144 --dns-result-order=ipv4first' | ||
COVERAGE: true | ||
PROJECT_ARTIFACT_NAME: 'project-artifact' | ||
COMBINATIONS: | | ||
[ | ||
[], | ||
["SECURITY"], | ||
["WORKSPACE", "DATA_SOURCE", "QUERY_ENHANCEMENTS", "USE_NEW_HOME_PAGE"] | ||
] | ||
|
||
jobs: | ||
build-and-validate: | ||
name: Build and validate | ||
runs-on: ubuntu-latest | ||
steps: | ||
- name: Checkout code | ||
uses: actions/checkout@v4 | ||
|
||
- name: Setup Node | ||
uses: actions/setup-node@v4 | ||
with: | ||
node-version-file: '.nvmrc' | ||
registry-url: 'https://registry.npmjs.org' | ||
|
||
- name: Setup Yarn | ||
run: | | ||
npm uninstall -g yarn | ||
npm i -g [email protected] | ||
yarn config set network-timeout 1000000 -g | ||
|
||
- name: Run bootstrap | ||
run: yarn osd bootstrap | ||
|
||
- name: Check for yarn.lock changes | ||
run: | | ||
if [[ `git status --porcelain yarn.lock` ]]; then | ||
echo -e "\033[31mThe yarn.lock file is out of sync!\033[0m" | ||
git diff | ||
exit 1 | ||
fi | ||
|
||
- name: Generate dev docs | ||
run: yarn docs:generateDevDocs | ||
|
||
- name: Check for dev docs changes | ||
run: | | ||
if [[ `git status --porcelain docs/_sidebar.md` ]]; then | ||
echo -e "\033[31mThe dev docs are out of sync; run yarn docs:generateDevDocs and amend the PR.\033[0m" | ||
git diff | ||
exit 1 | ||
fi | ||
|
||
- name: Run linter | ||
id: linter | ||
run: yarn lint | ||
|
||
- name: Validate NOTICE file | ||
id: notice-validate | ||
run: yarn notice:validate | ||
|
||
- name: Validate licenses | ||
id: i18n-licenses | ||
run: yarn checkLicenses | ||
|
||
- name: Check i18n | ||
id: i18n-check | ||
run: yarn i18n:check | ||
|
||
- name: Build plugins | ||
run: node scripts/build_opensearch_dashboards_platform_plugins --no-examples --workers 12 | ||
|
||
# upload-artifact looses permissions; we need to tar it to maintain them | ||
- name: Archive content | ||
run: tar -czf /tmp/osd-build.tar.gz --exclude=.git --exclude=.yarn . | ||
|
||
- name: Upload artifact | ||
uses: actions/upload-artifact@v4 | ||
with: | ||
name: ${{ env.PROJECT_ARTIFACT_NAME }} | ||
path: /tmp/osd-build.tar.gz | ||
retention-days: 1 | ||
|
||
prepare-cypress-matrix: | ||
name: Prepare Cypress tests | ||
needs: build-and-validate | ||
runs-on: ubuntu-latest | ||
outputs: | ||
test-matrix: ${{ steps.create-list.outputs.list }} | ||
steps: | ||
- name: Checkout code | ||
uses: actions/checkout@v4 | ||
|
||
- name: Trigger tests for combinations | ||
id: create-list | ||
run: | | ||
declare -a inputs_array | ||
|
||
process_spec() { | ||
local spec_path="$1" | ||
local spec_name="$2" | ||
local counter=1 | ||
|
||
while IFS= read -r combo; do | ||
echo "Preparing tests for $spec_name with combination $counter: $combo" | ||
|
||
input_object=$(jq -n \ | ||
--arg spec_pattern "$spec_path" \ | ||
--arg run_flag "${spec_name}:${counter}" \ | ||
--argjson combo "$combo" \ | ||
'{ | ||
spec_pattern: $spec_pattern, | ||
run_flag: $run_flag | ||
} + ($combo | to_entries | map({key: .value, value: "true"}) | from_entries)') | ||
|
||
echo "Generated input_object:" | ||
echo "$input_object" | jq . | ||
|
||
inputs_array+=("$input_object") | ||
echo "Current size of inputs_array: ${#inputs_array[@]}" | ||
|
||
counter=$((counter + 1)) | ||
done < <(echo "$COMBINATIONS" | jq -c '.[]') | ||
} | ||
|
||
# Process spec files at the root | ||
root_specs=$(find ./cypress/integration -maxdepth 1 -name "*.spec.js") | ||
if [ -n "$root_specs" ]; then | ||
echo "Found root specs: $root_specs" | ||
process_spec "cypress/integration/*.spec.js" "root" | ||
fi | ||
|
||
# Process spec folders | ||
for folder in ./cypress/integration/*/; do | ||
if [ -d "$folder" ]; then | ||
folder_name=$(basename "$folder") | ||
echo "Processing folder: $folder_name" | ||
folder_specs=$(find "$folder" -name "*.spec.js") | ||
if [ -n "$folder_specs" ]; then | ||
echo "Found specs in $folder_name: $folder_specs" | ||
process_spec "cypress/integration/$folder_name/**/*.spec.js" "$folder_name" | ||
fi | ||
fi | ||
done | ||
|
||
if [ ${#inputs_array[@]} -eq 0 ]; then | ||
echo "No test configurations generated!" | ||
json_output='{"include":[]}' | ||
else | ||
json_output=$(printf '%s\n' "${inputs_array[@]}" | jq -s '{"include": .}') | ||
fi | ||
|
||
# Escape the JSON string for GitHub Actions | ||
escaped_json_output=$(echo "$json_output" | jq -c -r @json) | ||
echo "list=$escaped_json_output" >> $GITHUB_OUTPUT | ||
|
||
|
||
run-tests: | ||
name: Run tests (cypress:${{ matrix.run_flag }}) | ||
needs: [prepare-cypress-matrix, build-and-validate] | ||
runs-on: ubuntu-latest | ||
strategy: | ||
matrix: ${{ fromJson(needs.prepare-cypress-matrix.outputs.test-matrix) }} | ||
fail-fast: false | ||
steps: | ||
- name: Display run details | ||
run: | | ||
echo "Flag: ${{ matrix.run_flag }}" | ||
echo "Spec pattern: ${{ matrix.spec_pattern }}" | ||
echo "Security enabled: ${{ matrix.SECURITY || 'false' }}" | ||
echo "Workspace enabled: ${{ matrix.WORKSPACE || 'false' }}" | ||
echo "Data Source enabled: ${{ matrix.DATA_SOURCE || 'false' }}" | ||
echo "Query Enhancements enabled: ${{ matrix.QUERY_ENHANCEMENTS || 'false' }}" | ||
echo "New Home enabled: ${{ matrix.USE_NEW_HOME_PAGE || 'false' }}" | ||
|
||
- name: Checkout code | ||
uses: actions/checkout@v4 | ||
|
||
- name: Download built artifact | ||
uses: actions/download-artifact@v4 | ||
with: | ||
name: ${{ env.PROJECT_ARTIFACT_NAME }} | ||
path: . | ||
|
||
- name: Extract archive | ||
run: | | ||
tar -xzf osd-build.tar.gz --overwrite --ignore-command-error | ||
rm osd-build.tar.gz | ||
|
||
- name: Setup Node | ||
uses: actions/setup-node@v4 | ||
with: | ||
node-version-file: '.nvmrc' | ||
registry-url: 'https://registry.npmjs.org' | ||
|
||
- name: Setup Yarn | ||
run: | | ||
npm uninstall -g yarn | ||
npm i -g [email protected] | ||
yarn config set network-timeout 1000000 -g | ||
|
||
- name: Setup JDK | ||
uses: actions/setup-java@v4 | ||
with: | ||
java-version: '21' | ||
distribution: 'corretto' | ||
|
||
- name: Get package version | ||
run: | | ||
echo "OSD_VERSION=$(yarn --silent pkg-version)" >> $GITHUB_ENV | ||
|
||
- name: Generate random password | ||
run: | | ||
CYPRESS_password=$(openssl rand -base64 20) | ||
echo "CYPRESS_password=$CYPRESS_password" >> $GITHUB_ENV | ||
|
||
- name: Setup OpenSearch and Dashboards | ||
run: | | ||
npm i osd-launcher -g | ||
|
||
if [ "${{ matrix.SECURITY }}" = "true" ]; then | ||
osd-launcher -os ${{ env.OSD_VERSION }} -osd . -p ${{ env.CYPRESS_password }} --no-plugins --no-build --destination ~/ | ||
else | ||
osd-launcher -os ${{ env.OSD_VERSION }} -osd . --no-security --no-plugins --no-build --destination ~/ | ||
fi | ||
|
||
- name: Run OpenSearch | ||
run: | | ||
# Launch and forget | ||
/bin/bash -c ~/OpenSearch-v${{ env.OSD_VERSION }}/bin/opensearch & | ||
|
||
- name: Run OSD | ||
run: | | ||
WORKSPACE_FLAG=$([ "${{ matrix.WORKSPACE }}" = "true" ] && echo "--workspace.enabled=true" || echo "--workspace.enabled=false") | ||
DATA_SOURCE_FLAG=$([ "${{ matrix.DATA_SOURCE }}" = "true" ] && echo "--data_source.enabled=true" || echo "--data_source.enabled=false") | ||
QUERY_ENHANCEMENTS_FLAG=$([ "${{ matrix.QUERY_ENHANCEMENTS }}" = "true" ] && echo "--uiSettings.overrides["query:enhancements:enabled"]=true" || echo "--uiSettings.overrides["query:enhancements:enabled"]=false") | ||
USE_NEW_HOME_PAGE_FLAG=$([ "${{ matrix.USE_NEW_HOME_PAGE }}" = "true" ] && echo "--uiSettings.overrides["home:useNewHomePage"]=true" || echo "--uiSettings.overrides["home:useNewHomePage"]=false") | ||
|
||
# Launch and forget | ||
/bin/bash -c "node scripts/opensearch_dashboards --dev --no-base-path --no-watch --savedObjects.maxImportPayloadBytes=10485760 --server.maxPayloadBytes=1759977 --logging.json=false --data.search.aggs.shardDelay.enabled=true $WORKSPACE_FLAG $DATA_SOURCE_FLAG $QUERY_ENHANCEMENTS_FLAG $USE_NEW_HOME_PAGE_FLAG" & | ||
|
||
- name: Wait for 15 seconds | ||
run: sleep 15s | ||
shell: bash | ||
|
||
- name: Run tests | ||
run: yarn test:cypress --spec "${{ matrix.spec_pattern }}" | ||
|
||
- name: Upload coverage | ||
id: upload-code-coverage | ||
uses: codecov/codecov-action@v5 | ||
with: | ||
token: ${{ secrets.CODECOV_TOKEN }} | ||
directory: ./coverage | ||
flags: "cypress:${{ matrix.run_flag }}" | ||
|
||
- name: Generate safe artifact name | ||
if: always() | ||
id: safe-name | ||
run: | | ||
value=$(echo "${{ matrix.run_flag }}" | sed -E 's/[^a-z0-9_-]+/_/gi' | sed -E 's/^_|_$//g') | ||
echo "value=${value}" >> $GITHUB_OUTPUT | ||
|
||
- name: Upload test artifacts | ||
if: always() | ||
uses: actions/upload-artifact@v4 | ||
with: | ||
name: cypress_artifact_${{ steps.safe-name.outputs.value }} | ||
path: | | ||
cypress/screenshots | ||
cypress/videos | ||
coverage | ||
retention-days: 1 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
test: | ||
- Introduce code coverage collection to Cypress runs ([#9084](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/9084)) | ||
- Enable Cypress tests to detect features of the OSD instance they are running against ([#9084](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/9084)) | ||
- Turn on test isolation when security is disabled ([#9084](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/9084)) | ||
- Add auto-login to Cypress tests when security is enabled ([#9084](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/9084)) | ||
|
||
infra: | ||
- Add coverage reporting to Cypress CI ([#9084](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/9084)) | ||
- Add comprehensive testing based on features during CI ([#9084](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/9084)) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can code coverage work with split test buckets? We currently got all of our
query-enhancements
test split into 5 buckets due to how long they takeThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes.