Generate coverage value for both unit and E2E tests #6690

cnotv · 2022-08-18T14:29:05Z

Summary

Fixes #6501
Add coverage report for Rancher Dashboard using Codecov.

Occurred changes and/or fixed issues

Correct existing Jest configuration
Add configuration for E2E with Cypress
Merge coverage values for both the tests
Integrate and publish to Codecov GHA
Upload coverage using flags for each type: unit, e2e, merged
Added documentation about coverage
Correct documentation tree

Technical notes summary

Added text-summary report for both the tests
Corrected Sorry Cypress group-id value with run number and attempt, to provide unique values
Added instrumentalization for Nuxt within Babel configuration, required for generating coverage
Replaced existing NYC configuration with own file .nycrc to increase visibility and simplify reviews
Moved commands interface inside its own global distributed file
Fixed issue with /shell/scripts/ folder not excluded from tests and generating non breaking errors in the logs
Added @cypress/code-coverage library for integrating coverage with Cypress
Added babel-plugin-istanbul library for transpiling code to generate instrumentation through Babel

Notes

Coverage files are copied under another name, as they are required to be in the same folder
The folder defined in the path of the artifact action is now omitted in GHA Omitting the enclosing directory actions/upload-artifact#30
Coverage is collected also on PR as they may be analyzed separately and generate a delta

Areas or cases that should be tested

Values should be displayed on Rancher Dashboard Codecov page.
Example on my fork with already merged branch.

Areas which could experience regressions

Existing pipeline, no actual check is needed.

Screenshot/Video

https://app.codecov.io/gh/rancher/dashboard/pulls

cnotv · 2022-08-18T15:35:48Z

It seems like we are not allowed to use Codecov on this repository, which makes exclude the tests from GHA.
Could we take a look @nwmac ?

I had no problem to use that configuration on my fork and PR.

cnotv · 2022-08-22T06:56:23Z

Action added to the repository and PR can be now reviewed.

codecov-commenter · 2022-08-22T07:17:54Z

Codecov Report

❗ No coverage uploaded for pull request base (master@3812fa5). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #6690   +/-   ##
=========================================
  Coverage          ?   16.50%           
=========================================
  Files             ?     1094           
  Lines             ?    41821           
  Branches          ?    10044           
=========================================
  Hits              ?     6902           
  Misses            ?    34919           
  Partials          ?        0

Flag	Coverage Δ
e2e	`43.16% <0.00%> (?)`
merged	`16.50% <0.00%> (?)`
unit	`3.60% <0.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

cnotv · 2022-08-22T15:15:48Z

I take the chance to correct the (very!) latest changes about testing in the documentation.

richard-cox

One question and a couple of minor comments.

I looked at https://app.codecov.io/gh/rancher/dashboard and couldn't see anything in coverage, will that start displaying when things get in to master? I noticed there was an entry in https://app.codecov.io/gh/rancher/dashboard/pulls, though clicking on that gives some kind of 404 page

cypress.config.ts

jest.config.js

cnotv · 2022-08-22T20:14:04Z

@richard-cox commits diff can be seen only after merge. PR are not working probably due the checkout in the GHA. I'm going to disable it to see if it changes, as that caused problems also somewhere else, e.g. Sorry Cypress.

cnotv · 2022-08-23T07:26:03Z

I'd to merge the conflicts for yarn.lock already 3 times, we may consider push #6035 for it.

cnotv · 2022-08-23T18:27:04Z

I took the chance to upload separated coverage, so they can be analyzed separately.

cnotv · 2022-08-23T18:28:34Z

@richard-cox the 404 for the PR is due to the lack of coverage on master. It's explained here that to generate PR dashboards it requires a base and a head of the request.

In my fork you can see that a second PR is actually working fine.

richard-cox

Regarding the sum of coverage being lower than the sum of e2e and unit test coverage (https://app.codecov.io/gh/cnotv/dashboard/compare/4/overview .. e2e 43.11% + unit 3.61% = 16.48%)

Looking at the e2e (cypress,config.ts) and unit test ('jest.config.js) config it looks like the codeCoverageandcollectCoverageFromvalues should be pretty similar .. except thatcypress//.and/tests/**/.` are not explicitly ignored in jest. I don't think this though would account for the @20% drop
above both config's there's a line that might disable the codeCoverage/collectCoverageFrom config
- coverage: false,
- collectCoverage: false,
it's super hard to tell from the json output, but is there anything obvious in the unit test one (like cypress or shell/scripts) that is there that shouldn't be?
Could nyc config like from https://github.com/cypress-io/code-coverage#exclude-files-and-folders help solve the problem?

yarn.lock

cnotv · 2022-08-31T13:56:21Z

Looking at the e2e (cypress,config.ts) and unit test ('jest.config.js) config it looks like the codeCoverageandcollectCoverageFromvalues should be pretty similar .. except thatcypress//.and/tests/**/.` are not explicitly ignored in jest. I don't think this though would account for the @20% drop

I think is computed with the weight of tests we have.
Unit tests we have 400 tests.
E2E we have 18 tests active of 24.

Also if you check the configuration it is actually ignored for searching tests, based on your previous suggestion. On top of that, the configuration uses an allowing list, instead of an blocking list.

  modulePathIgnorePatterns: [
    '<rootDir>/cypress/',
    '<rootDir>/scripts/',
    '<rootDir>/docusaurus/',
    '<rootDir>/stories/',
    '<rootDir>/shell/scripts/',
  ],

above both config's there's a line that might disable the codeCoverage/collectCoverageFrom config
coverage: false,
collectCoverage: false,

This is intended and should stay like this, otherwise it generates coverage at every test even locally.

it's super hard to tell from the json output, but is there anything obvious in the unit test one (like cypress or shell/scripts) that is there that shouldn't be?

Based on the HTML output and the configuration, there's nothing else as far as I am aware of.

Could nyc config like from https://github.com/cypress-io/code-coverage#exclude-files-and-folders help solve the problem?

Which problem? I think we may use the same configuration of Jest and Cypress, as we already do.

richard-cox

There are still merge conflicts in yarn.lock. Interestingly the e2e tests pass still

yarn install v1.22.19
info Merge conflict detected in yarn.lock and successfully merged.

https://github.com/rancher/dashboard/runs/8119029734?check_suite_focus=true#step:4:2

cnotv · 2022-09-08T18:36:34Z

I may finish the yarn.lock issue by the time we merge this at that point 😬

…lback for commits

cnotv · 2022-09-28T19:08:19Z

@richard-cox Generated new yarn.lock solving conflicts issues.

nwmac

LGTM

yarn.lock has been adressed

* Correct Jest coverage configuration * Add coverage upload * Exclude operationsl scripts from linting * Rephrase title based on pull request title, run and attempt, with fallback for commits * Add instrumentation for E2E coverage report * Add merge logic to the coverage files * Add instrumentalization for Cypress to generate coverage for E2E * Download artifacts before merge coverage in CI * Merge fix * Copy coverage files before artifact upload * Add summary after running tests * Change restriction rules about coverage for pull_request and push * Add documentation for the coverage * Correct test documentation folder restructuring * Add inclusion list for Cypress coverage; add scripts as exclusion * Extend ignored path to be parsed for Jest to improve performances * Workflow cleanup * Checkout current branch head on test runs * Submit separated coverage with different flags

cnotv added this to the v2.7.0 milestone Aug 18, 2022

cnotv requested a review from richard-cox August 18, 2022 14:29

cnotv self-assigned this Aug 18, 2022

cnotv changed the title ~~Feature/6501 coverage ops~~ Generate coverage value for both unit and E2E tests Aug 18, 2022

cnotv force-pushed the feature/6501-coverage-ops branch from 04ec511 to 932d250 Compare August 22, 2022 06:55

cnotv force-pushed the feature/6501-coverage-ops branch 2 times, most recently from 85e0832 to ccde634 Compare August 22, 2022 13:59

cnotv force-pushed the feature/6501-coverage-ops branch from ccde634 to 4ac6940 Compare August 22, 2022 15:22

richard-cox requested changes Aug 22, 2022

View reviewed changes

cypress.config.ts Show resolved Hide resolved

jest.config.js Show resolved Hide resolved

cnotv force-pushed the feature/6501-coverage-ops branch 8 times, most recently from 78f4cad to 4ddb126 Compare August 23, 2022 17:34

cnotv requested a review from richard-cox August 23, 2022 18:28

cnotv force-pushed the feature/6501-coverage-ops branch from 4ddb126 to c0a8674 Compare August 26, 2022 08:12

richard-cox requested changes Aug 31, 2022

View reviewed changes

yarn.lock Outdated Show resolved Hide resolved

cnotv force-pushed the feature/6501-coverage-ops branch from c0a8674 to 5fbf95b Compare August 31, 2022 16:16

cnotv requested a review from richard-cox August 31, 2022 16:17

richard-cox previously requested changes Sep 8, 2022

View reviewed changes

cnotv added 19 commits September 28, 2022 20:42

Correct Jest coverage configuration

b0a320b

Add coverage upload

00bd707

Exclude operationsl scripts from linting

cb694f9

Rephrase title based on pull request title, run and attempt, with fal…

d0bd11a

…lback for commits

Add instrumentation for E2E coverage report

0b26b48

Add merge logic to the coverage files

f200cd1

Add instrumentalization for Cypress to generate coverage for E2E

2f82131

Download artifacts before merge coverage in CI

5210b1a

Merge fix

a0f8ae9

Copy coverage files before artifact upload

59bf4d6

Add summary after running tests

3bed03f

Change restriction rules about coverage for pull_request and push

0e65693

Add documentation for the coverage

7a9d78e

Correct test documentation folder restructuring

60b3386

Add inclusion list for Cypress coverage; add scripts as exclusion

d93f39f

Extend ignored path to be parsed for Jest to improve performances

aa77205

Workflow cleanup

a9cd371

Checkout current branch head on test runs

bb4acd4

Submit separated coverage with different flags

d288161

cnotv force-pushed the feature/6501-coverage-ops branch from 5fbf95b to d288161 Compare September 28, 2022 19:06

nwmac approved these changes Sep 30, 2022

View reviewed changes

cnotv merged commit 1c6b5ac into rancher:master Sep 30, 2022

Generate coverage value for both unit and E2E tests #6690

Generate coverage value for both unit and E2E tests #6690

Uh oh!

Conversation

cnotv commented Aug 18, 2022 • edited by zube bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Occurred changes and/or fixed issues

Technical notes summary

Notes

Areas or cases that should be tested

Areas which could experience regressions

Screenshot/Video

Uh oh!

cnotv commented Aug 18, 2022

Uh oh!

cnotv commented Aug 22, 2022

Uh oh!

codecov-commenter commented Aug 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cnotv commented Aug 22, 2022

Uh oh!

richard-cox left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cnotv commented Aug 22, 2022

Uh oh!

cnotv commented Aug 23, 2022

Uh oh!

cnotv commented Aug 23, 2022

Uh oh!

cnotv commented Aug 23, 2022

Uh oh!

richard-cox left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cnotv commented Aug 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

richard-cox left a comment

Choose a reason for hiding this comment

Uh oh!

cnotv commented Sep 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cnotv commented Sep 28, 2022

Uh oh!

nwmac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cnotv commented Aug 18, 2022 •

edited by zube bot

Loading

codecov-commenter commented Aug 22, 2022 •

edited

Loading

cnotv commented Aug 31, 2022 •

edited

Loading

cnotv commented Sep 8, 2022 •

edited

Loading