Skip to content

Print percent of *tests* where element of a model coverage is involved #95

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
buzden opened this issue Oct 5, 2023 · 0 comments
Open
Labels
code: enhancement New feature or improvement part: model coverage Related to model coverage of generators status: discussion Suggested or reported thing is not obvious to be good enough status: feature request Request for new functionality or improvement status: good first issue Good for newcomers

Comments

@buzden
Copy link
Owner

buzden commented Oct 5, 2023

Currently, after the #94, count of presence in a model coverage is printed along with coverage status.

Now it looks like this (takes from test here):

PrintCoverage.Y covered partially (100 times)
  - Y1: covered (41 times)
  - Y2: not covered
  - Y3: covered (59 times)

There were 100 tests generated, Y was used in each test once, thus the whole count of usages of the type Y is 100% for the tests, where individual actually generated constructors have 41% and 59% respectively.

But when type is recursive, or just is used several times, one can get percentage >100% (taken from another test here):

PrintCoverage.X covered partially (118 times)
  - X1: covered (54 times)
  - X2: not covered
  - X3: covered (64 times)

PrintCoverage.Y covered partially (100 times)
  - Y1: covered (41 times)
  - Y2: not covered
  - Y3: covered (59 times)

Actually, type X is present in 59% of the tests, along with the Y3 constructor, but each time it is present twice. If we were printing this with a percentage, 118% near to the X type would be a misleading number, where "present in 59% of tests" could be feasible one.

So, it is suggested to add percentage of presence in the tests, but we must be accurate and (unlike the current times coverage) to not to count each model coverage element several times in a single test.

The current design does not support this directly, we don't store this type of data. There are at least two possible solutions to this:

  • change the order of ModelCoverage collection so that each test uses registerCoverage independently, and thus is can be changed to calculate percentage of mentions of types and constructors per test; I don't like this solution because of potential high count of calls to relatively expensive registerCoverage function;

  • to have two separate Semigroups implemented for the ModelCoverage type: one is for use with the Writer inside appropriate unGen which does not accumulate the usage-per-test metric, and is not exported outside the module; and the other which does accumulate usage-per-test metric and is exported outside; the latter one should be declared in the bottom of the module to not be used accidentally in the Writer context.

@buzden buzden added code: enhancement New feature or improvement status: good first issue Good for newcomers status: discussion Suggested or reported thing is not obvious to be good enough status: feature request Request for new functionality or improvement part: model coverage Related to model coverage of generators labels Oct 5, 2023
@buzden buzden changed the title Print percent of _tests_ where element of a model coverage is involved Print percent of *tests* where element of a model coverage is involved Oct 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code: enhancement New feature or improvement part: model coverage Related to model coverage of generators status: discussion Suggested or reported thing is not obvious to be good enough status: feature request Request for new functionality or improvement status: good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

1 participant