suite creation (seems) limited to pwd

[kuykens]

While trying to work with the Matlab devops plugin I was having troule to get my suites filled properly.
The current script seems to collect tests by searching the subfolders from the root using testsuite call.
However, the testsuite call accepts different types of input including paths into packages.
It would be preferable if the plugin would be more versatile in selection of the testscripts into the suite.

[mcafaro]

Thanks for the feedback! Have you tried the (relatively new) selectByFolder setting on RunMATLABTests? It allows you to filter the tests that run down to a specified set of folders:

# Run only the tests in the test/unit folder
- task: RunMATLABTests@0
  inputs:
    selectByFolder: test/unit

Curious if that satisfies your need or you are looking for something more.

[kuykens]

Hi Mcafaro,

Thank you for your response.
My situation is that I have a bunch of tests inside a tree strucrure of packages.
The command suite = testsuite('.', 'IncludeSubfolders', true); does not seem to iterate through these packages when I am at the root of the tree.
As a result the selectByFolder option does not work either as it operates on the (pre-) filled suite.

The testsuite command itself does allow to pass the reference to the package folder:
suite = testsuite('package1.subpackage2') will find all tests in the selected package.

As the current plugin is based on finding tests in the folder tree it just fails.
Is this solvable with the current test plugin or should I rely on the command line interface and call a script that manages the test run. In that case, will I still be able to import the test results XML file?

[kuykens]

sorry, didn't mean to close issue...

[mcafaro]

Ah, I see what you mean. Yes, you will currently need to call a script to manage your test run. Your script can add plugins to the test runner to produce the test results XML file.

It might look something like this:

runAllTests.m

import matlab.unittest.TestRunner;
import matlab.unittest.plugins.XMLPlugin;

% Create the test suite from tests within packages.
suite = testsuite('package1.subpackage2');

% Create a test runner.
runner = TestRunner.withTextOutput('OutputDetail', 3);

% Add a plugin to the runner to produce a JUnit-style XML report.
xmlFile = 'junittestresults.xml';
xmlPlugin = XMLPlugin.producingJUnitFormat(xmlFile);
runner.addPlugin(xmlPlugin);

% Run the suite and assert all tests passed.
results = runner.run(suite);
assertSuccess(results);

Then your Pipelines config might look something like this:

azure-pipelines.yml

pool:
  vmImage: Ubuntu 16.04
steps:
  - task: InstallMATLAB@0
  
  # Run all tests and produce a junittestresults.xml file
  - task: RunMATLABCommand@0
    inputs:
      command: runAllTests
  
  # Publish the junittestresults.xml file
  - task: PublishTestResults@2
    condition: succeededOrFailed()
    inputs:
      testResultsFiles: junittestresults.xml

[acampbel]

Hi Mcafaro,

Thank you for your response.
My situation is that I have a bunch of tests inside a tree strucrure of packages.
The command suite = testsuite('.', 'IncludeSubfolders', true); does not seem to iterate through these packages when I am at the root of the tree.
As a result the selectByFolder option does not work either as it operates on the (pre-) filled suite.

The testsuite command itself does allow to pass the reference to the package folder:
suite = testsuite('package1.subpackage2') will find all tests in the selected package.

As the current plugin is based on finding tests in the folder tree it just fails.
Is this solvable with the current test plugin or should I rely on the command line interface and call a script that manages the test run. In that case, will I still be able to import the test results XML file?

Note I think another option you have is to put your code into a MATLAB project. The project labels feature that enables tests to be labeled in the project works well with tests inside packages. See here.