Watch mode

[jason0x43]

The goal of this issue is to suggest how a "watch" mode would behave for Intern.

"Watch" mode is where Intern is started and stays running, watching for changes to files. When it sees changes, it will re-execute tests. In a naive implementation, Intern could just call intern.run() every time something changed. A more advanced implementation would only re-run tests that had changed, or tests that were related to changed files.

Some questions to answer:

• What would typical usage patterns be?
• Would watch mode work for functional tests as well as unit tests?
• How would it work in different situations? (e.g., bare JS vs compiled flows)
• What changes would Intern need to support a basic watch mode?
• What changes would Intern need to be able to run only changed tests?
• What changes would Intern need to be able to determine which tests relate to which application files?

[jason0x43]

There hasn't been a lot going on here, so I'll take a stab at this. Note that these are just my thoughts based on my own experiences with watch mode. Other viewpoints are encouraged.

Typical usage patterns

At a basic level, Intern's watch mode should work similarly to Typescript's or webpack's. A user will start Intern. Intern will watch some set of files for changes. When changes occur, Intern will run tests.

Intern should be able to watch test files and application files. Ideally, Intern should be able to decide which files to watch based on the suites specified in the test config and their dependency trees. However, an initial implementation should at least allow the user to explicitly indicate which files to watch.

As far as running tests goes, Intern should at a basic level be able to rerun all the tests when something changes. A better implementation would only run changed suites. The best implementation would have a dependency graph for the project, and would be able to run suites related to whatever changed, whether it was an application file, a test support file, or a suite file.

This re-run functionality should also extend to the browser. Intern should be able to start a remote session in watch mode. Unlike normal remote sessions, which run tests and then close, this session should stay open and wait for instructions from an Intern host. The intern host will tell the remote to run tests when necessary.

Functional vs unit tests

In general, watch mode is probably most valuable for unit tests as functional tests tend to be slower to start and execute. You never know, though, so Intern's watch mode should allow for watching and rerunning functional tests. It should default to only unit tests, but provide an option to also execute functional tests.

How would it work for different dev flows

Flows that don't require building are the most straightforward to handle. Intern will be started in watch mode and told to watch the tests, and possibly the app source. When something changes, Intern reloads the affected code and re-runs tests.

Flows that do require building will probably operate in a similar fashion. However, Intern will watch the built test and app files rather than the source. An external build watcher, such as webpack or tsc, will need to be started to update the built files when the sources change.

Changes needed for basic watch mode

Intern needs a way to watch files. Probably the most common library for that purpose is chokidar. Intern inherited a basic chokidar setup when the cli source was added to the repo. Usage of chokidar may need to be moved to the Node executor, or a support module used by the Node executor, assuming it's decided that watching should be a responsibility of core Intern rather than just the CLI.

Changes needed to run only changed tests

At a coarse level, Intern can watch for changed suites. If a suite changes, Intern re-runs it.

As an extension to this, Intern should also re-run a suite if any of its dependencies change. For example, suites may import modules with testing utility functions. If those support modules are updated, affected suites should also be re-run. This will require Intern to generate dependency graphs for the test modules.

At a finer level, Intern could inspect a changed suite to determine which tests inside it were changed, and run only those tests. This would likely require Intern to be able to

1. generate ASTs from test files and cache those ASTs for later comparisons
2. identify tests within an AST
3. diff two ASTs to determine which tests changed

Changes needed to determine test relationships to application files

Intern would need to be able to generate dependency graphs for the test and application modules. Any time a module is changed, Intern would find all the connected tests and re-run them.

[maxbeatty]

Agree watch mode should default to only unit tests. We would unlikely never execute functional tests while watching because of the interruption to development locally (browsers launching in foreground) and queueing consequences remotely (BrowserStack in our case).

We are writing our tests in TypeScript which, as you alluded to, will add to the complexity of adopting watch mode.

[aciccarello]

One feature of the jest watch mode (related to #1021 Interactivity) is that after tests complete, the user has the option of modifying what jest is watching. Something like this makes it easy to trigger a test run if Intern doesn't realize that a test may have been changed.

[rhpijnacker]

Cool stuff!
We've been considering some code-coverage based approach to do "on a need" basis retesting.
It would be awesome if Intern could do that built-in in watch mode.

[rhpijnacker]

We typically write our unit tests and use F5 (browser reload) in a TDD way.
Although this is not functionalTesting, it requires to have the browser open. I think enabling this in watch mode would be pretty sweet.