feat: Plug in Linter framework and lint test framework

[markjschreiber]

Motivation

...
I wanted to create a pluggable mechanism to allow the creation and extension of Lint tests without the need to overload the core Lint.py. I wanted to be able to categorize lints by severity and lint type. I wanted to be able to create simple tests for custom Linters to ensure they find issues as expected without false positives.

Approach

...
Adds a pluggable lint framework to allow the creation of custom linters. Includes a linter testing framework to ensure linters behave as expected. Adds linter severity levels and categories. Adds configuration options to modify and customize what lints are used. Adds docs covering how to create custom linters, how to configure them, patterns used and how to use the linter testing framework. Adds a linter testing example and comprehensive unit tests. Adds an example lint extension package.

Checklist

• Add appropriate test(s) to the automatic suite
• Use make pretty to reformat the code with ruff format
• Use make check to statically check the code using ruff check and mypy
• Send PR from a dedicated branch without unrelated edits
• Ensure compatibility with this project's MIT license

[Copilot]

Pull Request Overview

This PR introduces a pluggable lint framework with severity levels and categories, plus a dedicated testing framework for linters. It updates the CLI, configuration, and documentation to support custom linters and allows filtering by severity or category. Example linters and usage demos are also included.

Reviewed Changes

Copilot reviewed 35 out of 35 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/test_lint_testing_framework.py	Add unit tests for lint-testing helpers (validate_linter, create_test_wdl, assertions)
tests/test_lint_severity.py	Add tests for LintSeverity, LintCategory, and base Linter attributes
tests/test_lint_exit_codes_simple.py	Add CLI exit-code tests for --strict and --exit-on-lint-severity
tests/test_lint_execution.py	Add execution tests covering error handling and linter filtering
tests/test_lint_error_handling.py	Add tests for graceful handling of linter exceptions
tests/test_lint_discovery.py	Add tests for dynamic discovery and filtering of linter plugins
tests/test_lint_config.py	Add tests for config file and environment variable overrides
tests/test_lint_cli_updates.py	Add tests for --list-linters, category filtering, and exit severity behavior
tests/test_lint_cli.py	Add tests for CLI config precedence between env and config files
tests/test_integration_performance.py	Add performance benchmark tests for linting scalability and memory
tests/test_integration_linting.py	Add end-to-end integration tests for discovery, execution, and CLI flags
tests/test_integration_edge_cases.py	Add edge-case tests for malformed, minimal, and deeply nested WDL files
tests/test_example_linters.py	Add unit tests validating the provided example linters package
examples/linter_testing_example.py	Provide example script demonstrating the lint testing framework
examples/example_linters_demo.py	Provide a CLI-driven demo of example linters
examples/example_linters/style_linters.py	Add sample style-focused lint rules
examples/example_linters/security_linters.py	Add sample security-focused lint rules
examples/example_linters/performance_linters.py	Add sample performance-focused lint rules
examples/example_linters/best_practices_linters.py	Add sample best-practice-focused lint rules
examples/example_linters/init.py	Initialize the example linters package and define collections

[mlin]

@markjschreiber Great stuff, thank you! #667

It looks like there are some remaining issues in the integration tests we have for miniwdl check (run with prove -v tests/check.t)

[markjschreiber]

@markjschreiber Great stuff, thank you! #667

It looks like there are some remaining issues in the integration tests we have for miniwdl check (run with prove -v tests/check.t)

Thanks for the review @mlin, good catch, the check.t test had a different expectation about when Lint warnings need de-duplication and I have updated my logic accordingly.

[markjschreiber]

I have setup my dev environment to be more similar to the CICD environment and adjusted my code to ensure the full make test test suite passes.

[coveralls]

Pull Request Test Coverage Report for Build 16452461200

Details

• 310 of 350 (88.57%) changed or added relevant lines in 5 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.3%) to 94.985%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
WDL/Lint.py	154	173	89.02%
WDL/LintPlugins/plugins.py	85	106	80.19%

Totals
Change from base Build 16411827554:	-0.3%
Covered Lines:	7766
Relevant Lines:	8176

---

💛 - Coveralls