Skip to content

Add script to validate code examples in sniff documentation #185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from

Conversation

rodrigoprimo
Copy link
Collaborator

@rodrigoprimo rodrigoprimo commented May 30, 2025

This PR introduces a new script that verifies code examples in PHPCS sniff documentation. The script checks that "Valid" examples don't trigger the sniff and "Invalid" examples do trigger the sniff, ensuring code examples correctly demonstrate the behavior of the sniff.

Noteworthy implementation decisions:

  • The script will be available in the Composer vendor/bin directory as phpcs-check-sniff-doc-code-examples.
  • To check for syntax errors in the code examples, the code uses token_get_all() with the flag TOKEN_PARSE. This means that PHP >= 7.0 is required to run this script while the rest of the repository requires PHP >= 5.4. A requirement check was added to phpcs-check-sniff-doc-code-examples to alert users if they are running PHP < 7.0, and a warning was added to the top of the file informing that phpcs-check-sniff-doc-code-examples must remain compatible with PHP 5.4 for the requirement check to work.
  • Since the code uses DOM and libxml, those two PHP extensions were added as requirements to the composer.json file.
  • The script is unable to differentiate between multiple code examples in the same <code> block. This means that for invalid examples, the script might miss ones that don't trigger the sniff if, in the same <code> block, there is at least one example that does trigger the sniff.
  • The tests for this script run on a separate GitHub action job as they need to be tested against a different set of PHP versions than the rest of the tests in this repository, and they need to be tested against multiple PHPCS versions and OSes. The DebugSniff tests need to be tested against multiple PHPCS versions but not multiple OSes, while the tests for the other tools need to be tested against multiple OSes but not multiple PHPCS versions.

How it works

The script:

  1. Finds XML documentation files for PHPCS sniffs based on the parameters passed in the command line. If no path is specified, it searches the directory where the script was executed and its sub-directories.
  2. Extracts code blocks marked as "Valid" or "Invalid" from these files
  3. Programatically calls PHPCS on these code blocks to verify that:
    • Valid examples don't trigger the sniff (no violations)
    • Invalid examples do trigger the sniff (have violations)
  4. Reports any discrepancies

How to use the script

  • Run it from the root directory of a PHPCS standard:
$ /path/to/script/phpcs-check-sniff-doc-code-examples

The script will search for all sniff documentation files and check their code examples. For other ways to run the script check the help page:

$ /path/to/script/phpcs-check-sniff-doc-code-examples --help

Related issues

Closes #142

This commit moves the code related to checking color and formatting the help text to a new helper class. This way it can be reused when the new DocCodeExamples script is introduced.

Tests were added for the new HelpTextFormatter::format() method.
This commit introduces a new script that verifies code examples in PHPCS sniff documentation.
The script checks that "Valid" examples don't trigger the sniff and "Invalid" examples do trigger
the sniff, ensuring code examples correctly demonstrate the behavior of the sniff.

Noteworthy implementation decisions:
- The script will be available in the Composer vendor/bin directory as `phpcs-check-sniff-doc-code-examples`.
- To check for syntax errors in the code examples, the code uses `token_get_all()` with the flag TOKEN_PARSE. This means that PHP >= 7.0 is required to run this script while the rest of the repository requires PHP >= 5.4. A requirement check was added to `phpcs-check-sniff-doc-code-examples` to alert users if they are running PHP < 7.0, and a warning was added to the top of the file informing that `phpcs-check-sniff-doc-code-examples` must remain compatible with PHP 5.4 for the requirement check to work.
- Since the code uses DOM and libxml, those two PHP extensions were added as requirements to the composer.json file.
- The script is unable to differentiate between multiple code examples in the same <code> block. This means that for invalid examples, the script might miss ones that don't trigger the sniff if, in the same <code> block, there is at least one example that does trigger the sniff.
- The tests for this script run on a separate GitHub action job as they need to be tested against a different set of PHP versions than the rest of the tests in this repository, and they need to be tested against multiple PHPCS versions and OSes. The DebugSniff tests need to be tested against multiple PHPCS versions but not multiple OSes, while the tests for the other tools need to be tested against multiple OSes but not multiple PHPCS versions.

How to use the script:

- Run it from the root directory of a PHPCS standard:

```
$ /path/to/script/phpcs-check-sniff-doc-code-examples
```

The script will search for all sniff documentation files and check their code examples. For other ways to run the script check the help page:

```
$ /path/to/script/phpcs-check-sniff-doc-code-examples --help
```
@rodrigoprimo rodrigoprimo force-pushed the check-xml-documentation-code-examples branch from 8298e09 to cae0df1 Compare May 30, 2025 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CLI tool to check the code samples used in XML docs against the sniff
2 participants