Skip to content

Add script to validate code examples in sniff documentation #185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from
Open

Add script to validate code examples in sniff documentation #185

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b59c267
Refactor FeatureComplete code for color support and formatting help text
rodrigoprimo May 21, 2025
cae0df1
Add script to validate code examples in sniff documentation
rodrigoprimo Apr 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 89 additions & 2 deletions .github/workflows/quicktest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,10 +68,14 @@ jobs:
if: ${{ matrix.php >= '7.2' }}
run: composer lint

- name: Lint against parse errors (PHP < 7.2)
if: ${{ matrix.php < '7.2' }}
- name: Lint against parse errors (PHP < 7.2 && PHP >= 7.0)
if: ${{ matrix.php < '7.2' && matrix.php >= '7.0' }}
run: composer lintlt72

- name: Lint against parse errors (PHP < 7.0)
if: ${{ matrix.php < '7.0' }}
run: composer lintlt70

# Check that any sniffs available are feature complete.
# This acts as an integration test for the feature completeness script,
# which is why it is run against various PHP versions.
Expand Down Expand Up @@ -173,3 +177,86 @@ jobs:

- name: Run the unit tests for the PHPCSDebug sniff
run: composer test-sniff${{ steps.phpunit_script.outputs.SUFFIX }}

# Run CI checks/tests for the DocCodeExamples script as it requires PHP >= 7.0.
quicktest-doc-code-examples:
runs-on: ${{ matrix.os }}

strategy:
matrix:
os: ['ubuntu-latest', 'windows-latest']
php: ['7.0', 'latest']
phpcs_version: ['dev-master']

include:
- os: 'ubuntu-latest'
php: '7.0'
phpcs_version: '3.1.0'
- os: 'windows-latest'
php: '7.0'
phpcs_version: '3.1.0'

name: "QTest: PHP ${{ matrix.php }} - PHPCS ${{ matrix.phpcs_version }} (${{ matrix.os == 'windows-latest' && 'Win' || 'Linux' }})"

steps:
- name: Prepare git to leave line endings alone
run: git config --global core.autocrlf input

- name: Checkout code
uses: actions/checkout@v4

# On stable PHPCS versions, allow for PHP deprecation notices.
# Unit tests don't need to fail on those for stable releases where those issues won't get fixed anymore.
- name: Setup ini config
id: set_ini
shell: bash
run: |
if [ "${{ matrix.phpcs_version }}" != "dev-master" ]; then
echo 'PHP_INI=error_reporting=E_ALL & ~E_DEPRECATED, display_errors=On' >> "$GITHUB_OUTPUT"
else
echo 'PHP_INI=error_reporting=-1, display_errors=On' >> "$GITHUB_OUTPUT"
fi

- name: Install PHP
uses: shivammathur/setup-php@v2
with:
php-version: ${{ matrix.php }}
ini-values: ${{ steps.set_ini.outputs.PHP_INI }}
coverage: none

- name: 'Composer: adjust dependencies'
run: |
# Set the PHPCS version to be used in the tests.
composer require --no-update squizlabs/php_codesniffer:"${{ matrix.phpcs_version }}" --no-scripts --no-interaction
# Remove the PHPCSDevCS dependency as it has different PHPCS requirements and would block installs.
composer remove --no-update --dev phpcsstandards/phpcsdevcs --no-scripts --no-interaction

# Install dependencies and handle caching in one go.
# @link https://github.com/marketplace/actions/install-php-dependencies-with-composer
- name: Install Composer dependencies
uses: "ramsey/composer-install@v3"
with:
# Bust the cache at least once a month - output format: YYYY-MM.
custom-cache-suffix: $(date -u "+%Y-%m")

- name: Grab PHPUnit version
id: phpunit_version
shell: bash
# yamllint disable rule:line-length
run: echo "VERSION=$(vendor/bin/phpunit --version | grep --only-matching --max-count=1 --extended-regexp '\b[0-9]+\.[0-9]+')" >> "$GITHUB_OUTPUT"
# yamllint enable rule:line-length

- name: Determine PHPUnit composer script to use
id: phpunit_script
shell: bash
run: |
if [ "${{ startsWith( steps.phpunit_version.outputs.VERSION, '11.' ) }}" == "true" ]; then
echo 'SUFFIX=' >> "$GITHUB_OUTPUT"
elif [ "${{ startsWith( steps.phpunit_version.outputs.VERSION, '10.' ) }}" == "true" ]; then
echo 'SUFFIX=' >> "$GITHUB_OUTPUT"
else
echo 'SUFFIX=-lte9' >> "$GITHUB_OUTPUT"
fi

- name: Run the unit tests for the DocCodeExamples script
run: composer test-doc-code-examples${{ steps.phpunit_script.outputs.SUFFIX }}
199 changes: 196 additions & 3 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,9 +73,13 @@ jobs:
if: ${{ matrix.php >= '7.2' }}
run: composer lint -- --checkstyle | cs2pr

- name: Lint against parse errors (PHP < 7.2)
if: ${{ matrix.php < '7.2' }}
run: composer lintlt72 -- --checkstyle | cs2pr
- name: Lint against parse errors (PHP < 7.2 && PHP >= 7.0)
if: ${{ matrix.php < '7.2' && matrix.php >= '7.0' }}
run: composer lintlt72

- name: Lint against parse errors (PHP < 7.0)
if: ${{ matrix.php < '7.0' }}
run: composer lintlt70

# Check that any sniffs available are feature complete.
# This also acts as an integration test for the feature completeness script,
Expand Down Expand Up @@ -230,3 +234,192 @@ jobs:

- name: Run the unit tests for the PHPCSDebug sniff
run: composer test-sniff${{ steps.phpunit_script.outputs.SUFFIX }}

# Run CI checks/tests for the DocCodeExamples script as it requires PHP >= 7.0.
test-doc-code-examples:
runs-on: ${{ matrix.os }}

strategy:
matrix:
os: ['ubuntu-latest', 'windows-latest']
# IMPORTANT: test runs shouldn't fail because of PHPCS being incompatible with a PHP version.
# - PHPCS will run without errors on PHP 7.0 - 7.2 on any version.
# - PHP 7.3 needs PHPCS 3.3.1+ to run without errors.
# - PHP 7.4 needs PHPCS 3.5.0+ to run without errors.
# - PHP 8.0 needs PHPCS 3.5.7+ to run without errors.
# - PHP 8.1 needs PHPCS 3.6.1+ to run without errors.
# - PHP 8.2 needs PHPCS 3.6.1+ to run without errors.
# - PHP 8.3 needs PHPCS 3.8.0+ to run without errors (though the errors don't affect this package).
# - PHP 8.4 needs PHPCS 3.8.0+ to run without errors (officially 3.11.0, but 3.8.0 will work fine).
php: ['7.0', '7.1', '7.2']
phpcs_version: ['3.1.0', 'dev-master']

include:
# Complete the matrix, while preventing issues with PHPCS versions incompatible with certain PHP versions.
- os: 'ubuntu-latest'
php: '7.3'
phpcs_version: 'dev-master'
- os: 'windows-latest'
php: '7.3'
phpcs_version: 'dev-master'
- os: 'ubuntu-latest'
php: '7.3'
phpcs_version: '3.3.1'
- os: 'windows-latest'
php: '7.3'
phpcs_version: '3.3.1'

- os: 'ubuntu-latest'
php: '7.4'
phpcs_version: 'dev-master'
- os: 'windows-latest'
php: '7.4'
phpcs_version: 'dev-master'
- os: 'ubuntu-latest'
php: '7.4'
phpcs_version: '3.5.0'
- os: 'windows-latest'
php: '7.4'
phpcs_version: '3.5.0'

- os: 'ubuntu-latest'
php: '8.0'
phpcs_version: 'dev-master'
- os: 'windows-latest'
php: '8.0'
phpcs_version: 'dev-master'
- os: 'ubuntu-latest'
php: '8.0'
phpcs_version: '3.5.7'
- os: 'windows-latest'
php: '8.0'
phpcs_version: '3.5.7'

- os: 'ubuntu-latest'
php: '8.1'
phpcs_version: 'dev-master'
- os: 'windows-latest'
php: '8.1'
phpcs_version: 'dev-master'
- os: 'ubuntu-latest'
php: '8.1'
phpcs_version: '3.6.1'
- os: 'windows-latest'
php: '8.1'
phpcs_version: '3.6.1'

- os: 'ubuntu-latest'
php: '8.2'
phpcs_version: 'dev-master'
- os: 'windows-latest'
php: '8.2'
phpcs_version: 'dev-master'
- os: 'ubuntu-latest'
php: '8.2'
phpcs_version: '3.6.1'
- os: 'windows-latest'
php: '8.2'
phpcs_version: '3.6.1'

- os: 'ubuntu-latest'
php: '8.3'
phpcs_version: 'dev-master'
- os: 'windows-latest'
php: '8.3'
phpcs_version: 'dev-master'
- os: 'ubuntu-latest'
php: '8.3'
phpcs_version: '3.8.0'
- os: 'windows-latest'
php: '8.3'
phpcs_version: '3.8.0'

- os: 'ubuntu-latest'
php: '8.4'
phpcs_version: 'dev-master'
- os: 'windows-latest'
php: '8.4'
phpcs_version: 'dev-master'
- os: 'ubuntu-latest'
php: '8.4'
phpcs_version: '3.8.0'
- os: 'windows-latest'
php: '8.4'
phpcs_version: '3.8.0'

# Experimental builds. These are allowed to fail.
- os: 'ubuntu-latest'
php: '7.4'
phpcs_version: '4.0.x-dev'

- os: 'ubuntu-latest'
php: '8.5' # Nightly.
phpcs_version: 'dev-master'

name: "Test: PHP ${{ matrix.php }} - PHPCS ${{ matrix.phpcs_version }} (${{ matrix.os == 'windows-latest' && 'Win' || 'Linux' }})"

continue-on-error: ${{ matrix.php == '8.5' || matrix.phpcs_version == '4.0.x-dev' }}

steps:
- name: Prepare git to leave line endings alone
run: git config --global core.autocrlf input

- name: Checkout code
uses: actions/checkout@v4

- name: Setup ini config
id: set_ini
shell: bash
run: |
# On stable PHPCS versions, allow for PHP deprecation notices.
# Unit tests don't need to fail on those for stable releases where those issues won't get fixed anymore.
if [[ "${{ matrix.phpcs_version }}" != "dev-master" && "${{ matrix.phpcs_version }}" != "4.0.x-dev" ]]; then
echo 'PHP_INI=error_reporting=E_ALL & ~E_DEPRECATED, display_errors=On' >> "$GITHUB_OUTPUT"
else
echo 'PHP_INI=error_reporting=-1, display_errors=On' >> "$GITHUB_OUTPUT"
fi

- name: Install PHP
uses: shivammathur/setup-php@v2
with:
php-version: ${{ matrix.php }}
ini-values: ${{ steps.set_ini.outputs.PHP_INI }}
coverage: none

- name: 'Composer: adjust dependencies'
run: |
# Set the PHPCS version to be used in the tests.
composer require --no-update squizlabs/php_codesniffer:"${{ matrix.phpcs_version }}" --no-scripts --no-interaction
# Remove the PHPCSDevCS dependency as it has different PHPCS requirements and would block installs.
composer remove --no-update --dev phpcsstandards/phpcsdevcs --no-scripts --no-interaction

# Install dependencies and handle caching in one go.
# @link https://github.com/marketplace/actions/install-php-dependencies-with-composer
- name: Install Composer dependencies
uses: "ramsey/composer-install@v3"
with:
composer-options: ${{ matrix.php == '8.5' && '--ignore-platform-req=php+' || '' }}
# Bust the cache at least once a month - output format: YYYY-MM.
custom-cache-suffix: $(date -u "+%Y-%m")

- name: Grab PHPUnit version
id: phpunit_version
shell: bash
# yamllint disable rule:line-length
run: echo "VERSION=$(vendor/bin/phpunit --version | grep --only-matching --max-count=1 --extended-regexp '\b[0-9]+\.[0-9]+')" >> "$GITHUB_OUTPUT"
# yamllint enable rule:line-length

- name: Determine PHPUnit composer script to use
id: phpunit_script
shell: bash
run: |
if [ "${{ startsWith( steps.phpunit_version.outputs.VERSION, '11.' ) }}" == "true" ]; then
echo 'SUFFIX=' >> "$GITHUB_OUTPUT"
elif [ "${{ startsWith( steps.phpunit_version.outputs.VERSION, '10.' ) }}" == "true" ]; then
echo 'SUFFIX=' >> "$GITHUB_OUTPUT"
else
echo 'SUFFIX=-lte9' >> "$GITHUB_OUTPUT"
fi

- name: Run the unit tests for the DocCodeExamples script
run: composer test-doc-code-examples${{ steps.phpunit_script.outputs.SUFFIX }}
39 changes: 39 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ This is a set of tools to assist developers of sniffs for [PHP CodeSniffer][phpc
+ [Stand-alone Installation](#stand-alone-installation)
* [Features](#features)
+ [Checking whether all sniffs in a PHPCS standard are feature complete](#checking-whether-all-sniffs-in-a-phpcs-standard-are-feature-complete)
+ [Checking code examples in sniff documentation are correct](#checking-code-examples-in-sniff-documentation-are-correct)
+ [Sniff Debugging](#sniff-debugging)
+ [Documentation XSD Validation](#documentation-xsd-validation)
* [Contributing](#contributing)
Expand Down Expand Up @@ -133,6 +134,44 @@ directories <dir> One or more specific directories to examine.
-V, --version Display the current version of this script.
```

### Checking code examples in sniff documentation are correct

You can verify that the code examples in your sniff documentation XML files correctly demonstrate the behavior of the sniffs. In other words, check that "Valid" examples don't trigger the sniff and "Invalid" examples do trigger the sniff.

Note that while this repository requires PHP >= 5.4, this script requires PHP >= 7.0.

Be aware that this script is unable to differentiate between multiple code examples in the same `<code>` block. This means that for invalid examples, the script might miss ones that don't trigger the sniff if, in the same `<code>` block, there is at least one example that does trigger the sniff.

To use the tool, run it from the root of your standards repo like so:
```bash
# When installed as a project dependency:
vendor/bin/phpcs-check-sniff-doc-code-examples

# When installed globally with Composer:
phpcs-check-sniff-doc-code-examples

# When installed as a git clone or otherwise:
php path/to/PHPCSDevTools/bin/phpcs-check-sniff-doc-code-examples
```

If all is good, the script will exit with code 0 and no output.

If there are issues with the code examples, you will see error messages for each problematic example, like so:
```text
Errors found while processing path/to/project/StandardName/Docs/Category/SniffNameStandard.xml:
- Invalid example #1 does not trigger any errors, but it should.
- Valid example #2 triggers errors, but it shouldn't.
```

#### Options
```text
directories|files One or more specific directories or files to examine.
Defaults to the directory from which the script is run.
--exclude=<dir1,dir2> Comma-delimited list of relative paths of directories to
exclude from the scan.
--ignore-sniffs=<sniff1,sniff2> Comma-delimited list of sniffs to ignore.
--help Print the script help page.
```

### Sniff Debugging

Expand Down
Loading
Loading