Skip to content

java: Implement TestNG XML formatter #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Apr 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/renovate.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"github>cucumber/renovate-config"
]
}
18 changes: 18 additions & 0 deletions .github/workflows/release-github.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
name: Release GitHub

on:
push:
branches: [release/*]

jobs:
create-github-release:
name: Create GitHub Release and Git tag
runs-on: ubuntu-latest
environment: Release
permissions:
contents: write
steps:
- uses: actions/checkout@v4
- uses: cucumber/[email protected]
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
25 changes: 25 additions & 0 deletions .github/workflows/release-mvn.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: Release Maven

on:
push:
branches: [release/*]

jobs:
publish-mvn:
name: Publish Maven Package
runs-on: ubuntu-latest
environment: Release
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
distribution: 'temurin'
java-version: '11'
cache: 'maven'
- uses: cucumber/[email protected]
with:
gpg-private-key: ${{ secrets.GPG_PRIVATE_KEY }}
gpg-passphrase: ${{ secrets.GPG_PASSPHRASE }}
nexus-username: cukebot
nexus-password: ${{ secrets.SONATYPE_PASSWORD }}
working-directory: java
34 changes: 34 additions & 0 deletions .github/workflows/test-java.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: test-java

on:
push:
branches:
- main
- renovate/**
pull_request:
branches:
- main
workflow_call:

jobs:
test-java:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os:
- ubuntu-latest
java: ["11", "17"]

steps:
- uses: actions/checkout@v4

- uses: actions/setup-java@v4
with:
distribution: "zulu"
java-version: ${{ matrix.java }}
cache: "maven"

- run: mvn verify
working-directory: java

32 changes: 32 additions & 0 deletions .github/workflows/test-testdata.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
name: test-testdata

on:
push:
branches:
- main
pull_request:
branches:
- main

jobs:
test-testdata:
runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v4

- uses: actions/setup-node@v4
with:
cache: 'npm'
cache-dependency-path: testdata/package-lock.json

- run: npm ci
working-directory: testdata

- name: check repository is not dirty
run: "[[ -z $(git status --porcelain) ]]"

- name: show diff
if: ${{ failure() }}
run: git status --porcelain
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
.idea/
*.iml
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
### Added
- Java implementation

## 0.0.0

[Unreleased]: https://github.com/cucumber/cucumber-junit-xml-formatter/compare/v0.0.0...HEAD
[0.0.0]: https://github.com/cucumber/cucumber-junit-xml-formatter/compare/438ec1f6218a849eb2a684982e2ff7e304a3155f...v0.0.0
3 changes: 3 additions & 0 deletions java/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
.idea/
*.iml
target/
130 changes: 130 additions & 0 deletions java/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
[![Maven Central](https://img.shields.io/maven-central/v/io.cucumber/testng-xml-formatter.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:%22io.cucumber%22%20AND%20a:%22testng-xml-formatter%22)

⚠️ This is an internal package; you don't need to install it in order to use the TestNG XML Formatter.

TestNG XML Formatter
===================

Writes Cucumber message into a TestNG XML report.

The TestNG XML report does not come with an XSD and the [TestNG documentations](https://testng-docs.readthedocs.io/testresults/#xml-reports)
only provides a minimal example. Nevertheless, we had this formatter since 2013
with no recent issues. So there is a good your tools will understand it.

If not, please let us know in the issues!

## Features and Limitations

### Test outcome mapping

Cucumber and the TestNG XML Report support a different set of test outcomes.
These are mapped according to the table below.

Additionally, it is advisable to run Cucumber in strict mode. When used in
non-strict mode scenarios with a pending or undefined outcome will not fail
the test run ([#714](https://github.com/cucumber/common/issues/714)). This
can lead to a xml report that contains `failure` outcomes while the build
passes.

| Cucumber Outcome | XML Outcome | Passes in strict mode | Passes in non-strict mode |
|------------------|-------------|-----------------------|---------------------------|
| UNKNOWN | n/a | n/a | n/a |
| PASSED | PASS | yes | yes |
| SKIPPED | SKIP | yes | yes |
| PENDING | FAIL | no | yes |
| UNDEFINED | FAIL | no | yes |
| AMBIGUOUS | FAIL | no | no |
| FAILED | FAIL | no | no |


### Step reporting

The TestNG XML report assumes that a test is a method on a class. Yet a scenario
consists of multiple steps. To provide info about the failing step, the `message`
element will contain a rendition of steps and their result.

```xml
<exception class="AssertionError">
<message><![CDATA[
Given there are 12 cucumbers................................................passed
When I eat 5 cucumbers......................................................passed
Then I should have 7 cucumbers..............................................failed
]]></message>
<full-stacktrace>
..the actual stack trace...
</full-stacktrace>
</exception>
```

### Naming Rules and Examples

Cucumber does not require that scenario names are unique. To disambiguate
between similarly named scenarios and examples the report prefixes the rule
to the scenario or example name.

```feature
Feature: Rules

Rule: a sale cannot happen if change cannot be returned
Example: no change
...
Example: exact change
...

Rule: a sale cannot happen if we're out of stock
Example: no chocolates left
...
```

```xml
<class name="Rules">
<test-method name="a sale cannot happen if change cannot be returned - exact change" status="PASS"
duration-ms="7" started-at="1970-01-01T00:00:00.001Z" finished-at="1970-01-01T00:00:00.008Z"/>
<test-method name="a sale cannot happen if we're out of stock - no chocolates left" status="PASS"
duration-ms="7" started-at="1970-01-01T00:00:00.001Z" finished-at="1970-01-01T00:00:00.008Z"/>
</class>
```

Likewise for example tables, the rule (if any), scenario outline name, example
name, and number are included.

```feature
Feature: Examples Tables

Scenario Outline: Eating cucumbers
Given there are <start> cucumbers
When I eat <eat> cucumbers
Then I should have <left> cucumbers

Examples: These are passing
| start | eat | left |
| 12 | 5 | 7 |
| 20 | 5 | 15 |

Examples: These are failing
| start | eat | left |
| 12 | 20 | 0 |
| 0 | 1 | 0 |
```

```xml
<class name="Examples Tables">
<test-method name="Eating cucumbers - These are passing - Example #1.1" status="PASS" duration-ms="7"
started-at="1970-01-01T00:00:00.001Z" finished-at="1970-01-01T00:00:00.008Z"/>
<test-method name="Eating cucumbers - These are passing - Example #1.2" status="PASS" duration-ms="7"
started-at="1970-01-01T00:00:00.009Z" finished-at="1970-01-01T00:00:00.016Z"/>
<test-method name="Eating cucumbers - These are failing - Example #2.1" status="FAIL" duration-ms="7"
started-at="1970-01-01T00:00:00.017Z" finished-at="1970-01-01T00:00:00.024Z">
<exception class="AssertionError">...</exception>
</test-method>
<test-method name="Eating cucumbers - These are failing - Example #2.2" status="FAIL" duration-ms="7"
started-at="1970-01-01T00:00:00.025Z" finished-at="1970-01-01T00:00:00.032Z">
<exception class="AssertionError">...</exception>
</test-method>
</class>
```
## Contributing

Each language implementation validates itself against the examples in the
`testdata` folder. See the [testdata/README.md](testdata/README.md) for more
information.
113 changes: 113 additions & 0 deletions java/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-parent</artifactId>
<version>4.2.0</version>
</parent>

<artifactId>testng-xml-formatter</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>TestNG XML Formatter</name>
<description>Renders Cucumber Messages as TestNG XML</description>
<url>https://github.com/cucumber/testng-xml-formatter</url>

<properties>
<project.Automatic-Module-Name>io.cucumber.testngxmlformatter</project.Automatic-Module-Name>
<project.build.outputTimestamp>1711217894</project.build.outputTimestamp>
</properties>

<scm>
<connection>scm:git:git://github.com/cucumber/testng-xml-formatter.git</connection>
<developerConnection>scm:git:[email protected]:cucumber/testng-xml-formatter.git</developerConnection>
<url>git://github.com/cucumber/testng-xml-formatter.git</url>
<tag>HEAD</tag>
</scm>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.junit</groupId>
<artifactId>junit-bom</artifactId>
<version>5.10.2</version>
<type>pom</type>
<scope>import</scope>
</dependency>

<dependency>
<groupId>com.fasterxml.jackson</groupId>
<artifactId>jackson-bom</artifactId>
<version>2.17.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>

<dependencies>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>messages</artifactId>
<version>[24.0.0,25.0.0)</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>query</artifactId>
<version>[12.1.1,13.0.0)</version>
</dependency>

<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jdk8</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-parameter-names</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest</artifactId>
<version>2.2</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<version>3.25.3</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.xmlunit</groupId>
<artifactId>xmlunit-assertj</artifactId>
<version>2.9.1</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-params</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>
Loading