Working with Test Results

1. Publishing JUnit Test Results

Property	Value	Description
junit	`*/TEST-.xml`	Pattern to match JUnit test result files
keepLongStdio	`false`	Whether to keep long standard output/error logs
allowEmptyResults	`false`	Whether to allow builds with no test results

Example: Publishing JUnit Test Results in Jenkins Pipeline

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                sh 'mvn clean compile'
            }
        }
        stage('Test') {
            steps {
                sh 'mvn test'
            }
            post {
                always {
                    junit 'target/surefire-reports/TEST-*.xml'
                }
            }
        }
    }
}

Note: JUnit test results must be in XML format to be published correctly. The target/surefire-reports directory is the default location for Maven-generated JUnit reports.

Warning: If the test results pattern does not match any files, the build will fail unless allowEmptyResults is set to true.

2. Using TestNG Results

Property	Values	Description
testResults	`*/test-results/.xml`	Path pattern to locate TestNG XML result files
testResultsFormat	`testng`	Specifies the format of test results to parse

Example: Configure TestNG Results in Jenkins Pipeline

pipeline {
    agent any
    stages {
        stage('Run Tests') {
            steps {
                sh 'mvn test'
            }
        }
        stage('Publish Test Results') {
            steps {
                publishTestResults testResults: '**/test-results/*.xml', testResultsFormat: 'testng'
            }
        }
    }
}

Note: Ensure that your TestNG configuration outputs results in the standard XML format for Jenkins to parse correctly.

3. Publishing Code Coverage

Property	Values	Description
codeCoverageTool	`jacoco`, `cobertura`	Tool used to generate code coverage reports
sourceEncoding	`UTF-8`	Encoding used for source files
failBuildOnCoverage	`true`, `false`	Fail build if coverage threshold is not met

Example: Configuring JaCoCo for Code Coverage

publishCodeCoverage(
    codeCoverageTool: 'jacoco',
    sourceEncoding: 'UTF-8',
    failBuildOnCoverage: true,
    threshold: 80
)

Note: JaCoCo is the most commonly used code coverage tool in Java projects.

Warning: Ensure that the code coverage tool is correctly integrated into your build process.

4. Using xUnit Plugin

The xUnit plugin in Jenkins allows you to publish and visualize test results from various testing frameworks. It supports multiple test result formats, including JUnit, NUnit, and XUnit.

Property	Values	Description
testResults	`*/TEST-.xml`	Pattern to match test result files. Default is `*/TEST-.xml`.
testClassname	`org.junit.Test`	Class name to filter tests by. Optional.
failOnTestFailures	`false`	If `true`, the build will fail if tests fail. Default is `false`.

Example: Publishing JUnit Test Results

step([
    $class: 'XUnitPublisher',
    testTimeMargin: '3000',
    tools: [
        [
            $class: 'JUnitTestType',
            pattern: '**/TEST-*.xml',
            skipErrors: false
        ]
    ]
])

Note: Ensure that your test framework generates JUnit-compatible XML files for the xUnit plugin to process them correctly.

5. Displaying Test Trends

Option	Values	Notes
test.trend.chart.type	`line`, `bar`	Chart type for displaying test trends
test.trend.show.passed	`true`, `false`	Whether to show passed tests in trend chart
test.trend.show.failed	`true`, `false`	Whether to show failed tests in trend chart
test.trend.show.skipped	`true`, `false`	Whether to show skipped tests in trend chart

Example: Configure Test Trend Chart

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                sh 'mvn clean compile'
            }
        }
        stage('Test') {
            steps {
                sh 'mvn test'
            }
        }
        stage('Publish Test Results') {
            steps {
                publishTestResults(
                    testResults: 'target/surefire-reports/*.xml',
                    trendChartType: 'line',
                    showPassed: true,
                    showFailed: true,
                    showSkipped: false
                )
            }
        }
    }
}

Note: Test trend charts are generated automatically when test results are published using the publishTestResults step.

Warning: Ensure that test result files are in the correct format (e.g., JUnit XML) to enable proper trend chart generation.

6. Configuring Test Thresholds

Threshold Type	Configuration Property	Description	Default Value
Unstable Threshold	`unstableThreshold`	Percentage of tests that must pass to avoid marking build as unstable	`80`
Fail Threshold	`failThreshold`	Percentage of tests that must pass to avoid marking build as failed	`60`

Example: Setting Test Thresholds in Jenkins Pipeline

pipeline {
    agent any
    stages {
        stage('Test') {
            steps {
                sh 'mvn test'
            }
            post {
                always {
                    junit 'target/surefire-reports/*.xml' 
                }
            }
        }
    }
    post {
        always {
            publishTestResults (
                testResults: 'target/surefire-reports/*.xml',
                unstableThreshold: 90,
                failThreshold: 75
            )
        }
    }
}

Note: These thresholds apply to the overall test suite and can be adjusted based on project requirements.

7. Using Test Result Analyzers

Analyzer	Description	Configuration	Notes
JUnit	Reports test results in JUnit XML format	`test-results.xml`	Standard for Java testing frameworks
Cucumber	Reports Cucumber test results in JSON format	`cucumber.json`	Supports BDD-style tests
XUnit	Reports test results in XUnit XML format	`xunit.xml`	Generic format for various testing tools
Checkmarx KICS	Reports security and compliance issues	`kics-results.json`	Static analysis for infrastructure as code

Example: Configuring JUnit Test Results

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                sh 'mvn clean compile'
            }
        }
        stage('Test') {
            steps {
                sh 'mvn test'
            }
            post {
                always {
                    junit 'target/surefire-reports/*.xml'
                }
            }
        }
    }
}

Note: Ensure that test result files are generated in the correct location and format for the analyzer to process them.

8. Publishing HTML Reports

Property	Values	Description
htmlPublisher	`html.publisher.HtmlPublisher`	Plugin class for publishing HTML reports
allowMissing	`true` \| `false`	If `true`, report is not required to exist
alwaysLinkToLastBuild	`true` \| `false`	If `true`, links to last build's report
keepAll	`true` \| `false`	If `true`, retains all reports from builds
reportDir	`string`	Directory where HTML reports are stored
reportFiles	`string`	Pattern to match report files (e.g., `*/.html`)
excludedFiles	`string`	Pattern to exclude files (e.g., `/test-reports/`)
includes	`string`	Include pattern for report files (e.g., `**/index.html`)
excludes	`string`	Exclude pattern for report files (e.g., `/temp/`)
reportName	`string`	Name displayed in Jenkins UI for the report

Example: Publishing HTML Reports

publishHTML(
    allowMissing: false,
    alwaysLinkToLastBuild: true,
    keepAll: true,
    reportDir: 'build/reports',
    reportFiles: '**/*.html',
    reportName: 'Test Reports'
)

Note: The publishHTML step requires the HTML Publisher Plugin to be installed.

Warning: If reportDir is not set correctly, Jenkins may fail to locate the report files.

Configuration Summary

allowMissing: Set to false to ensure reports are generated.
alwaysLinkToLastBuild: Useful for tracking report changes over time.
keepAll: Retains all builds' reports for historical analysis.
reportDir: Must point to the directory containing report files.
reportFiles: Use glob patterns to match relevant report files.

9. Using Allure Reports

Allure is a flexible, lightweight test reporting tool that provides detailed and interactive reports for test execution. It supports multiple programming languages and frameworks, making it a versatile choice for CI/CD pipelines.

Feature	Description	Support
Allure Jenkins Plugin	Enables integration with Jenkins to generate and display Allure reports directly in the build pipeline.	Good
Report Generation	Generates detailed reports with test results, attachments, and timelines.	Good
Test Categories	Supports categorization of tests (e.g., smoke, regression) for better organization.	Good

Example: Configuring Allure in Jenkins Pipeline

pipeline {
    agent any
    stages {
        stage('Run Tests') {
            steps {
                sh 'mvn test'
            }
        }
        stage('Generate Allure Report') {
            steps {
                allure([
                    includeProperties: false,
                    jdk: '',
                    properties: [],
                    reportDir: 'allure-results',
                    reportVersion: '2.13.8',
                    results: ['allure-results']
                ])
            }
        }
    }
}

Note: Ensure Allure is installed on the Jenkins server or available in the build environment.

Allure Report Features

Interactive test execution timelines
Test case details with attachments and logs
Support for multiple test frameworks (JUnit, TestNG, etc.)
Integration with CI/CD tools like Jenkins

10. Setting Build Status from Tests

Parameter	Description	Default Value	Notes
junit	Path to JUnit XML report files	`*/TEST-.xml`	Supports glob patterns
failBuildOnTestFailures	Whether to fail the build if tests fail	`false`	Set to `true` to mark build as unstable or failed
testResults	Path to test result files (e.g., JUnit, TestNG)	`*/TEST-.xml`	Can be a comma-separated list of paths

Example: Configure JUnit Test Results in Jenkins Pipeline

pipeline {
    agent any
    stages {
        stage('Run Tests') {
            steps {
                sh 'mvn test'
            }
        }
        stage('Publish Test Results') {
            steps {
                junit 'target/surefire-reports/*.xml'
            }
        }
    }
}

Note: Jenkins will automatically mark the build as unstable or failed based on test results if failBuildOnTestFailures is set to true.