Implementing Monitoring and Logging

1. Accessing Build Console Logs

Method	Description	Notes
Build Console Log	Access logs directly from the build page	Available via the "Console Output" link in the build details
API Endpoint	`/job/<job_name>/<build_number>/consoleText`	Provides raw console output via HTTP GET
Log Rotation	Configurable in Jenkins global settings	Defaults to retaining last 5 builds or 100MB of logs

Example: Fetching Console Output via API

curl -X GET "http://jenkins.example.com/job/my-job/1/consoleText"

Note: Console logs are useful for debugging build failures and tracking real-time execution.

2. Using Log Parsers

Log Parser	Description	Usage	Notes
Log Parser Plugin	Enables parsing of log files to extract build information and errors.	Install via Jenkins Plugin Manager	Supports various log formats including Maven, Gradle, and custom formats.
Log Parser Configuration	Allows defining custom patterns to parse logs and extract specific messages.	Defined in Jenkins job configuration or global settings	Supports regex patterns for flexible parsing.
Build Log Analysis	Automatically analyzes build logs for warnings and errors.	Enabled in job configuration under "Post-build Actions"	Can be configured to fail builds based on log content.

Example: Configuring Log Parser Plugin

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                sh 'mvn clean install'
            }
        }
    }
    post {
        always {
            logParser pattern: 'maven'
        }
    }
}

Note: The Log Parser plugin requires the maven pattern to be defined in Jenkins configuration or job settings.

Warning: Ensure that log patterns are correctly defined to avoid false positives in build analysis.

Summary of Log Parser Features

Supports predefined and custom log patterns.
Integrates with Jenkins Pipeline for automated build analysis.
Provides detailed error and warning reporting in build logs.

3. Configuring Log Rotation

Property	Values	Description
log.rotation.maxSize	`100MB`	Maximum size of a log file before rotation occurs
log.rotation.maxHistory	`10`	Number of rotated log files to keep
log.rotation.cleanHistoryOnStart	`false`	Whether to clean up old log files on Jenkins startup

Example: Jenkins Log Rotation Configuration

import jenkins.model.Jenkins
import hudson.util.RingBufferLogHandler

def jenkins = Jenkins.instance
def logHandler = jenkins.getHandler("jenkins")
logHandler.setRotatingFileHandler(
    maxSize: "100MB",
    maxHistory: 10,
    cleanHistoryOnStart: false
)

Note: Log rotation is handled by the RotatingFileHandler in Jenkins, which can be configured via the UI or programmatically using Groovy scripts.

4. Using Build Timestamps

Property	Value	Description
build.timestamp	`true`	Enables timestamp logging for each build step
build.timestamp.format	`yyyy-MM-dd HH:mm:ss`	Format string for timestamp display

Example: Configuring Build Timestamps

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                script {
                    // Enable timestamp logging
                    timestamps {
                        sh 'echo "Build started at ${BUILD_TIMESTAMP}"'
                    }
                }
            }
        }
    }
}

Note: The BUILD_TIMESTAMP environment variable is automatically set by Jenkins when using the timestamps directive.

5. Integrating with ELK Stack

Jenkins can be integrated with the ELK (Elasticsearch, Logstash, Kibana) stack to centralize and visualize logs from builds and pipelines.

Component	Description	Integration Method
Elasticsearch	Stores and indexes logs and metrics from Jenkins.	Use the `elasticsearch` plugin or direct HTTP logging to ES.
Logstash	Processes and enriches log data before sending to Elasticsearch.	Configure Logstash to receive logs via TCP/UDP or file input.
Kibana	Provides visualization and dashboard capabilities for logs.	Connect to Elasticsearch instance to create dashboards and visualizations.

Example: Configuring Jenkins to send logs to Logstash

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                echo 'Building...'
                // Send logs to Logstash via TCP
                sh "echo 'Build started at $(date)' | nc logstash.host 5000"
            }
        }
    }
}

Note: Ensure that the Logstash server is configured to accept logs from Jenkins and that network connectivity is established.

Key Steps for ELK Integration

Install and configure the elasticsearch plugin in Jenkins.
Set up Logstash to receive and process logs from Jenkins.
Configure Kibana to connect to Elasticsearch for dashboard creation.

6. Using Monitoring Plugins

Plugin	Description	Metrics Collected	Installation
Jenkins Monitoring Plugin	Provides real-time monitoring of Jenkins jobs and system performance.	`build duration`, `job status`, `system load`	`Manage Jenkins > Manage Plugins`
Prometheus Plugin	Exposes Jenkins metrics in Prometheus format for integration with monitoring systems.	`job metrics`, `build counts`, `queue length`	`Manage Jenkins > Manage Plugins`
Grafana Plugin	Enables visualization of Jenkins metrics in Grafana dashboards.	`build success rate`, `job execution time`	`Manage Jenkins > Manage Plugins`

Example: Configuring Prometheus Plugin

curl -X GET "http://jenkins-server:8080/prometheus"

Note: Ensure that the Prometheus endpoint is accessible from your monitoring system.

Warning: The Prometheus plugin requires the prometheus port to be exposed.

7. Tracking Build Metrics

Metric	Description	Collection Method	Notes
Build Duration	Time taken to complete a build	Jenkins API / UI	Useful for identifying slow builds
Build Success Rate	Percentage of successful builds	Jenkins API / UI	Indicates system stability
Build Frequency	How often builds are triggered	Jenkins API / UI	Helps in understanding CI/CD workflow
Test Coverage	Percentage of code covered by tests	Integration with testing tools (e.g., JaCoCo)	Measures code quality
Failed Build Count	Number of builds that failed	Jenkins API / UI	Used for tracking build reliability

Example: Fetching Build Metrics via Jenkins API

curl -u "user:token" "http://jenkins.example.com/job/MyJob/api/json?tree=builds[number,duration,timestamp,result]"

Note: Jenkins provides a rich REST API to fetch build metrics. You can use tools like curl or Jenkins plugins to integrate with monitoring systems.

Warning: Ensure that sensitive data (like tokens) are not exposed in logs or API responses.

8. Using Build Health Indicators

Indicator	Description	Configuration	Notes
Build Health Score	Represents the overall health of a build based on test results and build duration.	`healthScore`	Calculated from pass/fail counts and build time thresholds.
Test Results	Tracks pass/fail rates of tests executed in the build.	`testResults`	Can be configured to fail builds based on thresholds.
Build Duration	Measures how long a build takes to complete.	`buildDuration`	Used in health score calculation and can trigger alerts.
Build Stability	Indicates the consistency of build success over time.	`stability`	Calculated as a percentage of successful builds in a window.

Example: Configuring Build Health Indicators

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                sh 'mvn clean compile'
            }
        }
    }
    post {
        always {
            publishTestResults(
                testResults: 'target/surefire-reports/*.xml',
                healthScaleFactor: 1.0
            )
        }
    }
}

Note: Build health indicators can be customized via Jenkins plugins like Build Time Trend or Test Results.

Warning: Ensure that health thresholds are set appropriately to avoid false positives or negatives in build status reporting.

9. Configuring System Logs

Property	Value	Description
jenkins.log.level	`INFO`	Log level for Jenkins system logs. Options: `SEVERE`, `WARNING`, `INFO`, `FINE`, `FINER`, `FINEST`
hudson.log	`logs/hudson.log`	Path to the main Jenkins log file
jenkins.log.rotation.maxFileSize	`100MB`	Maximum size of log file before rotation
jenkins.log.rotation.maxHistory	`10`	Number of rotated log files to keep

Example: Setting Log Level via Jenkins CLI

java -jar jenkins-cli.jar -s http://localhost:8080/ -auth <user>:<token> set-log-level INFO

Example: Rotating Logs with Logback Configuration

<configuration>
  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>logs/jenkins.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
      <fileNamePattern>logs/jenkins.log.%i</fileNamePattern>
      <minIndex>1</minIndex>
      <maxIndex>10</maxIndex>
    </rollingPolicy>
    <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
      <maxFileSize>100MB</maxFileSize>
    </triggeringPolicy>
  </appender>
</configuration>

Note: Log rotation is handled by Logback, which is the logging framework used by Jenkins. Ensure that the log directory has appropriate permissions for file creation and rotation.

10. Using External Monitoring Tools

Tool	Integration Method	Features	Notes
Prometheus	`jenkins-prometheus-plugin`	Metrics collection, Grafana integration	Exports Jenkins metrics in Prometheus format
ELK Stack (Elasticsearch, Logstash, Kibana)	Log aggregation via Logstash	Centralized logging, real-time search, visualization	Requires configuration of Logstash inputs and Kibana dashboards
Datadog	Agent-based or API integration	Infrastructure monitoring, logs, APM	Supports Jenkins plugin for metrics and trace collection
New Relic	Browser and agent-based monitoring	Application performance monitoring, logs	Provides Jenkins-specific dashboards and alerts

Example: Prometheus Integration

curl -X GET http://jenkins.example.com/prometheus

Note: Prometheus requires the jenkins-prometheus-plugin to be installed and configured.

Warning: Ensure proper firewall rules and authentication when exposing metrics endpoints externally.