Implementing High Availability
1. Understanding Jenkins HA Architecture
| Component | Description | Role in HA |
|---|---|---|
| Master Node | Primary Jenkins instance that manages jobs, configurations, and UI. | Central control plane; handles build scheduling and job execution coordination. |
| Slave/Agent Nodes | Worker nodes that execute builds and tasks assigned by the master. | Distribute workload; provide compute resources for parallel builds. |
| Shared Storage | Centralized storage for Jenkins home directory, logs, and artifacts. | Ensures consistency across nodes; enables failover without data loss. |
| Load Balancer | Routes incoming requests to active Jenkins instances. | Provides redundancy and distributes traffic for high availability. |
| Database | Stores job configurations, build history, and plugin data. | Ensures persistence of configuration and state across HA nodes. |
Example: Basic HA Setup Components
# Example configuration for Jenkins HA setup
# Master Node Configuration
JENKINS_HOME=/shared/jenkins-home
DATABASE_URL=jdbc:h2:file:/shared/db/jenkins
# Load Balancer (e.g., NGINX) configuration
upstream jenkins_ha {
server jenkins-master-1:8080;
server jenkins-master-2:8080;
}
Note: Jenkins HA requires shared storage and a database to maintain state consistency across nodes.
Key Benefits of Jenkins HA
- Redundancy: Failover to backup nodes ensures uptime.
- Scalability: Distribute builds across multiple agent nodes.
- High Availability: Continuous operation even during node failures.
2. Using Jenkins Controllers with Load Balancer
| Component | Description | Configuration |
|---|---|---|
| Jenkins Controller | Primary Jenkins instance managing jobs and configurations | Run with jenkins.war using Java 11+ |
| Load Balancer | Manages traffic between multiple Jenkins controllers | Can be NGINX, HAProxy, or cloud load balancer (e.g., AWS ELB) |
| Session Affinity | Ensures client requests are routed to the same controller | Use sticky sessions or cookie-based routing |
Example: NGINX Load Balancer Configuration
upstream jenkins_controllers {
server jenkins1.example.com:8080;
server jenkins2.example.com:8080;
}
server {
listen 80;
location / {
proxy_pass http://jenkins_controllers;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
Note: Ensure all Jenkins controllers are configured with the same shared configuration and plugins for consistent behavior.
Warning: Session affinity must be enabled to prevent job execution inconsistencies across controllers.
Key Considerations
- All Jenkins controllers must share the same configuration and plugins
- Use a shared data store (e.g., database or file system) for job history and logs
- Implement session affinity to maintain user sessions
- Ensure all controllers are in sync for high availability
3. Configuring Shared File Systems
| Configuration | Description | Notes |
|---|---|---|
| shared.file.system.path | /shared/jenkins |
Path to the shared file system mount |
| backup.location | /backup/jenkins |
Backup directory for critical data |
Example: Mounting Shared File System
mount -t nfs jenkins-nfs-server:/shared/jenkins /shared/jenkins
Note: Ensure all Jenkins nodes have read/write access to the shared file system.
Warning: Inconsistent file system permissions can lead to build failures or data corruption.
| File Type | Location | Access Mode |
|---|---|---|
| Jenkins Home | /shared/jenkins/home |
Read/Write |
| Logs | /shared/jenkins/logs |
Read/Write |
| Workspace | /shared/jenkins/workspace |
Read/Write |
4. Using Database Backend Storage
| Option | Values | Notes |
|---|---|---|
| database.type | mysql, postgresql, mariadb |
Supported database types for Jenkins backend storage |
| database.url | jdbc:mysql://localhost:3306/jenkins |
Connection URL to the database instance |
| database.username | jenkins_user |
Database user with read/write permissions |
| database.password | secret_password |
Password for the database user |
| database.driver | com.mysql.cj.jdbc.Driver |
JDBC driver class for the database type |
Example: Database Configuration in Jenkins
<?xml version='1.0' encoding='UTF-8'?>
<jenkins>
<database>
<type>mysql</type>
<url>jdbc:mysql://localhost:3306/jvmbuild</url>
<username>jenkins_user</username>
<password>secret_password</password>
<driver>com.mysql.cj.jdbc.Driver</driver>
</database>
</jenkins>
Note: Ensure the database user has appropriate permissions to create tables and manage schema changes.
Warning: Using an external database for Jenkins state is critical for HA setups and must be highly available.
5. Implementing Session Persistence
| Configuration Option | Description | Default Value | Notes |
|---|---|---|---|
| jenkins.session.persistence | Enables or disables session persistence | false | Set to true to enable session persistence |
| jenkins.session.store.type | Defines the session store type (e.g., file, database) | file | Supports file-based and database-based stores |
Example: Enabling Session Persistence
export JENKINS_SESSION_PERSISTENCE=true
export JENKINS_SESSION_STORE_TYPE=database
Note: Session persistence requires a shared storage backend like a database or NFS mount to maintain session state across Jenkins instances.
6. Configuring Failover Strategies
| Strategy | Description | Configuration | Notes |
|---|---|---|---|
| Primary-Backup | One master Jenkins instance and one or more backup instances | jenkins.model.JenkinsLocationConfiguration |
Requires manual failover or external tooling |
| Active-Active | Multiple Jenkins masters running simultaneously | jenkins.model.Jenkins with shared storage |
Requires shared filesystem or database |
| Load Balancing | Use a load balancer to distribute jobs across multiple instances | jenkins.model.Jenkins with lb.host |
Supports horizontal scaling but not true HA |
Example: Primary-Backup Configuration
# Set up primary Jenkins instance
export JENKINS_HOME=/var/lib/jenkins-primary
# Configure backup instance to replicate data
export JENKINS_HOME=/var/lib/jenkins-backup
Note: Failover strategies require careful configuration of shared storage and network settings to ensure seamless transitions.
Warning: Active-Active setups can lead to data inconsistencies if not properly synchronized.
7. Using Warm Standby Instances
Warm standby instances in Jenkins provide a high availability solution where a secondary Jenkins instance is kept running and synchronized with the primary instance to ensure minimal downtime during failover.
| Configuration | Description | Notes |
|---|---|---|
| jenkins.url | http://primary-jenkins:8080 |
Primary Jenkins URL for configuration |
| standby.url | http://standby-jenkins:8080 |
Warm standby Jenkins URL |
| sync.plugins | true |
Enable plugin synchronization |
| sync.jobs | true |
Enable job configuration sync |
| sync.credentials | true |
Enable credential sync |
Example: Configuring Warm Standby
<?xml version='1.0' encoding='UTF-8'?>
<jenkins>
<url>http://primary-jenkins:8080</url>
<standby>
<url>http://standby-jenkins:8080</url>
<syncPlugins>true</syncPlugins>
<syncJobs>true</syncJobs>
<syncCredentials>true</syncCredentials>
</standby>
</jenkins>
Note: The warm standby instance must be configured with the same plugins and configurations as the primary to ensure seamless failover.
Warning: Ensure that the standby instance is not used for active builds to prevent job conflicts.
Warm Standby Setup Summary
- Primary Jenkins URL must be defined for configuration sync
- Standby instance must have identical plugin versions and configurations
- Sync settings should be enabled for jobs, plugins, and credentials
- Regular health checks should be implemented to detect failover conditions
8. Implementing Health Checks
Example: Basic Jenkins Health Check Configuration
@Library('jenkins-shared-library') _
import org.jenkinsci.plugins.workflow.job.WorkflowJob
import jenkins.model.Jenkins
def job = Jenkins.instance.getItem('my-pipeline') as WorkflowJob
def healthCheck = job.getBuildHealth()
println "Build Health: ${healthCheck}"
| Check Type | Description | Configuration | Status |
|---|---|---|---|
| Disk Space | Monitors available disk space on Jenkins master | diskSpaceThreshold: 100MB |
NEW |
| Memory Usage | Tracks memory consumption of Jenkins process | memoryThreshold: 80% |
DEPRECATED |
| Build Queue | Monitors number of queued builds | queueLimit: 50 |
BETA |
Note: Health checks can be configured via the Jenkins UI under
Manage Jenkins > Configure System.
Warning: Custom health checks may impact performance if not implemented efficiently.
Health Check Summary
- Disk space monitoring is enabled by default
- Memory usage check was deprecated in v2.400
- Queue size limit can be adjusted for high-load environments
9. Using Jenkins Operator for Kubernetes
The Jenkins Operator for Kubernetes is a powerful tool that automates the deployment, configuration, and management of Jenkins instances within a Kubernetes cluster. It leverages the jenkins-operator to provide a declarative approach to Jenkins infrastructure.
| Feature | Description | Notes |
|---|---|---|
| Jenkins CRD | Custom Resource Definition (CRD) for Jenkins instances | Defines the structure and configuration of Jenkins deployments |
| Operator Deployment | Deploys Jenkins as a Kubernetes StatefulSet | Ensures high availability and persistent storage |
| Configuration Management | Manages Jenkins configuration via ConfigMaps and Secrets | Supports external configuration management for security |
| Plugin Management | Automates plugin installation and updates | Supports custom plugin repositories and versioning |
Example: Jenkins CRD Definition
apiVersion: jenkins.io/v1
kind: Jenkins
metadata:
name: jenkins
spec:
version: "2.401.3-latest"
master:
resources:
limits:
cpu: "1"
memory: "2Gi"
requests:
cpu: "500m"
memory: "1Gi"
serviceType: "LoadBalancer"
persistence:
storageClass: "standard"
size: "20Gi"Example: Operator Installation
kubectl apply -f https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/deploy/crds/jenkins.io_jenkins_crd.yaml
kubectl apply -f https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/deploy/operator.yaml
Note: The Jenkins Operator requires Kubernetes 1.19 or higher and is compatible with Jenkins versions 2.361.1 and later.
Key Benefits of Jenkins Operator
- Automated deployment and scaling of Jenkins masters
- Support for high availability through StatefulSets
- Centralized configuration management via CRDs
- Integration with Kubernetes-native tools and workflows
10. Monitoring HA Cluster Status
| Property | Value | Description |
|---|---|---|
| jenkins.ha.enabled | true |
Enables high availability mode |
| jenkins.ha.node.id | node-1 |
Unique identifier for each node in the cluster |
| jenkins.ha.cluster.name | my-ha-cluster |
Name of the HA cluster for identification |
Note: The HA cluster status can be monitored via the Jenkins UI under
Manage Jenkins > Cluster Status.
Warning: Ensure all nodes in the HA cluster are running and synchronized to avoid job failures.