Managing GitLab Runner at Scale

1. Implementing Runner Autoscaling

Example: Configure autoscaling with Kubernetes

runner:
  name: "autoscaled-runner"
  executor: "kubernetes"
  kubernetes:
    namespace: "ci"
    replicas: 2
    auto_scale:
      enabled: true
      min_replicas: 1
      max_replicas: 10
      target_cpu_utilization_percentage: 70

Property	Description	Default	Notes
auto_scale.enabled	Enables or disables autoscaling	false	Requires Kubernetes executor
auto_scale.min_replicas	Minimum number of runner pods	1
auto_scale.max_replicas	Maximum number of runner pods	10
auto_scale.target_cpu_utilization_percentage	Target CPU usage for scaling	70	Percentage of CPU utilization

Note: Autoscaling is only available for Kubernetes runners.

Warning: Ensure sufficient resources in the Kubernetes cluster to support scaling.

Key Considerations

Autoscaling requires Kubernetes-based runners
Resource limits must be set to prevent overconsumption
Monitor CPU and memory usage for optimal scaling thresholds

2. Monitoring Runner Health

Example: Check runner status via API

curl --header "PRIVATE-TOKEN: glpat-xxxxxxxx" \
  "https://gitlab.example.com/api/v4/runners"

Field	Description	Notes
id	Runner identifier	Unique integer ID
name	Runner name	Configurable display name
status	Runner health status	`online`, `offline`, or `paused`
version	Runner version	Current GitLab Runner version
platform	Runner platform	e.g., `linux`, `darwin`
architecture	Architecture type	e.g., `amd64`, `arm64`

Note: Use status field to detect unhealthy runners.

Example: Monitor runner metrics with Prometheus

scrape_configs:
  - job_name: "gitlab-runner"
    metrics_path: /metrics
    static_configs:
      - targets: ["runner.example.com:9252"]

Key Metrics to Monitor

gitlab_runner_builds_total: Total builds executed
gitlab_runner_builds_running: Currently running builds
gitlab_runner_builds_failed: Failed builds count
gitlab_runner_up: Runner up/down status

3. Using Runner Fleet Management

Example: Configure a GitLab Runner with Fleet Management

gitlab-runner register \
  --url "https://gitlab.example.com/" \
  --registration-token "PROJECT_REGISTRATION_TOKEN" \
  --executor "docker" \
  --docker-image "alpine:latest" \
  --name "fleet-runner-01" \
  --tag-list "docker,linux" \
  --run-untagged="true" \
  --locked="false"

Configuration Option	Description	Default Value	Notes
url	GitLab instance URL	None	Required for registration
registration-token	Project or group registration token	None	Must match the project's token
executor	Runner executor type	shell	Options: shell, docker, kubernetes, etc.
name	Runner name for identification	None	Unique within the GitLab instance
tag-list	Tags to assign to the runner	None	Used for job scheduling
run-untagged	Run jobs without tags	false	Set to true to run untagged jobs
locked	Lock runner to prevent job assignment	true	Set to false for active runners

Note: Runner Fleet Management allows centralized control of multiple GitLab Runners. It enables scaling runners dynamically based on job demand and simplifies configuration management.

Warning: Ensure that the registration token has appropriate permissions to register runners in the desired project or group.

Key Features of GitLab Runner Fleet Management

NEW Centralized runner registration and configuration
NEW Dynamic scaling of runners based on job queue
NEW Tag-based job scheduling for better resource allocation
NEW Support for multiple executor types (Docker, Kubernetes, etc.)

4. Configuring Runner Load Balancing

Configuration Option	Description	Default Value	Notes
concurrent	Maximum number of jobs that can run concurrently on a runner	`1`	Set to higher values for parallel job execution
check_interval	How often to check for new jobs (in seconds)	`3`	Lower values increase responsiveness but may increase load
maximum_timeout	Maximum time a job can run before being killed	`3600`	Set to `0` for no timeout

Example: Configuring concurrent jobs

[runner]
  name = "scale-runner"
  url = "https://gitlab.example.com/"
  token = "runner-token"
  concurrent = 4
  check_interval = 5

Note: Load balancing is handled by GitLab CI/CD coordinator, not the runner itself. Runners must be registered with the same project or group to participate in shared job distribution.

Load Balancing Best Practices

Set concurrent based on available system resources (CPU, memory)
Use check_interval to balance responsiveness and system load
Monitor job execution times to optimize maximum_timeout

5. Implementing Runner High Availability

Example: Configuring HA runners with shared cache

variables:
  GIT_STRATEGY: clone
cache:
  key: "${CI_COMMIT_REF_SLUG}"
  paths:
    - node_modules/
  policy: pull-push

Runner Setting	Description	Value
concurrent	Number of jobs that can run simultaneously	`10`
check_interval	How often to check for new jobs (seconds)	`30`
maximum_timeout	Max time a job can run (seconds)	`3600`

Note: Ensure runners are registered with the same runner.token for HA setup.

Runner HA vs Single Instance

HA Setup: Multiple runners share state via shared cache and coordination
Single Instance: Only one runner handles jobs, potential bottleneck

High Availability Runner Setup Steps

Register multiple runners with same runner.token
Configure shared cache and storage (e.g., NFS or S3)
Set concurrent to allow parallel job execution
Monitor runner status using gitlab-runner status

6. Using Runner Monitoring and Alerting

Example: Configure runner monitoring with Prometheus

runner:
  metrics:
    enabled: true
    listen_address: "0.0.0.0:9252"

Configuration	Description	Default Value	Notes
metrics.enabled	Enables Prometheus metrics collection	`false`	Required for monitoring integration
metrics.listen_address	Address to bind the metrics endpoint	`localhost:9252`	Must be accessible to Prometheus server

Note: Prometheus metrics are exposed at /metrics endpoint on the configured address.

Example: Set up alerting rules in Prometheus

groups:
- name: gitlab-runner-alerts
  rules:
  - alert: RunnerDown
    expr: up{job="gitlab-runner"} == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Runner is down"
      description: "Runner has been unreachable for more than 5 minutes"

Runner Alerting Best Practices

Monitor runner availability and health status
Set up alerts for long-running jobs or job queue backlogs
Use Prometheus and Alertmanager for scalable alerting
Configure alerts based on runner resource usage (CPU, memory)

Runner Monitoring Tools

Prometheus: Collects and stores metrics; requires configuration
GitLab Runner Metrics: Built-in Prometheus exporter
External Monitoring Tools: Grafana, Datadog, New Relic

7. Optimizing Runner Resource Allocation

Resource Type	Configuration	Description	Notes
concurrent_jobs	`1`	Number of jobs that can run concurrently on a runner	Default is 1. Increase for parallel job execution
max_timeout	`3600`	Maximum time (in seconds) a job can run before being killed	Set to prevent runaway jobs from consuming resources
idle_timeout	`60`	Time (in seconds) a runner waits before shutting down when idle	Helps reduce resource usage for temporary runners
maximum_preemption	`false`	Allow preemption of jobs on runners	Use with caution to avoid job disruption

Example: Configuring concurrent_jobs

runner:
  concurrent_jobs: 4
  max_timeout: 7200
  idle_timeout: 120

Note: Increasing concurrent_jobs can improve throughput but may lead to resource contention. Monitor system performance after adjustments.

Warning: Setting max_timeout too high can cause runners to hang indefinitely if jobs do not complete.

8. Implementing Runner Cost Optimization

Example: Configuring runner limits and resource constraints

runner:
  limit: 5
  request_timeout: 30
  maximum_timeout: 3600

Configuration	Description	Default Value	Notes
limit	Maximum number of jobs that can run concurrently	0 (unlimited)	Set to prevent overloading the runner
request_timeout	Timeout for job request in seconds	30	Prevents hanging requests
maximum_timeout	Maximum job execution time in seconds	3600	Prevents long-running jobs from consuming resources

Note: Setting limit to a low value can cause job queuing, while setting it too high may lead to resource exhaustion.

Example: Using tags to manage runner cost

runner:
  tags:
    - production
    - cost-optimized

Runner Cost Optimization Strategies

Use limit to control concurrent job execution
Set request_timeout and maximum_timeout to prevent resource waste
Assign tags to runners for better job routing and cost tracking
Use maximum_timeout to prevent long-running jobs from consuming resources

9. Using Runner Capacity Planning

Example: Configuring runner limits

runner:
  limit: 10
  idle_timeout: 300
  max_timeout: 3600

Property	Description	Default Value	Notes
limit	Maximum number of jobs this runner can execute concurrently	0 (unlimited)	Set to a positive integer to enforce limits
idle_timeout	Time in seconds after which an idle runner shuts down	300	Helps manage resource usage in cloud environments
max_timeout	Maximum time in seconds a job can run before being canceled	3600	Prevents long-running jobs from consuming resources indefinitely

Capacity Planning Best Practices

Monitor job queue length to determine optimal runner count
Use idle_timeout to reduce costs in cloud environments
Set max_timeout to prevent runaway jobs from blocking the queue
Consider using limit to control concurrency per runner instance

Note: Runner capacity planning is crucial for maintaining performance and cost efficiency in large-scale CI/CD pipelines.

10. Upgrading GitLab Runner

Version	Upgrade Method	Notes
17.0+	`gitlab-runner update`	Use this command to upgrade the runner to the latest version. Requires root or admin privileges.
16.0 and below	`gitlab-runner install`	For older versions, reinstallation is required to upgrade. Ensure configuration files are preserved.

Example: Upgrade GitLab Runner via Package Manager

sudo apt update
sudo apt install gitlab-runner

Note: Ensure that the GitLab Runner is stopped before upgrading to prevent any inconsistencies in job execution.

Warning: Upgrading GitLab Runner may introduce breaking changes. Always test in a staging environment before upgrading production runners.