Setting Up High Availability

1. Using Shared Database

Configuration Option	Description	Default Value	Notes
database.type	Database type to use for shared storage	`mysql`	Supported values: `mysql`, `postgres`, `mssql`
database.host	Database server host address	`localhost`	Must be accessible from all Grafana instances in the cluster
database.name	Database name to use for Grafana data	`grafana`	Must be created and accessible by Grafana user
database.user	Database user for Grafana to connect with	`grafana`	Must have appropriate permissions to read/write to database tables
database.password	Password for the database user	`""`	Must be properly escaped in config file or set via environment variable
database.sslmode	SSL mode for database connection	`disable`	Valid options: `disable`, `require`, `verify-full`

Example: Grafana Configuration with Shared Database

[database]
type = mysql
host = db.example.com:3306
name = grafana_db
user = grafana_user
password = "secure_password"
sslmode = require

[server]
domain = grafana.example.com
root_url = %(protocol)s://%(domain)s:%(http_port)s/

Note: Ensure that all Grafana instances in the HA setup point to the same shared database. This includes the same database name, user credentials, and connection parameters.

Warning: Using a shared database requires careful configuration of database permissions and network access to ensure high availability and data consistency across all Grafana instances.

2. Configuring Session Storage

Property	Value	Description
session.storage.type	`redis`	Storage backend for session data. Options: `redis`, `memory`, `database`
session.storage.redis.host	`localhost`	Redis server host address
session.storage.redis.port	`6379`	Redis server port number
session.storage.redis.db	`0`	Redis database index to use for sessions
session.storage.redis.password	`""`	Password for Redis authentication (if required)
session.storage.memory.size	`1000`	Maximum number of sessions to store in memory (when `type` is `memory`)

Example: Redis Session Configuration

[session]
storage.type = redis
storage.redis.host = redis.example.com
storage.redis.port = 6379
storage.redis.db = 1
storage.redis.password = secret123

Note: Using Redis for session storage is recommended in HA setups to ensure session consistency across Grafana instances.

3. Setting Up Load Balancer

Example: Nginx Load Balancer Configuration

# /etc/nginx/conf.d/grafana-lb.conf

upstream grafana_servers {
    server grafana1.example.com:3000;
    server grafana2.example.com:3000;
    server grafana3.example.com:3000;
}

server {
    listen 80;
    server_name grafana.example.com;

    location / {
        proxy_pass http://grafana_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Directive	Description	Default	Notes
upstream	Defines a group of servers to balance requests	None	Must be defined outside any server block
server	Defines a single server in the upstream group	None	Can include weight, max_fails, fail_timeout
proxy_pass	Specifies the address of the upstream server	None	Must match the Grafana instance address

Note: Ensure all Grafana instances are configured with the same root_url and domain settings to avoid redirect loops.

Warning: The load balancer must be configured to handle WebSocket connections for real-time data streaming.

Load Balancer Setup Summary

Use a reverse proxy (e.g., Nginx, HAProxy) to distribute requests
Ensure all Grafana instances are in the same cluster with shared data sources
Configure session affinity if needed for user sessions
Set up health checks to detect failed instances

4. Using Sticky Sessions

Option	Values	Notes
session_affinity	`cookie`	Enables sticky sessions using cookies
session_cookie_name	`grafana_session`	Name of the session cookie used for affinity
session_cookie_path	`/`	Path for the session cookie
session_cookie_domain	`example.com`	Domain for the session cookie (optional)
session_cookie_secure	`true`	Set to `true` if using HTTPS
session_cookie_http_only	`true`	Prevents client-side JavaScript access to the cookie

Example: Configuring Sticky Sessions in Grafana

[server]
domain = example.com
root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana
http_port = 3000

[session]
provider = memory
cookie_name = grafana_session
cookie_secure = true
cookie_http_only = true
session_affinity = cookie
session_cookie_path = /
session_cookie_domain = example.com

Note: Sticky sessions ensure that a user's requests are consistently routed to the same Grafana instance in a high availability setup, which is essential for maintaining session state and user preferences across multiple nodes.

5. Configuring Remote Rendering

Property	Value	Description
rendering.remote.url	`http://localhost:8081`	URL of the remote rendering service
rendering.remote.timeout	`30s`	Timeout for remote rendering requests
rendering.remote.concurrent	`10`	Maximum concurrent remote rendering requests

Example: Remote Rendering Configuration

rendering.remote.url = http://localhost:8081
rendering.remote.timeout = 30s
rendering.remote.concurrent = 10

Note: Ensure the remote rendering service is accessible from the Grafana server.

Warning: Misconfiguration can lead to rendering timeouts or failures.

6. Setting Up Alerting HA

Setting up high availability (HA) for Grafana alerting ensures that alerting rules and notifications remain operational even during system failures. This involves configuring multiple Grafana instances to work in a clustered setup.

Component	Description	Configuration
alerting	Enables or disables alerting functionality.	`true`
cluster.enabled	Enables clustering for HA support.	`true`
cluster.name	Name of the cluster for identification.	`grafana-ha-cluster`
cluster.server.url	URL of the cluster server (e.g., for leader election).	`http://grafana-ha-server:3000`

Example: Alerting HA Configuration

[alerting]
enabled = true
execute_alerts = true

[cluster]
enabled = true
name = grafana-ha-cluster
server.url = http://grafana-ha-server:3000

Note: Alerting HA requires a shared database and consistent configuration across all nodes in the cluster.

Warning: Ensure that all Grafana instances in the HA setup point to the same database and storage backend for alert state consistency.

7. Using External Cache

Option	Values	Notes
cache.type	`redis`	External cache type to use for high availability
cache.host	`redis.example.com`	Hostname or IP of Redis instance
cache.port	`6379`	Port number for Redis connection
cache.db	`0`	Database number to use in Redis instance
cache.password	`secret`	Password for Redis authentication (if required)

Example: Redis Cache Configuration

[cache]
type = redis
host = redis.example.com
port = 6379
db = 0
password = secret

Note: External cache is essential for Grafana HA setups to ensure consistent dashboard rendering and query results across multiple instances.

8. Configuring Shared Plugin Storage

When running Grafana in a high availability (HA) setup with multiple instances, plugin storage must be shared to ensure consistency and prevent data loss.

Configuration Option	Description	Default Value	Notes
plugins	Path to the plugins directory	`/var/lib/grafana/plugins`	Must be a shared filesystem or network storage accessible by all instances
plugin_admin_enabled	Enables plugin administration via the UI	`false`	Recommended to disable in HA setups to prevent inconsistencies

Example: Shared Plugin Storage Configuration

[plugins]
path = /shared/grafana/plugins

[plugin_admin]
enabled = false

Note: Ensure that all Grafana instances in the HA cluster have read/write access to the shared plugin directory.

Warning: Using local plugin directories in HA setups can lead to inconsistent plugin states and data corruption.

Best Practices for Plugin Storage in HA

Use a shared filesystem (e.g., NFS, GlusterFS) for plugin storage
Ensure all Grafana instances mount the same plugin directory at the same path
Disable plugin administration in HA environments to avoid conflicts
Back up plugins regularly to prevent data loss during upgrades or failures

9. Setting Up Database Backup

Example: Configure PostgreSQL Backup

# Create a backup script
#!/bin/bash
BACKUP_DIR="/var/backups/grafana"
DB_NAME="grafana_db"
DB_USER="grafana_user"

pg_dump -U $DB_USER $DB_NAME > $BACKUP_DIR/backup_$(date +%Y%m%d_%H%M%S).sql

Configuration Option	Description	Default Value	Notes
database.backup.enabled	Enables automatic database backup	`false`	Set to `true` to enable
database.backup.path	Directory to store backups	`/var/lib/grafana/backups`	Must be writable by Grafana process
database.backup.retention_days	Number of days to keep backups	`7`	Older backups are deleted automatically

Note: Ensure the backup directory has sufficient disk space and proper permissions for the Grafana service user.

Warning: Backups should be tested regularly to ensure data integrity and restore capability.

Backup Strategy Summary

Full database dumps are created daily
Retention policy keeps backups for 7 days
Backups are stored in a dedicated directory
Automated cleanup removes old backups

10. Using Horizontal Scaling

Example: Grafana HA Configuration with Multiple Instances

# grafana-ha-config.yaml
grafana:
  replicas: 3
  image:
    repository: grafana/grafana
    tag: 11.2.0
  service:
    type: LoadBalancer
    port: 80
  persistence:
    enabled: true
    storageClass: "fast"
    size: 10Gi
  resources:
    limits:
      cpu: "500m"
      memory: "1Gi"
    requests:
      cpu: "250m"
      memory: "512Mi"

Component	Description	Configuration
replicas	Number of Grafana instances to run in HA mode	`3`
image.repository	Container image repository for Grafana	`grafana/grafana`
image.tag	Version of Grafana image to use	`11.2.0`
service.type	Kubernetes service type for load balancing	`LoadBalancer`
persistence.enabled	Enable persistent storage for dashboards and data	`true`
persistence.storageClass	Storage class for persistent volumes	`fast`
persistence.size	Size of persistent volume per instance	`10Gi`

Note: Ensure that all Grafana instances are configured to use the same database and storage backend for consistent state across nodes.

Warning: When using horizontal scaling, make sure that your database and storage systems can handle the increased load from multiple Grafana instances.

Key Considerations for Horizontal Scaling

Use a shared database (e.g., PostgreSQL or MySQL) for all Grafana instances
Enable persistent storage to maintain dashboards and configurations
Configure load balancing to distribute traffic evenly across instances
Ensure consistent configuration across all nodes
Monitor resource usage to prevent overloading individual instances