Skip to main content
Mapagam

Hardening Prometheus Security

1. Enabling TLS for Web Interface

Configuration Option Description Default Value Notes
web.tls.config Path to the TLS configuration file empty Required when enabling TLS
web.listen-address Address to listen on for web interface :9090 Must include https:// when TLS is enabled

Example: TLS Configuration File

tls_config:
  cert_file: /path/to/cert.pem
  key_file: /path/to/key.pem
  client_ca_file: /path/to/client-ca.pem
Note: Ensure that the certificate and key files are accessible to the Prometheus process and have appropriate permissions.

2. Restricting Network Access

Configuration Option Description Default Value Notes
web.listen-address Address to listen on for web interface and telemetry :9090 Bind to localhost for local access or specific IP for remote access
web.read-timeout Maximum duration before timing out read of the request 30s Prevents long-running requests from consuming resources
web.write-timeout Maximum duration before timing out write of the response 30s Prevents long-running writes from consuming resources

Example: Secure Prometheus Configuration

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
- job_name: 'prometheus'
  static_configs:
  - targets: ['localhost:9090']

web:
  listen-address: '127.0.0.1:9090'
  read-timeout: 30s
  write-timeout: 30s
Note: Binding to 127.0.0.1 restricts access to the local machine only, which is recommended for security.
Warning: Exposing Prometheus on a public IP without authentication can lead to unauthorized access and data leakage.

Network Access Best Practices

  • Bind Prometheus to 127.0.0.1 for local access or specific IP addresses
  • Use a reverse proxy like NGINX or Apache for external access with authentication
  • Set appropriate read and write timeouts to prevent resource exhaustion
  • Implement firewall rules to restrict access to Prometheus ports (default 9090)

3. Disabling Admin API Endpoints

Endpoint Description Status Configuration
/admin/config Expose Prometheus configuration DEPRECATED Use --web.enable-admin-api to control access
/admin/status Expose Prometheus status information DEPRECATED Use --web.enable-admin-api to control access
/admin/tsdb Expose TSDB (Time Series Database) operations DEPRECATED Use --web.enable-admin-api to control access
Note: Admin API endpoints are disabled by default in Prometheus v2.35+.

Example: Disabling Admin API Endpoints

# Start Prometheus with admin API disabled
prometheus --web.enable-admin-api=false

# Or in configuration file (prometheus.yml)
web:
  enable-admin-api: false

4. Securing TSDB Admin Endpoints

Endpoint Description Security Considerations Recommended Protection
/api/v1/admin/tsdb/snapshot Creates a snapshot of the TSDB Allows creation of backups without authentication Require API key or token for access
/api/v1/admin/tsdb/delete_series Deletes series from the TSDB Highly destructive operation; can delete all data Implement strict RBAC and audit logging
/api/v1/admin/tsdb/clean_tombstones Cleans up tombstones from the TSDB Can affect data consistency if misused Restrict to admin users only

Example: Protecting TSDB Admin Endpoints with Basic Auth

prometheus:
  web:
    basic_auth:
      username: "admin"
      password: "secure_password"
Note: Ensure that the basic_auth credentials are stored securely and rotated regularly.
Warning: These endpoints should never be exposed to public networks without proper authentication and authorization.

5. Using Reverse Proxy Authentication

Configuration Description Example
reverse_proxy Reverse proxy server (e.g., Nginx, Apache) handling authentication nginx
auth_header Header name used to pass authenticated user info X-Forwarded-User
auth_url URL to the reverse proxy's authentication endpoint https://proxy.example.com/auth

Example: Nginx Configuration with Basic Auth

location /prometheus {
    auth_basic "Prometheus";
    auth_basic_user_file /etc/nginx/.htpasswd;

    proxy_pass http://localhost:9090;
    proxy_set_header X-Forwarded-User $remote_user;
}
Note: Ensure that the reverse proxy is configured to forward the authenticated user in a header like X-Forwarded-User so Prometheus can use it for authorization.
Warning: Never expose Prometheus directly to the internet without authentication or reverse proxy setup. Misconfigured proxies can lead to unauthorized access.

6. Configuring HTTPS Scraping

Option Values Notes
scrape_config.http_config.tls_config {} Configuration for TLS settings when scraping targets over HTTPS
scrape_config.http_config.tls_config.ca_file /path/to/ca.crt Path to the CA certificate file used to verify the target's certificate
scrape_config.http_config.tls_config.cert_file /path/to/client.crt Path to the client certificate file for mTLS
scrape_config.http_config.tls_config.key_file /path/to/client.key Path to the client private key file for mTLS
scrape_config.http_config.tls_config.insecure_skip_verify false Set to true to skip TLS certificate verification (not recommended in production)

Example: HTTPS scraping with TLS config

scrape_configs:
  - job_name: "https-target"
    metrics_path: /metrics
    scheme: https
    static_configs:
      - targets: ["secure.target.com:9090"]
    http_config:
      tls_config:
        ca_file: "/etc/prometheus/ssl/ca.crt"
        cert_file: "/etc/prometheus/ssl/client.crt"
        key_file: "/etc/prometheus/ssl/client.key"
        insecure_skip_verify: false
Note: Ensure that the CA certificate, client certificate, and key files are properly secured and accessible to the Prometheus process.

7. Implementing mTLS

Transport Layer Security (mTLS) is a security protocol that ensures secure communication between services by authenticating both the client and server using certificates. In Prometheus, mTLS can be used to secure communication between Prometheus servers, remote storage systems, and other components.

Configuration Option Description Example Value
tls_config Specifies TLS configuration for secure connections { "cert_file": "/path/to/cert.pem", "key_file": "/path/to/key.pem", "ca_file": "/path/to/ca.pem" }

Example: mTLS Configuration in Prometheus

remote_write:
  - url: https://prometheus-remote-write.example.com/api/v1/write
    tls_config:
      ca_file: /etc/prometheus/certs/ca.pem
      cert_file: /etc/prometheus/certs/client.crt
      key_file: /etc/prometheus/certs/client.key
      server_name: prometheus-remote-write.example.com
Note: Ensure that the certificate files are accessible to the Prometheus process and have appropriate permissions set.
Warning: Misconfiguring mTLS can lead to connection failures. Always verify the certificate chain and ensure that the server name matches the certificate's Subject Alternative Name (SAN).

8. Securing Alertmanager Communication

Configuration Option Description Default Value Notes
alertmanager.tls-config Configures TLS settings for Alertmanager communication None Must be specified to enable TLS
alertmanager.http.config HTTP client configuration for Alertmanager Default HTTP client settings Allows customization of timeouts and retries

Example: Alertmanager TLS Configuration

alertmanager:
  tls-config:
    ca-file: /etc/prometheus/certs/ca.crt
    cert-file: /etc/prometheus/certs/client.crt
    key-file: /etc/prometheus/certs/client.key
    server-name: alertmanager.example.com
    insecure-skip-tls-verify: false
Note: Ensure that the Alertmanager server has a matching TLS configuration to establish secure communication.
Warning: Disabling TLS or setting insecure-skip-tls-verify to true can expose sensitive alert data.
Field Type Description Required
ca-file string Path to the CA certificate file Yes, if TLS is enabled
cert-file string Path to the client certificate file No, unless mTLS is required
key-file string Path to the client key file No, unless mTLS is required
server-name string Server name to verify against the certificate No
insecure-skip-tls-verify boolean Disable TLS certificate verification (not recommended) No

9. Using Secrets Management

Integrating secrets management with Prometheus is crucial for securing sensitive configuration data like authentication tokens, API keys, and credentials.

Component Secrets Management Tool Integration Method Notes
Prometheus Config file_sd_configs Configuration files with secret placeholders Requires external secret management system to inject values at runtime
Alertmanager Config webhook_configs Secrets in url or headers Supports environment variable substitution via ${VAR}
Remote Write basic_auth Secrets in username and password_file Passwords can be stored in files or environment variables

Example: Using Environment Variables for Secrets

remote_write:
  - url: http://prometheus-remote-write:9090/api/v1/write
    basic_auth:
      username: ${PROM_USER}
      password_file: /etc/prometheus/secrets/password.txt
Note: Ensure that secrets are not stored in plain text within configuration files. Use tools like envsubst, secretsmanager, or hashicorp/vault to inject secrets at runtime.
Warning: Never commit secrets to version control systems. Use secret management solutions like HashiCorp Vault, AWS Secrets Manager, or Kubernetes Secrets for secure handling.

Best Practices for Secret Management

  • Use dedicated secret management tools to store and rotate credentials.
  • Implement role-based access control (RBAC) for secret access.
  • Automate secret rotation to reduce exposure window.
  • Use environment variable substitution in configuration files for runtime injection.

10. Auditing Prometheus Access

Event Type Description Fields Notes
access_granted Successful access to Prometheus metrics endpoint user, ip, endpoint Logged when authentication is successful
access_denied Failed access attempt to Prometheus endpoint user, ip, endpoint Logged when authentication fails or user lacks permissions
config_change Change in Prometheus configuration file user, config_file, changes Logged when config file is modified or reloaded

Example: Audit Log Entry Format

{
  "timestamp": "2026-01-15T10:30:00Z",
  "event_type": "access_granted",
  "user": "monitoring_admin",
  "ip": "192.168.1.100",
  "endpoint": "/metrics",
  "status": "success"
}
Note: Ensure audit logs are stored securely and rotated regularly to prevent log exhaustion attacks.
Warning: Access logs should not contain sensitive data like passwords or tokens. Use log filtering to remove such fields.

11. Implementing Role-Based Access

Permission Description Required Role Notes
read Allows reading metrics and configuration data Viewer Can view all exposed metrics but not modify anything
write Allows modifying configuration and rules Editor Can add or remove recording rules, alerting rules, and configuration changes
admin Full administrative access including system-level operations Admin Can manage users, permissions, and Prometheus server settings

Example: RBAC Configuration in Prometheus

rule_files:
  - rules.yml

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: [localhost:9090]

# RBAC Configuration
rbac:
  roles:
    - name: "Viewer"
      permissions:
        - read
    - name: "Editor"
      permissions:
        - read
        - write
    - name: "Admin"
      permissions:
        - read
        - write
        - admin
Note: RBAC is implemented via the rbac configuration block in Prometheus, which allows fine-grained control over access levels for different users or services.

12. Securing Configuration Files

Configuration File Security Considerations Best Practices
prometheus.yml Contains sensitive metrics and alerting rules. Should not be readable by unauthorized users. Set restrictive file permissions (e.g., 600) and ensure only Prometheus user can read it.
alert.rules.yml Stores alerting rules that may expose system details or trigger actions. Encrypt if stored outside of Prometheus and restrict access to administrators only.
scrape_configs.yml Includes targets and relabeling rules that may reveal internal infrastructure. Use secrets management tools to store credentials and avoid hardcoding sensitive values.

Example: Secure prometheus.yml Permissions

chmod 600 /etc/prometheus/prometheus.yml
chown prometheus:prometheus /etc/prometheus/prometheus.yml
Note: Ensure that configuration files are backed up securely and rotated periodically to prevent unauthorized access.
Warning: Never store credentials or secrets directly in configuration files. Use environment variables or external secret stores.

Was this article helpful?

Related Cheat Sheets