Configuring Prometheus Operator

1. Installing Prometheus Operator

Install the Prometheus Operator using the official Helm chart or Kubernetes manifests.

Example: Install via Helm

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus-operator prometheus-community/kube-prometheus-stack

Component	Description	Version
prometheus-operator	Core operator for managing Prometheus instances	`v0.12.0`
prometheus	Core Prometheus server	`v2.49.1`
alertmanager	Alertmanager for handling alerts	`v0.25.0`

2. Creating Prometheus Custom Resource

Create a Prometheus custom resource to define the Prometheus instance configuration.

Example: Basic Prometheus CR

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus-instance
spec:
  serviceAccountName: prometheus
  resources:
    requests:
      memory: "4Gi"
      cpu: "2"
  retention: 15d
  storage:
    volumeClaimTemplate:
      spec:
        resources:
          requests:
            storage: "50Gi"

Field	Description	Required
serviceAccountName	Name of the service account to use for Prometheus	Yes
resources.requests.cpu	CPU request for the Prometheus pod	No
resources.requests.memory	Memory request for the Prometheus pod	No
retention	Duration to retain data	No
storage.volumeClaimTemplate	Template for persistent storage	No

3. Configuring ServiceMonitor Resources

Use ServiceMonitor to define how Prometheus scrapes services.

Example: ServiceMonitor CR

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-app-monitor
spec:
  selector:
    matchLabels:
      app: example-app
  endpoints:
  - port: metrics
    interval: 30s

Field	Description	Required
selector.matchLabels	Label selector to find services	Yes
endpoints.port	Port name or number to scrape	Yes
endpoints.interval	Scrape interval (default 30s)	No

4. Using PodMonitor Resources

PodMonitor is used to scrape metrics from pods directly.

Example: PodMonitor CR

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: example-pod-monitor
spec:
  selector:
    matchLabels:
      app: example-app
  podMetricsEndpoints:
  - port: metrics

Field	Description	Required
selector.matchLabels	Label selector to find pods	Yes
podMetricsEndpoints.port	Port name or number to scrape	Yes

5. Creating PrometheusRule Resources

PrometheusRule defines alerting and recording rules for Prometheus.

Example: PrometheusRule CR

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: example-rules
spec:
  groups:
  - name: example-group
    rules:
    - alert: HighCPUUsage
      expr: rate(container_cpu_usage_seconds_total[5m]) > 0.8
      for: 10m
      labels:
        severity: page
      annotations:
        summary: "High CPU usage detected"

Field	Description	Required
groups.name	Name of the rule group	Yes
groups.rules.alert	Name of the alert rule	Yes
groups.rules.expr	PromQL expression for the rule	Yes
groups.rules.for	Duration to wait before firing the alert	No

6. Configuring AlertmanagerConfig

AlertmanagerConfig allows customizing alert routing and inhibition rules.

Example: AlertmanagerConfig CR

apiVersion: monitoring.coreos.com/v1
kind: AlertmanagerConfig
metadata:
  name: example-alertmanager-config
spec:
  route:
    groupBy:
    - job
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 1h
    receiver: default
  receivers:
  - name: default
    emailConfigs:
    - to: "admin@example.com"

Field	Description	Required
route.groupBy	Labels to group alerts by	No
route.groupWait	Time to wait before sending the first alert	No
receivers.name	Name of the receiver	Yes
receivers.emailConfigs.to	Email address to send alerts to	Yes

7. Using ThanosRuler Resources

ThanosRuler is used to run alerting rules in a Thanos setup.

Example: ThanosRuler CR

apiVersion: monitoring.coreos.com/v1
kind: ThanosRuler
metadata:
  name: thanos-ruler-instance
spec:
  image: "quay.io/thanos/thanos:v0.32.0"
  replicas: 2
  resources:
    requests:
      memory: "2Gi"
      cpu: "1"

Field	Description	Required
image	Docker image for ThanosRuler	No
replicas	Number of replicas to run	No
resources.requests.cpu	CPU request for the ThanosRuler pod	No
resources.requests.memory	Memory request for the ThanosRuler pod	No

8. Setting Resource Limits

Define CPU and memory limits for Prometheus and related components.

Example: Resource Limits in CR

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus-with-limits
spec:
  resources:
    limits:
      cpu: "2"
      memory: "8Gi"
    requests:
      cpu: "1"
      memory: "4Gi"

Field	Description	Required
resources.limits.cpu	Maximum CPU allocation for the pod	No
resources.limits.memory	Maximum memory allocation for the pod	No
resources.requests.cpu	Minimum CPU allocation for the pod	No
resources.requests.memory	Minimum memory allocation for the pod	No

9. Configuring Persistent Storage

Configure persistent storage for Prometheus to retain data across restarts.

Example: Persistent Storage in CR

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus-with-storage
spec:
  storage:
    volumeClaimTemplate:
      spec:
        resources:
          requests:
            storage: "100Gi"
        accessModes:
        - ReadWriteOnce

Field	Description	Required
storage.volumeClaimTemplate.spec.resources.requests.storage	Storage size requested	No
storage.volumeClaimTemplate.spec.accessModes	Access mode for the volume (e.g., ReadWriteOnce)	No

10. Using Additional Scrape Configs

Add custom scrape configurations to Prometheus via additionalScrapeConfigs.

Example: Additional Scrape Configs

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus-with-additional-configs
spec:
  additionalScrapeConfigs:
    name: custom-scrape-configs
    key: prometheus.yml

Field	Description	Required
additionalScrapeConfigs.name	Name of the secret or configmap containing configs	No
additionalScrapeConfigs.key	Key in the secret or configmap containing the config file	No

11. Implementing Remote Write with Operator

Configure Prometheus to send metrics to a remote storage system using remoteWrite.

Example: Remote Write Configuration

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus-remote-write
spec:
  remoteWrite:
  - url: "https://prometheus.example.com/api/v1/write"
    queueConfig:
      capacity: 1000
      maxSamplesPerSend: 500

Field	Description	Required
remoteWrite.url	URL of the remote write endpoint	Yes
remoteWrite.queueConfig.capacity	Maximum number of samples to queue	No
remoteWrite.queueConfig.maxSamplesPerSend	Maximum samples to send per batch	No

12. Monitoring Operator Resources

Monitor the Prometheus Operator itself using Prometheus.

Example: Monitoring Operator

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-operator-monitor
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus-operator
  endpoints:
  - port: metrics

Field	Description	Required
selector.matchLabels.app.kubernetes.io/name	Label to select operator pods	Yes
endpoints.port	Port name or number to scrape operator metrics	Yes