Monitoring Grafana

1. Using Built-in Metrics (/metrics endpoint)

Metric	Description	Labels	Type
grafana_build_info	Build information for Grafana	`version`, `commit`, `branch`	Gauge
grafana_http_request_duration_seconds	HTTP request duration in seconds	`method`, `handler`, `status_code`	Histogram
grafana_http_inflight_requests	Number of inflight HTTP requests	`method`, `handler`	Gauge
grafana_datasource_query_duration_seconds	Duration of datasource queries in seconds	`datasource`, `status`	Histogram
grafana_alerting_rule_eval_duration_seconds	Duration of alert rule evaluations in seconds	`rule`, `status`	Histogram

Example: Fetching metrics via cURL

curl http://localhost:3000/metrics

Note: The /metrics endpoint is only accessible to admin users by default.

2. Monitoring Dashboard Load Times

Property	Description	Default	Notes
dashboard.load.timeout	Timeout for loading dashboard data	`30s`	Used to prevent long-running queries from blocking dashboard rendering
dashboard.load.maxRetries	Maximum number of retries for failed dashboard loads	`3`	Attempts to reload dashboard data on failure

Example: Configuring dashboard load timeout

[dashboard]
load.timeout = 60s
load.maxRetries = 5

Note: Grafana will log slow dashboard load times to the dashboard.load.duration metric for monitoring.

Dashboard Load Time Metrics

dashboard.load.duration: Histogram of dashboard load times in seconds
dashboard.load.errors: Counter for failed dashboard loads
dashboard.render.duration: Histogram of dashboard rendering times

3. Monitoring Query Performance

Property	Description	Default	Notes
query_timeout	Maximum time a query can run before being cancelled	`30s`	Configurable in `grafana.ini`
max_concurrent_queries	Maximum number of queries that can run simultaneously	`100`	Prevents resource exhaustion
query_stats	Enables query statistics logging	`false`	Set to `true` for performance insights

Example: Configuring Query Timeout

[analytics]
query_timeout = 60s

[performance]
max_concurrent_queries = 200

Note: Query performance can be monitored via the /api/datasources endpoint or Grafana's built-in performance dashboards.

Query Type	Timeout (seconds)	Notes
`prometheus`	`30`	Default Prometheus query timeout
`loki`	`60`	Higher timeout for log queries
`elasticsearch`	`45`	Adjust based on index size

4. Tracking Alert Evaluation

Property	Description	Values	Notes
alert.eval	Alert evaluation status	`pending`, `ok`, `alerting`, `no_data`, `error`	Indicates current state of alert evaluation
alert.lastEvalTime	Timestamp of last evaluation	`ISO 8601`	Time when alert was last evaluated
alert.threshold	Threshold value for alert condition	`number`	Value used to compare against metric data

Example: Alert Evaluation Status

{
  "alert": {
    "eval": "alerting",
    "lastEvalTime": "2026-01-15T14:30:00Z",
    "threshold": 95
  }
}

Note: Grafana evaluates alerts based on the configured time window and query results. The alert.eval field reflects whether the alert condition was met.

Warning: If alert.eval is error, check the alert query and data source configuration.

5. Monitoring Database Connections

Property	Description	Default Value	Notes
datasource	Name of the data source used for database connection monitoring	None	Required field to specify the DB source
query	SQL query to fetch connection metrics	`SELECT count(*) FROM pg_stat_activity`	Customizable based on DB type
interval	How often to poll the database for connection stats	`30s`	Can be adjusted for performance needs

Example: Configuring DB Connection Monitoring

datasource: "PostgreSQL"
query: "SELECT count(*) FROM pg_stat_activity WHERE state = 'active'"
interval: "60s"

Note: Ensure that the database user has sufficient permissions to execute the monitoring query.

Warning: Frequent polling can impact database performance; adjust interval accordingly.

Connection Status	Threshold	Action	Severity
`active`	> 100 connections	Alert	High
`idle`	> 500 connections	Log	Medium
`idle_in_transaction`	> 10 connections	Alert	High

6. Tracking User Sessions

Property	Description	Default Value	Notes
session.timeout	Duration after which a session expires	`24h`	Can be set in hours, minutes, or seconds
session.renew	Whether to extend session on activity	`true`	Set to `false` to disable auto-renewal

Example: Session Configuration in grafana.ini

[auth]
session.timeout = 12h
session.renew = false

Note: Session tracking is managed via cookies and stored in the Grafana database or Redis if configured.

Session Management Summary

Session timeout can be customized for security or convenience
Auto-renewal helps maintain active sessions without re-authentication
Session data is persisted in Grafana's storage backend

7. Monitoring HTTP API Performance

Endpoint	Method	Description	Notes
/api/http-metrics	`GET`	Retrieve HTTP metrics for Grafana	Returns detailed performance data including response times and throughput
/api/http-metrics/config	`GET`	Get current HTTP metrics configuration	Shows enabled/disabled metrics and collection intervals
/api/http-metrics/config	`PUT`	Update HTTP metrics configuration	Allows enabling/disabling specific metrics or adjusting collection intervals

Example: Fetching HTTP Metrics

curl -H "Authorization: Bearer <token>" \
     http://localhost:3000/api/http-metrics

Example: Updating HTTP Metrics Configuration

{
  "enabled": true,
  "interval": 60,
  "metrics": [
    "response_time",
    "throughput"
  ]
}

Note: HTTP metrics are collected by default in Grafana v10.0+ and can be configured via grafana.ini or the API.

8. Using Health Check Endpoint (/api/health)

Property	Description	Example
Endpoint	Health check endpoint for Grafana	`/api/health`
Method	HTTP method to access the endpoint	`GET`
Response Format	JSON response indicating Grafana status	`{ "database": "ok", "license": "valid" }`

Example: Check Grafana Health

$ curl http://localhost:3000/api/health
{
  "database": "ok",
  "license": "valid"
}

Note: The /api/health endpoint is useful for integration with monitoring systems and container orchestration tools like Kubernetes.

9. Creating Self-Monitoring Dashboard

Example: Basic self-monitoring dashboard setup

{
  "dashboard": {
    "title": "Grafana Self-Monitoring",
    "tags": ["grafana", "monitoring"],
    "timezone": "browser",
    "schemaVersion": 16,
    "version": 0
  },
  "panels": [
    {
      "id": 1,
      "type": "graph",
      "title": "Grafana CPU Usage",
      "targets": [
        {
          "expr": "rate(node_cpu_seconds_total{job=\"grafana\",mode!=\"iowait\"}[5m])",
          "legendFormat": "{{cpu}}"
        }
      ]
    },
    {
      "id": 2,
      "type": "stat",
      "title": "Active Users",
      "targets": [
        {
          "expr": "count by (user) (grafana_user_logged_in)"
        }
      ]
    }
  ]
}

Property	Description	Notes
dashboard.title	Dashboard name	Must be unique within the folder
dashboard.tags	Dashboard tags for filtering	Used in dashboard search and filtering
dashboard.timezone	Dashboard timezone	Defaults to browser timezone
panels[].id	Panel identifier	Unique within dashboard, required for editing
panels[].type	Panel type (graph, stat, etc.)	See Grafana panel types documentation
panels[].title	Panel title	Visible in dashboard view
panels[].targets[].expr	PromQL expression for data source	Used to query metrics from Prometheus or other sources

Note: Ensure the Prometheus data source is configured and accessible to Grafana for querying metrics.

Warning: Dashboard JSON must be valid to prevent import errors. Use Grafana's dashboard import feature to validate.

10. Setting Up Grafana Alerting

Example: Create a new alert rule

name: "High CPU Usage Alert"
condition: "A"
data:
  - refId: "A"
    datasourceUid: "grafana-loki-datasource"
    model:
      expr: "rate(container_cpu_usage_seconds_total[5m])>0.8"
      interval: "30s"
      maxDataPoints: 43200
      refId: "A"
for: "5m"
labels:
  severity: "warning"
annotations:
  summary: "CPU usage is above 80%"

Property	Description	Default
name	Name of the alert rule	None
condition	Reference to the query (e.g., "A")	None
data	Query configuration and datasource reference	None
for	Duration to wait before triggering the alert	"0m"
labels	Additional labels to attach to the alert	None
annotations	Metadata like summary or description	None

Note: Alert rules are evaluated based on the time range selected in the dashboard or query.

Warning: Ensure that your datasource is correctly configured before creating alert rules.

Alert Rule Configuration Summary

Name: Unique identifier for the alert
Condition: Query reference (e.g., "A")
Data Source: Must support metric queries
Duration: Time to wait before firing the alert
Labels & Annotations: For grouping and metadata