Troubleshooting Grafana

1. Debugging Data Source Connections

Step	Action	Description
1	Check Data Source Status	Go to `Configuration > Data Sources` and verify the data source is enabled and has no error messages.
2	Test Connection	Click `Test Connection` in the data source configuration. If it fails, review the error logs for details.
3	Review Logs	Check Grafana server logs for error messages related to the data source connection (e.g., `grafana.log`).
4	Verify Credentials	Ensure the data source credentials (username, password, API key) are correct and have required permissions.
5	Network Access	Confirm that the Grafana server can reach the data source endpoint (e.g., via `curl` or `ping`).

Example: Testing Data Source Connection via CLI

curl -v https://your-influxdb-host:8086/health

Note: If using InfluxDB, ensure the http port is accessible and the database name matches exactly.

Warning: Never store sensitive credentials in plain text in Grafana config files. Use secure secret management.

2. Resolving Query Timeout Issues

Configuration	Value	Description
query_timeout	`30s`	Default timeout for query execution
max_concurrent_queries	`100`	Maximum concurrent queries allowed per datasource
datasource.timeout	`60s`	Timeout for datasource requests (per datasource)

Example: Adjusting Query Timeout in Grafana Configuration

[query]
timeout = 60s

[datasources]
timeout = 90s

Note: Increasing query timeout can lead to longer dashboard load times and may impact Grafana's responsiveness under high load.

Common Causes of Query Timeouts

Slow database or data source
Complex queries without proper indexing
Network latency between Grafana and data source
High concurrent query load on the backend

3. Fixing Dashboard Load Errors

Error Type	Description	Common Causes	Solution
Dashboard not found	Dashboard fails to load due to missing or incorrect ID	Incorrect dashboard UID, deleted dashboard, wrong folder	Verify dashboard UID in URL, check folder permissions, recreate if deleted
Panel rendering errors	Individual panels fail to render due to query or data issues	Invalid queries, missing data sources, timeout errors	Check query syntax in panel editor, verify data source connectivity
Data source errors	Errors when fetching data from backend systems	Authentication issues, network problems, invalid credentials	Test data source connection in Grafana UI, verify credentials and network access
Permission denied	Access denied when trying to view or edit a dashboard	Insufficient user roles, missing permissions in organization	Ensure user has appropriate role (Viewer/Admin), check organization settings

Example: Panel Query Debugging

SELECT * FROM metrics WHERE time >= now() - 1h LIMIT 100

Note: Use Grafana's Explore feature to test queries before applying them to panels.

Warning: Dashboard load errors can sometimes be caused by browser cache. Try clearing the cache or using an incognito/private browsing window.

4. Debugging Alerts Not Firing

Step	Action	Details
Check Alert Rule	Verify rule syntax and conditions	Ensure the rule is correctly written in Grafana's alerting UI or YAML format. Check for syntax errors or invalid expressions.
Check Data Source	Confirm data source is accessible	Validate that the query used in the alert rule can successfully fetch data from the configured data source (e.g., Prometheus, InfluxDB).
Check Silence	Review silence settings	Ensure no silences are active that might suppress the alert. Silences can be managed in the Grafana UI under "Alerts" > "Silences".
Check Notification Channel	Verify alert notification channels	Confirm that the alert is configured to send notifications through a valid channel (e.g., email, Slack, webhook). Test the channel manually.
Check Alert State	Review alert state in UI	Go to the "Alerts" section in Grafana and check if the alert is in a "Firing" or "Pending" state. If not, it may not have triggered.
Check Logs	Review Grafana logs	Check Grafana server logs for errors or warnings related to alerting. Look for messages indicating issues with rule evaluation or notification delivery.

Example: Alert Rule Syntax

name: High CPU Usage Alert
expr: instance_cpu_usage > 80
for: 5m
labels:
  severity: critical
annotations:
  summary: "High CPU usage detected"
  description: "CPU usage is above 80% for more than 5 minutes"

Note: Ensure that the expression instance_cpu_usage > 80 returns valid data points when evaluated in the context of your data source.

Warning: If the alert condition is too strict or based on a metric that doesn't exist, it will never fire. Always validate the query before setting up an alert.

5. Resolving Permission Errors

Issue	Cause	Solution
Dashboard not loading	Insufficient user permissions to view dashboard	Ensure user has at least `Viewer` role in the organization or folder
Cannot edit dashboard	User lacks `Editor` or `Admin` role	Assign appropriate role in Grafana UI or via API
Plugin access denied	Plugin requires `Admin` privileges but user is not admin	Grant `Admin` role or use a plugin that does not require admin rights
Data source access fails	User does not have permission to access the data source	Verify user has `Viewer` or higher role on the data source

Example: Assigning Role via API

curl -X POST "http://localhost:3000/api/org/users/1" \
  -H "Authorization: Bearer <your_token>" \
  -H "Content-Type: application/json" \
  -d '{"role": "Editor"}'

Note: Ensure the user is part of the correct organization and that the role assignment matches the scope (org-level or folder-level).

6. Fixing Plugin Installation Errors

Error Message	Possible Cause	Solution
plugin not found	Plugin name misspelled or typo in config	Verify plugin name matches official Grafana plugin registry
plugin signature invalid	Plugin not signed or signature mismatch	Reinstall plugin from official Grafana plugin repository
plugin version incompatible	Plugin requires newer Grafana version	Update Grafana to compatible version or use older plugin
plugin directory permission denied	Insufficient permissions to write plugin files	Ensure Grafana user has write access to plugins directory

Example: Plugin Installation Command

grafana-cli plugins install grafana-azure-monitor-datasource

Note: Ensure the plugin is compatible with your Grafana version. Use grafana-cli plugins list to verify installed plugins.

Warning: Installing unsigned plugins can pose security risks. Only install from trusted sources.

7. Debugging Variable Issues

Issue	Potential Causes	Solutions
Variables not populating	Incorrect variable syntax in query Missing or incorrect data source reference Time range filter mismatch	Validate variable query syntax using Grafana's variable editor Ensure data source is selected and accessible Check time range filters align with data availability
Unexpected variable values	Incorrect data source type used Query returns unexpected results due to filters or formatting Variable is not updated on dashboard refresh	Verify data source type matches query expectations Review and adjust query filters or formatting logic Set variable refresh to "On Dashboard Load" if needed
Variable errors in logs	Permission issues with data source Network connectivity problems Invalid query syntax or unsupported features	Check user permissions and data source access Verify network connectivity to data source Validate query syntax and supported features in current Grafana version

Example: Variable Query Debugging

1. Open the dashboard variable editor
2. Select the problematic variable
3. Use "Test" button to run query manually
4. Check for error messages in the console or logs

Note: Use Grafana's built-in variable tester to isolate and fix query issues before applying changes to the dashboard.

8. Resolving CORS Errors

Issue	Description	Solution
CORS Policy Error	Browser blocks requests due to cross-origin restrictions	Configure `grafana.ini` with `allow_origin = *` or specific domain
Access-Control-Allow-Origin	Missing or incorrect `Origin` header in response	Add `allow_headers = *` and `allow_methods = GET, POST, PUT, DELETE`

Example: Configuring CORS in grafana.ini

[cors]
allow_origins = https://yourdomain.com, https://*.yourdomain.com
allow_headers = Content-Type, Authorization, X-Requested-With
allow_methods = GET, POST, PUT, DELETE, OPTIONS

Note: Ensure allow_origins includes all domains making requests to Grafana.

Warning: Setting allow_origins to * can be a security risk in production environments.

9. Fixing High Memory Usage

Issue	Solution	Notes
High memory usage in Grafana	Check for memory leaks or inefficient queries	Monitor with Grafana's built-in metrics and logs
Memory usage spikes	Review dashboard queries and data sources	Use query inspector to identify slow queries

Example: Memory Usage Monitoring

curl -X GET "http://localhost:3000/api/health" \
  -H "Authorization: Bearer <your_token>"

Example: Query Optimization

{"query": "SELECT * FROM metrics WHERE time > now() - 1h"}

Note: Use time > now() - 1h instead of time > $__from to avoid unnecessary data fetching.

Warning: Avoid using max or min without limits in queries as they can cause memory bloat.

Memory Optimization Tips

Use limit in queries to reduce data volume
Enable query caching for frequently accessed metrics
Review and optimize dashboard layouts to reduce concurrent queries

10. Debugging Authentication Issues

Issue	Possible Causes	Resolution Steps
User cannot login	`Invalid credentials`, `LDAP/SSO misconfiguration`, `User disabled`	Verify username/password Check `grafana.ini` auth settings Ensure user is not disabled in DB
SSO login fails	`OIDC/OAuth misconfigured`, `Token expired`, `Missing scopes`	Review OAuth provider settings in `grafana.ini` Check token expiration and refresh logic Ensure required scopes are granted
Permission denied after login	`Role not assigned`, `Org membership missing`, `Dashboard access denied`	Assign correct role in Grafana UI or API Verify user is part of organization Check dashboard permissions in Grafana

Example: Check Grafana logs for auth errors

tail -f /var/log/grafana/grafana.log | grep -i "auth"

Note: Enable log.level = debug in grafana.ini for detailed auth tracing.

Warning: Do not expose sensitive auth logs in production environments.

11. Resolving Rendering Issues

Issue	Possible Cause	Solution
Dashboard not loading	Network connectivity or permission issues	Check network access and user permissions in Grafana
Graphs not displaying data	Incorrect query or data source misconfiguration	Verify queries in Explore view and check data source settings
Panels appear blank	Missing or invalid panel configuration	Recreate the panel or check JSON configuration for errors
Time range issues	Timezone mismatch or incorrect time filters	Ensure timezone is set correctly in dashboard settings

Example: Check Data Source Configuration

datasources:
  - name: "Prometheus"
    type: prometheus
    access: proxy
    url: "http://localhost:9090"
    isDefault: true

Note: Ensure that the data source URL is accessible from the Grafana server.

Warning: Incorrect time range settings can cause panels to show no data even if the query is correct.

12. Using Grafana Support Bundle

Example: Generate and analyze support bundle

# Generate support bundle
grafana-cli support-bundle generate

# Upload to Grafana support
grafana-cli support-bundle upload

# View bundle contents locally
unzip support-bundle.zip -d /tmp/support-bundle

Command	Description	Options
grafana-cli support-bundle generate	Generates a support bundle containing logs, config, and system info	`--output-dir` to specify output directory
grafana-cli support-bundle upload	Uploads the generated bundle to Grafana support	`--api-key` for authentication, `--url` to specify endpoint
grafana-cli support-bundle analyze	Performs local analysis of the bundle for common issues	`--output-format` to specify output (json, html)

Note: The support bundle includes sensitive data such as configuration files and logs. Review before uploading.

Warning: Do not share the support bundle publicly or with unauthorized parties.

Support Bundle Contents

Grafana server logs
Configuration files (grafana.ini, provisioning)
Database schema and metadata
Plugin list and versions
System information (OS, CPU, memory)