Using Correlations

1. Understanding Correlations

Property	Description	Usage
correlation	Defines a relationship between two metrics or data sources	Used in dashboards to show how one metric changes with another
source	Initial metric or data source to correlate	Set as the base for comparison in a correlation
target	Secondary metric or data source to correlate with	Used to analyze the relationship with the source

Example: Correlation Query

SELECT 
  source_metric.value AS source_value,
  target_metric.value AS target_value
FROM 
  source_metric
JOIN 
  target_metric ON source_metric.time = target_metric.time

Note: Correlations are useful for identifying trends or dependencies between metrics in Grafana dashboards.

2. Creating Correlations Between Data Sources

Correlations in Grafana allow you to link data from different sources to provide context and enhance your dashboards. This feature is particularly useful for identifying relationships between metrics, logs, and traces.

Field	Description	Example
name	Name of the correlation	`"Service to Log Correlation"`
source	Source data source (e.g., Prometheus, Loki)	`"Prometheus"`
target	Target data source (e.g., Loki, Jaeger)	`"Loki"`
labels	Labels to match between sources	`["service", "instance"]`

Example: Creating a Correlation via API

{
  "name": "Service to Log Correlation",
  "source": {
    "uid": "prometheus-uid"
  },
  "target": {
    "uid": "loki-uid"
  },
  "labels": [
    "service",
    "instance"
  ]
}

Note: Correlations are only supported between data sources that are accessible from the same Grafana instance.

3. Configuring Correlation Labels

Label	Description	Usage
correlation.label	Defines the label used to identify a correlation in the data source	Used in query results to match related metrics or logs
correlation.value	Specifies the value that should be matched for correlation	Used to filter or group data based on the correlation value

Example: Setting up Correlation Labels in a Query

{"query": "SELECT * FROM logs WHERE correlation.label = 'request_id' AND correlation.value = 'abc123'"}

Note: Correlation labels must be defined in the data source configuration to enable cross-dataset filtering.

4. Setting Correlation Transformations

Transformation Type	Description	Usage	Notes
url	URL template for the correlation	`"url": "https://example.com/logs?query={{.raw}}"`	Use `{{.raw}}` to inject the raw value
label	Label to display for the correlation	`"label": "View Logs"`	Can include template variables
target	Target field to correlate with	`"target": "traceId"`	Must match a field in the data source

Example: Configuring URL Transformation

{
  "url": "https://logs.example.com/search?q={{.raw}}",
  "label": "View Logs",
  "target": "traceId"
}

Note: The {{.raw}} template variable is replaced with the actual value from the field being correlated.

Warning: Ensure that the URL template is properly escaped to avoid injection vulnerabilities.

5. Using Logs to Traces Correlations

Correlating logs with traces in Grafana allows you to connect log entries with distributed trace data, providing a holistic view of application behavior.

Field	Description	Example
log.line	Log line content	`[ERROR] User not found for ID: 12345`
trace.id	Trace identifier	`abc123def456`
span.id	Span identifier within trace	`def456ghi789`
timestamp	Log timestamp	`2024-05-20T10:00:00Z`

Example: Log Entry with Trace ID

{
  "log.line": "[ERROR] User not found for ID: 12345",
  "trace.id": "abc123def456",
  "span.id": "def456ghi789",
  "timestamp": "2024-05-20T10:00:00Z"
}

Note: Ensure that your logging system includes trace identifiers to enable correlation.

Warning: Correlation requires consistent trace ID formats across logs and traces.

Correlation Benefits

Enables root cause analysis by linking logs to trace data
Improves debugging experience in complex distributed systems
Provides end-to-end visibility of application flow

6. Using Traces to Logs Correlations

Field	Description	Example
traceId	Unique identifier for a trace	`1234567890abcdef`
logLevel	Severity level of the log entry	`INFO`, `ERROR`
serviceName	Name of the service that generated the log	`user-service`
spanId	Identifier for a specific span within a trace	`0987654321fedcba`

Example: Correlating Logs with Traces in Grafana

{
  "traceId": "1234567890abcdef",
  "logLevel": "ERROR",
  "serviceName": "user-service",
  "spanId": "0987654321fedcba"
}

Note: Ensure that your tracing backend (e.g., Jaeger, OpenTelemetry) is configured to send trace and span IDs to your logging system for effective correlation.

7. Using Metrics to Traces Correlations

Property	Description	Example
correlation	Defines the relationship between metrics and traces	`{ "from": "metrics", "to": "traces" }`

Example: Configuring Metrics to Traces Correlation

{
  "name": "Metrics to Traces",
  "from": "metrics",
  "to": "traces",
  "query": "rate(http_requests_total[5m])"
}

Note: Correlations are supported in Grafana v9.0+ and require Loki or Tempo to be configured.

Warning: Ensure that the metrics and traces are aligned in time to avoid incorrect correlations.

Correlation Type	Description	Support Status
Metrics to Traces	Links metrics data to trace spans for deeper analysis	NEW v9.0+
Traces to Metrics	Links trace data to metrics for performance monitoring	DEPRECATED v9.0+

8. Configuring Correlation Display

Property	Description	Values	Notes
correlation.displayMode	Controls how the correlation is displayed in the panel	`table`, `graph`	Default is `table`
correlation.showValues	Whether to show values in the correlation display	`true`, `false`	Default is `true`
correlation.threshold	Threshold for displaying correlations	`0.0` to `1.0`	Default is `0.5`

Example: Configuring Correlation Display in Panel

{
  "correlation": {
    "displayMode": "graph",
    "showValues": true,
    "threshold": 0.7
  }
}

Note: The correlation display settings can be adjusted per panel and will affect how the data is visualized in the Grafana dashboard.

9. Managing Correlations via API

Property	Description	Example Value
correlationId	Unique identifier for the correlation	`"12345"`
source	Source dashboard or panel ID	`"dashboard-abc"`
target	Target dashboard or panel ID	`"dashboard-def"`
label	Label for the correlation	`"User Sessions"`

Example: Create Correlation via API

curl -X POST http://localhost:3000/api/dashboards/correlations \
  -H "Content-Type: application/json" \
  -d '{
    "source": "dashboard-abc",
    "target": "dashboard-def",
    "label": "User Sessions"
  }'

Example: Update Correlation via API

curl -X PUT http://localhost:3000/api/dashboards/correlations/12345 \
  -H "Content-Type: application/json" \
  -d '{
    "label": "Updated User Sessions"
  }'

Example: Delete Correlation via API

curl -X DELETE http://localhost:3000/api/dashboards/correlations/12345

10. Debugging Correlation Issues

Issue Description Debugging Steps

Missing or incorrect correlation IDs

Correlation IDs are not present or do not match across services.

Issue	Description	Debugging Steps
Missing or incorrect correlation IDs	Correlation IDs are not present or do not match across services.	`1. Check logs for missing correlation IDs. 2. Verify that services are generating and propagating IDs correctly. 3. Use Grafana's trace view to inspect correlation flow.`
Trace data not showing up in Grafana	Traces from services are not appearing in the trace view.	`1. Confirm that tracing is enabled in Grafana. 2. Check if the trace data is being sent to the correct backend (e.g., Jaeger, Tempo). 3. Validate that trace data is being ingested properly.`
Inconsistent trace context propagation	Trace context is not consistently propagated across services.	`1. Inspect service configurations for trace propagation settings. 2. Use tools like Jaeger UI to verify trace context consistency. 3. Ensure all services use the same tracing library and version.`

1. Check logs for missing correlation IDs.
2. Verify that services are generating and propagating IDs correctly.
3. Use Grafana's trace view to inspect correlation flow.

Trace data not showing up in Grafana

Traces from services are not appearing in the trace view.

1. Confirm that tracing is enabled in Grafana.
2. Check if the trace data is being sent to the correct backend (e.g., Jaeger, Tempo).
3. Validate that trace data is being ingested properly.

Inconsistent trace context propagation

Trace context is not consistently propagated across services.

1. Inspect service configurations for trace propagation settings.
2. Use tools like Jaeger UI to verify trace context consistency.
3. Ensure all services use the same tracing library and version.

Example: Correlation ID in Logs

[traceId=1234567890abcdef] [spanId=0987654321fedcba] Starting service request
[traceId=1234567890abcdef] [spanId=0987654321fedcba] Processing data
[traceId=1234567890abcdef] [spanId=0987654321fedcba] Completed successfully

Note: Ensure that your tracing backend (e.g., Tempo, Jaeger) is correctly configured to receive and store trace data for correlation.