Writing TraceQL Queries
1. Understanding TraceQL Syntax
| Element | Description | Example |
|---|---|---|
| traceql | Query language for tracing data in Grafana | traceql |
| span | Represents a single operation or task in a trace | span.name = "GET /api/users" |
| trace | Collection of spans that represent a single request flow | trace.duration > 500ms |
| service | Logical grouping of spans related to a specific service | span.service = "user-service" |
| operation | Specific action or method performed by a span | span.operation = "POST /login" |
| tag | Key-value pairs attached to spans for filtering and grouping | span.tags["http.status_code"] = 200 |
| duration | Time taken for a span or trace to complete | span.duration > 100ms |
| startTime | Timestamp when a span started | span.startTime > 1634567890 |
| endTime | Timestamp when a span ended | span.endTime < 1634567900 |
| parentSpanId | Identifier of the parent span in a trace | span.parentSpanId = "abc123" |
2. Using Span Selectors ({resource.service.name="api"})
| Selector | Description | Example |
|---|---|---|
| resource.service.name | Matches spans by the service name attribute | {resource.service.name="api"} |
Note: Span selectors in TraceQL are used to filter and scope trace data. The
resource.service.name attribute is a common way to identify services within a trace.
3. Filtering by Span Attributes
| Attribute | Description | Example | Type |
|---|---|---|---|
| span.name | Name of the span | "GET /api/users" |
string |
| span.kind | Kind of the span (client, server, producer, consumer, internal) | "server" |
string |
| span.status | Status of the span (ok, error, unset) | "ok" |
string |
| span.duration | Duration of the span in nanoseconds | 123456789 |
int |
| service.name | Name of the service that generated the span | "user-service" |
string |
| http.method | HTTP method used in the span | "POST" |
string |
| http.status_code | HTTP status code returned by the span | 200 |
int |
| http.url | Full URL of the HTTP request | "https://api.example.com/users/123" |
string |
| db.system | Database system used (e.g., mysql, postgresql) | "postgresql" |
string |
| db.statement | Database query executed | "SELECT * FROM users WHERE id = $1" |
string |
4. Using Span Duration Filters
| Filter | Description | Example |
|---|---|---|
| span.duration | Filters spans by their duration in milliseconds | span.duration > 500 |
| span.duration < | Filters spans with duration less than specified value | span.duration < 100 |
| span.duration = | Filters spans with exact duration | span.duration = 250 |
| span.duration != | Filters out spans with specific duration | span.duration != 0 |
Note: Duration filters can be combined with other TraceQL operators for more complex queries.
5. Using Status Filters
| Filter | Description | Example |
|---|---|---|
| status | Filter traces by their status code | trace.status == 200 |
| status.code | Filter by numeric status code | trace.status.code >= 500 |
| status.message | Filter by status message text | trace.status.message == "Internal Server Error" |
6. Using Structural Operators (>>, ~, &&)
| Operator | Description | Example | Notes |
|---|---|---|---|
| > | Child operator - selects child elements | trace.span >> span.name |
Used to navigate to child spans or fields |
| ~ | Contains operator - checks if field contains a value | span.name ~ "user" |
Case-sensitive search within string fields |
| && | Logical AND operator - combines conditions | span.name ~ "login" && span.duration > 1000 |
Chain multiple conditions for filtering |
7. Using Aggregate Functions
| Function | Description | Example | Notes |
|---|---|---|---|
| avg() | Averages values in a series | avg(trace.duration) |
Useful for performance analysis |
| count() | Counts the number of values in a series | count(trace.duration) |
Can be used with filters to count specific traces |
| max() | Returns the maximum value in a series | max(trace.duration) |
Helps identify slowest traces |
| min() | Returns the minimum value in a series | min(trace.duration) |
Useful for finding fastest traces |
| sum() | Sums all values in a series | sum(trace.duration) |
Useful for total duration calculations |
| stddev() | Calculates standard deviation | stddev(trace.duration) |
Measures variability in trace durations |
Note: Aggregate functions can be combined with
by clauses to group results by specific labels or attributes.
8. Using rate() and count_over_time() Metrics
| Function | Description | Syntax | Notes |
|---|---|---|---|
| rate() | Calculates the per-second average rate of increase of a metric over a time window | rate(metric[time_window]) |
Useful for identifying trends in increasing metrics like request counts or error rates |
| count_over_time() | Counts the number of data points in a given time window | count_over_time(metric[time_window]) |
Helps in understanding data point density and sampling frequency |
Example: Using count_over_time() to check data point frequency
count_over_time(http_requests_total[1h])
Note: The
rate() function is particularly useful for counter metrics that increase over time, such as total HTTP requests or error counts. It accounts for resets in counter values.
Warning: When using
rate(), ensure that the time window is appropriate for your data sampling rate to avoid misleading results.
9. Combining Multiple Span Conditions
| Condition | Description | Example |
|---|---|---|
| span.name | Matches spans by name | span.name = "GET /api/users" |
| span.kind | Matches spans by kind (client, server, producer, consumer, internal) | span.kind = "server" |
| service.name | Matches spans by service name | service.name = "user-service" |
| span.duration | Matches spans by duration (in milliseconds) | span.duration > 500 |
| span.status | Matches spans by status (ok, error) | span.status = "error" |
| span.attributes | Matches spans by custom attributes | span.attributes.http.status_code = 500 |
Example: Combining Conditions in TraceQL
traceql
span.name = "GET /api/users"
AND span.kind = "server"
AND service.name = "user-service"
AND span.duration > 500
Example: Complex Attribute Filtering
traceql
service.name = "payment-service"
AND span.attributes.http.status_code = 500
AND span.status = "error"
Note: When combining multiple conditions, use parentheses to control precedence and improve readability.
10. Using TraceQL Editor
| Feature | Description | Usage |
|---|---|---|
| span.name | Filter by span name | span.name == "GET /api/users" |
| span.kind | Filter by span kind (server, client, etc.) | span.kind == "server" |
| span.duration | Filter by span duration | span.duration > 500ms |
| span.status | Filter by span status (ok, error) | span.status == "error" |
| span.attribute | Filter by custom attribute | span.attribute."http.status_code" == 500 |
Note: TraceQL supports complex queries with filtering, aggregation, and grouping. Use the
traceql language in Grafana's trace query editor.