Writing LogQL Queries

1. Using Stream Selectors ({app="nginx"})

Stream selectors in LogQL are used to filter log streams based on label matchers. They form the core of any LogQL query and are essential for targeting specific logs.

Selector	Description	Example
app="nginx"	Matches logs where the `app` label equals `nginx`	`{app="nginx"}`
job=~"nginx.*"	Matches logs where the `job` label matches a regex pattern	`{job=~"nginx.*"}`
level!="debug"	Matches logs where the `level` label does not equal `debug`	`{level!="debug"}`

Example: Filtering logs by app label

{app="nginx"}

This selector matches all log streams where the app label is set to nginx.

Example: Using regex matchers

{job=~"nginx.*"}

This selector matches all log streams where the job label starts with nginx.

Note: Stream selectors can be combined with other LogQL operations like | filter or | json to further refine results.

2. Using Label Filters

Filter Type	Description	Example
match	Matches logs with specific label values	`{job="api-server"}`
not match	Excludes logs with specific label values	`{job!="api-server"}`
regex match	Matches logs with label values matching a regex pattern	`{job=~"api-.*"}`
regex not match	Excludes logs with label values matching a regex pattern	`{job!~"api-.*"}`

Example: Filtering by Label Value

{job="nginx", instance="10.0.0.1"}

Example: Excluding Logs with Regex

{job!~"test-.*"}

3. Using Line Filters (|=, !=, |~, !~)

Operator	Description	Example	Notes
\|=	Matches lines containing the specified string	`{job="nginx"} \|= "error"`	Case-sensitive matching
!=	Excludes lines containing the specified string	`{job="nginx"} != "404"`	Case-sensitive exclusion
\|~	Matches lines matching the specified regex pattern	`{job="nginx"} \|~ "error.*500"`	Case-sensitive regex matching
!~	Excludes lines matching the specified regex pattern	`{job="nginx"} !~ "GET.*200"`	Case-sensitive regex exclusion

Example: Filtering Logs by Error Messages


{job="nginx"} |= "error"

Example: Excluding Specific Log Entries


{job="nginx"} != "404"

Example: Using Regex to Match Error Patterns


{job="nginx"} |~ "error.*500"

Example: Excluding Successful Requests


{job="nginx"} !~ "GET.*200"

Note: Line filters can be combined with other LogQL operators to build complex queries. For example, {job="nginx"} |= "error" |~ "500" will match lines that contain both "error" and a 500 error code.

4. Using json Parser (| json)

Operator	Description	Example
\| json	Parses log lines that contain JSON formatted content and extracts fields from the parsed JSON object.	`{"level": "info", "message": "user logged in", "userId": "12345"}`

Example: Parsing JSON logs

label_replace({job="myapp"} | json, "user_id", "$1", "userId", "(.*?)")

Note: The | json parser extracts fields from JSON logs and makes them available for filtering and aggregation in LogQL.

5. Using logfmt Parser (| logfmt)

Operator	Description	Example
\| logfmt	Parses log lines using the `logfmt` format and extracts key-value pairs into labels.	`{job="myapp"} \| logfmt`

Example: Parsing logfmt logs

{job="myapp"} | logfmt

Note: The logfmt parser expects key-value pairs in the format key=value. If a log line does not conform to this format, it will not be parsed correctly.

6. Using pattern Parser (| pattern)

Operator	Description	Example	Notes
\| pattern	Parses log lines using a pattern to extract structured fields	{job="nginx"} \| pattern `(?P<ip>\S+) - (?P<user>\S+) \[(?P<timestamp>[^]]+)\] "(?P<method>\S+) (?P<url>[^"]+)" (?P<status>\d+) (?P<size>\d+)`	Matches and extracts fields from log lines using regex-like syntax

Example: Parsing log lines with pattern

{job="nginx"} | pattern `(?P<ip>\S+) - (?P<user>\S+) \[(?P<timestamp>[^]]+)\] "(?P<method>\S+) (?P<url>[^"]+)" (?P<status>\d+) (?P<size>\d+)`

Note: The pattern operator supports named capture groups using ?P<name> syntax, similar to Go's regex engine.

Key Features of pattern Parser

Extracts structured fields from unstructured log lines
Uses named capture groups to define field names
Supports standard regex syntax for patterns
Can be combined with other LogQL operators like | json

7. Using line_format Expression (| line_format)

Expression	Description	Example
\| line_format	Formats log lines using a template string to extract and display specific parts of the log entry.	`{"timestamp": "{{.ts}}", "message": "{{.msg}}"}`

Example: Formatting Log Lines with line_format

query | line_format &#34;{&#34;timestamp&#34;: &#34;{{.ts}}&#34;, &#34;message&#34;: &#34;{{.msg}}&#34;}&#34;

Note: The line_format expression is useful for transforming raw log lines into structured JSON output, which can then be used for better visualization and analysis in Grafana.

8. Using label_format Expression (| label_format)

The label_format expression in LogQL allows you to rename, add, or transform labels in your log streams. It's particularly useful for normalizing label names or applying transformations to existing labels.

Expression	Description	Example
\| label_format	Renames or transforms labels in the log stream.	`{job="nginx"} \| label_format new_label=old_label`

Example: Renaming a Label

{job="nginx"} | label_format status_code=code

Example: Transforming a Label Value

{job="nginx"} | label_format status_code=upper(code)

Note: The label_format expression does not modify the original log line, only the labels associated with it.

9. Using Metric Queries

Query Type	Description	Example
rate()	Calculates the per-second average rate of increase of a metric over a time window.	`rate(http_requests_total[5m])`
sum()	Sums up values across a group of series.	`sum by (job) (http_requests_total)`
avg()	Averages values across a group of series.	`avg by (job) (http_requests_total)`
max()	Returns the maximum value across a group of series.	`max by (job) (http_requests_total)`
min()	Returns the minimum value across a group of series.	`min by (job) (http_requests_total)`
count()	Counts the number of series in a group.	`count by (job) (http_requests_total)`
stddev()	Calculates the standard deviation of values in a group.	`stddev by (job) (http_requests_total)`
quantile()	Calculates a quantile (e.g., 95th percentile) of values in a group.	`quantile(0.95, http_requests_total)`

Example: Rate of HTTP Requests


rate(http_requests_total[5m])

Example: Sum by Job


sum by (job) (http_requests_total)

Example: Quantile of Response Times


quantile(0.95, http_response_time_seconds)

Note: Metric queries in Grafana are used to aggregate and analyze time-series data. They can be combined with labels to filter or group results.

10. Using Aggregation Operators

Operator	Description	Example
sum	Returns the sum of values in a group	`sum by (job) (rate(logs[1m]))`
avg	Returns the average of values in a group	`avg by (job) (rate(logs[1m]))`
max	Returns the maximum value in a group	`max by (job) (rate(logs[1m]))`
min	Returns the minimum value in a group	`min by (job) (rate(logs[1m]))`
count	Returns the count of values in a group	`count by (job) (rate(logs[1m]))`
stddev	Returns the standard deviation of values in a group	`stddev by (job) (rate(logs[1m]))`
quantile	Returns the specified quantile of values in a group	`quantile 0.95 by (job) (rate(logs[1m]))`

Example: Aggregating Logs by Job

sum by (job) (rate(logs[1m]))

This query calculates the rate of logs per minute, grouped by job name, and sums them up.

11. Filtering by Log Level

Log Level	Expression	Description
trace	`level="trace"`	Most verbose logging level, includes detailed trace information
debug	`level="debug"`	Debug-level logs, useful for development and troubleshooting
info	`level="info"`	General information messages, typically used for operational events
warn	`level="warn"`	Warning messages indicating potential issues or non-critical problems
error	`level="error"`	Error messages indicating failures or critical issues that require attention
fatal	`level="fatal"`	Fatal errors that cause the application to stop or crash

Example: Filter logs by error level

{level="error"}

Example: Filter logs by multiple levels

{level="error" or level="warn"}

12. Using Detected Fields

Field	Description	Usage
detected_fields	Fields automatically detected from log lines	Used in LogQL to filter or group logs by detected fields

Example: Filtering Logs by Detected Fields

{job="application"} |= "error" | json | detected_fields

Example: Grouping Logs by Detected Fields

{job="application"} |= "error" | json | group by (detected_fields)