Skip to main content
Mapagam

Implementing Advanced Logstash Filters

1. Using Ruby Filter

Property Description Values
add_field Adds a field to the event string
remove_field Removes a field from the event string
source Source code to execute string
code Code to execute in Ruby context string

Example: Using Ruby Filter to Modify Fields

filter {
  ruby {
    code => "event.set('modified', event.get('message').upcase)"
  }
}

Example: Using Ruby Filter to Add Fields

filter {
  ruby {
    code => "event.set('processed_at', Time.now)"
  }
}
Note: The Ruby filter allows complex transformations using Ruby code. Be cautious with performance as Ruby code runs in the Logstash process.

2. Using Translate Filter

Property Description Default Example
source Field to translate nil source => "status_code"
target Field to store translated value nil target => "status_text"
dictionary_path Path to dictionary file (JSON or CSV) nil dictionary_path => "/etc/logstash/dictionaries/status.json"
fallback Value to use if translation not found "NOT_FOUND" fallback => "Unknown Status"

Example: Translate Filter Configuration

filter {
  translate {
    source => "status_code"
    target => "status_text"
    dictionary_path => "/etc/logstash/dictionaries/status.json"
    fallback => "Unknown Status"
  }
}

Example: Dictionary File (JSON)

{
  "200": "OK",
  "404": "Not Found",
  "500": "Internal Server Error"
}
Note: The translate filter supports both JSON and CSV formats for dictionaries. Ensure the dictionary file is accessible to the Logstash process.

3. Using Fingerprint Filter

Option Values Notes
fingerprint md5, sha1, sha256 Hash algorithm to use for fingerprinting
source message, field Field or message to fingerprint
target fingerprint Field to store the fingerprint
salt string Optional salt to add to the fingerprint

Example: Fingerprinting a message field

fingerprint {
  source => ["message"]
  target => "fingerprint"
  method => "MD5"
}
Note: The fingerprint filter is useful for deduplicating logs or creating unique identifiers based on content.

4. Using Throttle Filter

Option Values Notes
id throttle Unique identifier for the filter
period 1s, 1m, 1h Time period to apply throttling over
rate 10 Maximum number of events allowed per period
burst 20 Maximum burst size before throttling applies
key message Field to use for grouping events (optional)

Example: Basic Throttle Filter Configuration

filter {
  throttle {
    id => "my_throttle"
    period => "1m"
    rate => 50
    burst => 100
  }
}

Example: Throttle Filter with Key Field

filter {
  throttle {
    id => "user_throttle"
    period => "10s"
    rate => 10
    burst => 20
    key => "[user_id]"
  }
}
Note: The throttle filter uses a token bucket algorithm to enforce rate limits. Events exceeding the configured rate are dropped or delayed depending on configuration.

5. Using Elapsed Filter

Field Description Default Notes
source Field containing the start timestamp start Must be a valid timestamp field
target Field to store the elapsed time elapsed Can be any valid field name
unit Unit for elapsed time milliseconds Options: nanoseconds, microseconds, milliseconds, seconds, minutes, hours
reset Whether to reset the elapsed timer false Boolean value to control reset behavior

Example: Basic Elapsed Filter Usage

filter {
  elapsed {
    source => "@timestamp"
    target => "duration_ms"
    unit => "milliseconds"
  }
}

Example: Elapsed Filter with Reset

filter {
  elapsed {
    source => "start_time"
    target => "time_taken"
    unit => "seconds"
    reset => true
  }
}

6. Using Metrics Filter

Field Description Default Notes
add_field Adds a field to the event {} Map of field names to values
remove_field Removes a field from the event [] Array of field names to remove
metrics Defines the metrics to be collected {} Configuration block for metrics

Example: Basic Metrics Filter Configuration

filter {
  metrics {
    add_field => { "metric_name" => "request_count" }
    remove_field => [ "message" ]
    metrics {
      "requests" => { "type" => "counter", "value" => 1 }
    }
  }
}

Example: Metrics Filter with Timer

filter {
  metrics {
    add_field => { "timer_name" => "response_time" }
    metrics {
      "response_time_ms" => { "type" => "timer", "value" => "duration" }
    }
  }
}
Note: The Metrics filter is useful for collecting and aggregating statistics about log data, such as counters, timers, and gauges. It can be used to monitor application performance or track event rates.

7. Using Sleep Filter

Property Description Default Value Notes
duration Time to sleep in seconds 0 Must be a positive number
unit Time unit for duration seconds Valid values: seconds, milliseconds, microseconds

Example: Basic Sleep Filter Usage

filter {
  sleep {
    duration => 2
  }
}
Note: The sleep filter is primarily used for testing and debugging purposes to simulate latency in the Logstash pipeline.

8. Using Cidr Filter

Field Description Example
add_field Adds a field to the event with the result of the filter "network_category" => "internal"
add_tag Adds a tag to the event if the IP matches any CIDR "internal_network"
cidr List of CIDR blocks to match against ["192.168.0.0/16", "10.0.0.0/8"]
ip Field containing the IP address to check "client_ip"
remove_field Removes a field from the event after processing "client_ip"
remove_tag Removes a tag from the event if the IP matches "external_network"

Example: Cidr Filter Configuration

filter {
  cidr {
    ip => "client_ip"
    cidr => [
      "192.168.0.0/16",
      "10.0.0.0/8"
    ]
    add_tag => "internal_network"
    add_field => {
      "network_category" => "internal"
    }
  }
}
Note: The cidr filter is useful for categorizing traffic based on IP ranges. It supports both IPv4 and IPv6 CIDR notation.

9. Using DNS Filter

Field Description Default Notes
add_field Adds a field to the event {} Map of field names to values
remove_field Removes a field from the event [] Array of field names to remove
source Field containing the IP address to resolve host Must be a valid IP address string
target Field to store the resolved DNS name dns.name Can be any valid field name
timeout Timeout in seconds for DNS resolution 5 Integer value in seconds

Example: Resolving DNS Names from IP Addresses

filter {
  dns {
    source => "client_ip"
    target => "dns_name"
    timeout => 10
  }
}
Note: The DNS filter requires the resolv gem to be installed in Logstash.
Warning: DNS lookups can be slow and may block the pipeline if not configured with appropriate timeouts.

10. Using Kv Filter

Option Values Notes
source message Field to parse for key-value pairs
value_split = Separator between keys and values
field_split Separator between key-value pairs
include_brackets false Include brackets in field names if present
remove_char "" Characters to remove from keys/values
allow_duplicate_values false Allow multiple values for the same key

Example: Parsing Log Message

filter {
  kv {
    source => "message"
    value_split => "="
    field_split => " "
  }
}

Example: Parsing URL Query String

filter {
  kv {
    source => "query_string"
    value_split => "="
    field_split => "&"
    remove_char => "[]"
  }
}

11. Using CSV Filter

Field Description Default Notes
source Field containing the CSV data to parse message Required field
separator Character used to separate fields in CSV , Can be any character or string
skip_empty_lines Whether to skip empty lines in CSV false Boolean value
skip_header Whether to skip the first line of CSV (header) false Boolean value
columns List of column names to use for parsing nil If not provided, uses first line as headers
convert Map of field names to type conversions {} Supports integer, float, boolean

Example: Basic CSV Parsing

filter {
  csv {
    source => "message"
    separator => ","
    skip_header => true
  }
}

Example: CSV with Type Conversion

filter {
  csv {
    source => "log_data"
    separator => ";"
    columns => ["timestamp", "user_id", "amount"]
    convert => {
      "amount" => "float"
      "user_id" => "integer"
    }
  }
}
Note: If columns is not specified, the first line of CSV data is used as column headers.

12. Using XML Filter

Property Description Default Notes
source Field containing the XML content to parse message Can be any field in the event
target Field to store the parsed XML data xml Creates a nested structure in the event
force_array Force all elements to be arrays false Useful for consistent handling of XML elements
xpath XPath expression to extract specific elements nil Allows for selective parsing of XML content

Example: Parsing XML Log Content

filter {
  xml {
    source => "message"
    target => "parsed_xml"
    force_array => false
  }
}

Example: Extracting Specific XML Elements

filter {
  xml {
    source => "xml_data"
    target => "extracted"
    xpath => ["/log/entry[@type='error']/message"]
  }
}
Note: The XML filter requires valid XML content in the source field. Invalid XML will cause parsing errors.

Was this article helpful?

Related Cheat Sheets