Working with Logstash Codecs

1. Understanding Codec Purpose

Codec	Description	Use Case	Example
json	Parses JSON messages into structured data fields	Log parsing from applications using JSON logging	`{ "timestamp": "2024-01-01T12:00:00Z", "level": "INFO", "message": "User logged in" }`
grok	Matches log lines using patterns defined in a Grok pattern library	Custom parsing for unstructured logs like Apache/Nginx access logs	`%{TIMESTAMP_ISO8601:timestamp} %{IP:client_ip} %{WORD:method} %{URIPATH:uri}`
csv	Parses CSV formatted logs into field-value pairs	Log files with comma-separated values	`2024-01-01,INFO,User login successful`
multiline	Combines multi-line log entries into single events	Stack traces or multi-line error messages	`ERROR: Exception occurred at com.example.Main.main(Main.java:10) at com.example.Helper.process(Helper.java:25)`
plain	Handles plain text logs without structure	Simple text logs or when no parsing is needed	`2024-01-01 12:00:00 INFO User logged in`

Note: Codecs are applied in the order they appear in the Logstash configuration file.

2. Using JSON Codec

Property	Values	Description
codec	`json`	Specifies the codec to use for parsing JSON logs.
skip_on_invalid_json	`true` \| `false`	If `true`, skips events with invalid JSON. If `false`, raises an error.
message_key	`message`	Field name to use for the raw log message.
add_field	Object	Adds fields to the parsed event.

Example: Basic JSON Codec Configuration

input {
  file {
    path => "/var/log/app.log"
    codec => json
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "app-logs-%{+YYYY.MM.dd}"
  }
}

Example: JSON Codec with Custom Fields

input {
  file {
    path => "/var/log/app.log"
    codec => json {
      add_field => { "source" => "application" }
      skip_on_invalid_json => true
    }
  }
}

Note: The JSON codec expects each log line to be a valid JSON object. If the input contains non-JSON lines, they will cause parsing errors unless skip_on_invalid_json is set to true.

3. Using Plain Codec

Property	Value	Description
codec	`plain`	Specifies the codec to use for decoding log messages.
charset	`UTF-8`	Character encoding to use when reading the input.

Example: Configuring Plain Codec

{<br>  "input": {<br>    "stdin": {<br>      "codec": "plain"<br>    }<br>  },<br>  "output": {<br>    "stdout": {<br>      "codec": "plain"<br>    }<br>  }<br>}

Note: The plain codec treats each line as a separate log entry.

4. Using Line Codec

Property	Value	Description
codec	`line`	Specifies the codec to use for parsing lines of text.
charset	`UTF-8`	Character encoding to use when reading lines. Defaults to UTF-8.

Example: Configuring Line Codec

input {
  file {
    path => "/var/log/application.log"
    codec => line {
      charset => "UTF-8"
    }
  }
}

Note: The line codec treats each line as a separate event. It is commonly used for log files where each line represents a single log entry.

5. Using Multiline Codec

Property	Values	Description
multiline	`codec`	Codec to use for multiline messages
pattern	`string`	Regular expression to identify start of new log line
negate	`true` \| `false`	If true, matches are treated as continuation lines
what	`previous` \| `next`	Determines whether to merge with previous or next line

Example: Configuring Multiline Codec

input {
  file {
    path => "/var/log/app.log"
    codec => multiline {
      pattern => "^\d{4}-\d{2}-\d{2}"
      negate => true
      what => "previous"
    }
  }
}

Note: The multiline codec is useful for handling logs that span multiple lines, such as stack traces or multi-line error messages.

6. Using Rubydebug Codec

Property	Value	Description
codec	`rubydebug`	Codec to use for debugging log entries in Ruby format

Example: Configuring Rubydebug Codec

input {
  stdin {
    codec => rubydebug
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Note: The rubydebug codec is primarily used for debugging and development purposes to inspect the structure of log events as they pass through Logstash.

7. Using CEF Codec

Property	Values	Description
codec	`cef`	Specifies the CEF codec for parsing Common Event Format logs.

Example: CEF Codec Configuration

{
  "input": {
    "type": "file",
    "path": "/var/log/cef.log"
  },
  "codec": {
    "type": "cef"
  },
  "output": {
    "type": "stdout"
  }
}

Note: The CEF codec is designed for logs that follow the Common Event Format (CEF) standard, typically used in security information and event management (SIEM) systems.

8. Using Msgpack Codec

Property	Values	Description
codec	`msgpack`	Specifies the Msgpack codec for encoding/decoding log events
enable	`true` / `false`	Enables or disables the Msgpack codec

Example: Configuring Msgpack Codec

{
  "input": {
    "type": "tcp",
    "port": 5000,
    "codec": "msgpack"
  },
  "output": {
    "type": "elasticsearch",
    "hosts": ["localhost:9200"],
    "codec": "msgpack"
  }
}

Note: Msgpack is a binary serialization format that is more efficient than JSON for log data transmission.

9. Using Avro Codec

Property	Value	Description
codec	`avro`	Sets the codec to Avro for decoding log messages.
schema_registry_url	`http://localhost:8081`	URL of the Avro schema registry to fetch schemas.
schema_id	`1`	Schema ID to use for decoding messages.

Example: Configuring Avro Codec in Logstash

input {
  kafka {
    codec => avro {
      schema_registry_url => "http://localhost:8081"
      schema_id => 1
    }
  }
}

Note: Ensure that the Avro schema registry is running and accessible to Logstash for schema resolution.

10. Using Netflow Codec

Property	Value	Description
codec	`netflow`	Specifies the Netflow codec for parsing network flow data.

Example: Configuring Netflow Codec

{<br>  "input": {<br>    "type": "udp",<br>    "port": 9995,<br>    "codec": "netflow"<br>  }<br>}

Note: Netflow codec supports both NetFlow v5 and v9 formats. Ensure your Logstash instance is configured to handle the appropriate version of NetFlow data.

11. Using Protobuf Codec

Property	Value	Description
codec	`protobuf`	Specifies the use of the Protobuf codec for decoding log messages.
path	`/path/to/your/proto/file.proto`	Path to the .proto file that defines the message structure.
message_type	`com.example.MessageType`	Fully qualified name of the Protobuf message type to decode.
include_paths	`[/path/to/imports]`	Array of paths to search for imported .proto files.

Example: Configuring Protobuf Codec in Logstash

input {
  tcp {
    port => 5000
    codec => protobuf {
      path => "/etc/logstash/protobuf/example.proto"
      message_type => "com.example.LogEntry"
      include_paths => ["/etc/logstash/protobuf"]
    }
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Note: Ensure that the Protobuf schema is compatible with the data being sent to Logstash. The message_type must exactly match a defined message in the provided .proto file.

12. Configuring Codec Parameters

Parameter	Description	Default Value	Notes
codec	Specifies the codec to use for decoding messages	`plain`	Must be a valid codec name (e.g., `json`, `grok`)
charset	Character encoding for input data	`UTF-8`	Used when decoding byte streams to text
skip_on_error	Whether to skip events that fail to decode	`false`	Set to `true` to continue processing on decode errors
add_field	Adds a field to the decoded event	`nil`	Can be used to enrich decoded events with static or dynamic values

Example: Configuring JSON Codec with Parameters

input {
  file {
    path => "/var/log/app.log"
    codec => json {
      charset => "UTF-8"
      skip_on_error => true
    }
  }
}

Example: Using Grok Codec with Custom Fields

input {
  file {
    path => "/var/log/nginx.log"
    codec => grok {
      match => { "message" => "%{IP:client_ip} %{WORD:method} %{URIPATH:url}" }
      add_field => { "source" => "nginx" }
    }
  }
}