Configuring Logstash Outputs

1. Understanding Output Plugins

Plugin	Description	Use Case	Notes
elasticsearch	Sends logs to Elasticsearch for storage and indexing	Centralized logging, search, analytics	Supports bulk indexing, dynamic index templates
file	Writes logs to local or remote files	Log archiving, debugging, data export	Supports JSON and plain text formats
stdout	Prints logs to standard output (console)	Development, testing, debugging	Useful for real-time log inspection
tcp/udp	Sends logs over TCP or UDP network protocols	Log forwarding to remote systems	Can be used with other Logstash instances or external tools
http	Sends logs via HTTP/HTTPS requests to a web server	Integration with REST APIs, custom endpoints	Supports authentication and custom headers

Example: Elasticsearch Output Plugin Configuration

output {
  elasticsearch {
    hosts => [ "localhost:9200" ]
    index => "logstash-%{+YYYY.MM.dd}"
    document_id => "%{&#123;message}"
  }
}

Note: Output plugins are responsible for sending processed logs to their final destination. Each plugin supports specific configuration options and can be customized based on the target system.

2. Using Elasticsearch Output

Property	Description	Default	Notes
hosts	List of Elasticsearch nodes to send events to	`["localhost:9200"]`	Can include multiple hosts for redundancy
index	Name of the index to write events to	`logstash-%{+YYYY.MM.dd}`	Supports dynamic date patterns
user	Username for authentication	`null`	Required if authentication is enabled
password	Password for authentication	`null`	Required if authentication is enabled
ssl	Enable SSL/TLS encryption	`false`	Set to `true` for secure connections
timeout	Timeout in seconds for requests to Elasticsearch	`30`	Adjust based on network conditions

Example: Basic Elasticsearch Output Configuration

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
  }
}

Example: Elasticsearch Output with Authentication

output {
  elasticsearch {
    hosts => ["https://elasticsearch.example.com:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "changeme"
    ssl => true
  }
}

Note: Ensure that the Elasticsearch cluster is accessible from the Logstash instance and that proper permissions are configured for the user if authentication is used.

3. Using File Output

Property	Values	Description
path	`/var/log/myapp.log`	File path to write logs to. Supports dynamic variables like `%{+yyyy-MM-dd}`.
codec	`json`	Codec to use for encoding log entries. Default is `line`.
flush_interval	`1`	Seconds between flushes to disk. Default is `1`.
template	`false`	Whether to use a template for the file name. Default is `false`.

Example: Writing Logs to a File

output {
  file {
    path => "/var/log/myapp.log"
    codec => json
  }
}

Note: Ensure the Logstash user has write permissions to the target directory.

4. Using Stdout Output

Property	Value	Description
stdout	`{}`	Configures Logstash to output logs to standard output (stdout)

Example: Stdout Output Configuration

output {
  stdout {
    codec => json_lines
  }
}

Note: The stdout output is primarily used for debugging and testing purposes. It outputs logs directly to the console or terminal where Logstash is running.

5. Using Kafka Output

Property	Description	Default	Notes
kafka	Configures the Kafka output plugin	n/a	Required
bootstrap_servers	Comma-separated list of Kafka broker addresses	`localhost:9092`	Must be reachable from Logstash
topic	Kafka topic to send messages to	`logstash`	Must exist or be auto-created
client_id	Unique identifier for the Kafka client	`logstash-output`	Used for monitoring and logging
security_protocol	Security protocol to use (e.g., PLAINTEXT, SSL)	`PLAINTEXT`	Must match Kafka server settings
ssl_truststore_location	Path to the truststore file (if SSL enabled)	n/a	Required if `security_protocol` is SSL
ssl_truststore_password	Password for the truststore (if SSL enabled)	n/a	Required if `ssl_truststore_location` is set
ssl_keystore_location	Path to the keystore file (if SSL enabled)	n/a	Required if `security_protocol` is SSL
ssl_keystore_password	Password for the keystore (if SSL enabled)	n/a	Required if `ssl_keystore_location` is set
compression_type	Compression type for messages (none, gzip, snappy, lz4)	`none`	Improves throughput at cost of CPU
batch_size	Number of messages to batch before sending	`100`	Higher values increase throughput but may delay delivery
linger_ms	Time in milliseconds to wait before sending a batch	`0`	Useful for batching with small delays
max_request_size	Maximum size of a request in bytes	`1048576`	Must not exceed Kafka broker limits

Example: Basic Kafka Output Configuration

output {
  kafka {
    bootstrap_servers => "localhost:9092"
    topic => "logstash-logs"
    client_id => "logstash-output"
  }
}

Example: Kafka Output with SSL

output {
  kafka {
    bootstrap_servers => "kafka.example.com:9093"
    topic => "logstash-logs"
    security_protocol => "SSL"
    ssl_truststore_location => "/path/to/truststore.jks"
    ssl_truststore_password => "changeit"
    ssl_keystore_location => "/path/to/keystore.jks"
    ssl_keystore_password => "changeit"
  }
}

Note: Ensure that the Kafka brokers are configured to accept messages from Logstash and that appropriate security settings (e.g., SASL/SSL) are in place if required.

6. Using HTTP Output

Property	Description	Default	Notes
url	The HTTP endpoint to send logs to	None	Required field
http_method	HTTP method to use (POST, PUT)	POST
headers	Additional headers to include in the request	{}
format	Format of the payload (json, json_lines)	json
timeout	Timeout in seconds for the HTTP request	30

Example: Basic HTTP Output Configuration

output {
  http {
    url ="http://my-log-server:8080/logs"
    http_method ="POST"
    format ="json_lines"
    timeout ="60"
  }
}

Example: HTTP Output with Custom Headers

output {
  http {
    url ="https://my-secure-server.com/log"
    headers => {
      "Authorization" => "Bearer token123"
      "Content-Type" => "application/json"
    }
    format ="json"
  }
}

7. Using S3 Output

Property	Values	Description
bucket	`string`	Name of the S3 bucket to store logs in.
region	`string`	AWS region where the bucket is located (e.g., `us-east-1`).
access_key_id	`string`	AWS access key ID for authentication.
secret_access_key	`string`	AWS secret access key for authentication.
endpoint	`string`	Custom endpoint URL for S3-compatible services (optional).
file	`string`	File path prefix for logs (e.g., `/logs/logstash-`).
size	`integer`	Maximum size (in MB) of each log file before rotation.
interval	`integer`	Interval (in seconds) to flush logs to S3.

Example: S3 Output Configuration

{<br>  "output": {<br>    "s3": {<br>      "bucket": "my-log-bucket",<br>      "region": "us-west-2",<br>      "access_key_id": "AKIAIOSFODNN7EXAMPLE",<br>      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",<br>      "file": "/logs/logstash-",<br>      "size": 50,<br>      "interval": 30<br>    }<br>  }<br>}

Note: Ensure that the IAM role or user associated with access_key_id has sufficient permissions to write to the specified S3 bucket.

8. Using Email Output

Property	Values	Description
email	`string`	Email address to send logs to
from	`string`	Sender email address
subject	`string`	Subject line for the email
body	`string`	Body content of the email
smtp_host	`string`	SMTP server host address
smtp_port	`number`	SMTP server port (e.g., 587)
authentication	`string`	Authentication method (e.g., `plain`, `login`)
username	`string`	SMTP username for authentication
password	`string`	SMTP password for authentication

Example: Email Output Configuration

{
  "email": {
    "to": "admin@example.com",
    "from": "logstash@example.com",
    "subject": "Log Alert - {{event.timestamp}}",
    "body": "Log entry: {{event.message}}",
    "smtp_host": "smtp.example.com",
    "smtp_port": 587,
    "authentication": "plain",
    "username": "user@example.com",
    "password": "secret123"
  }
}

Note: Ensure that the SMTP server allows sending emails from Logstash and that credentials are securely managed (e.g., via environment variables or secure configuration files).

9. Using Redis Output

Property	Values	Description
redis.host	`localhost`	Hostname or IP address of the Redis server
redis.port	`6379`	Port number of the Redis server
redis.key	`logstash`	Redis key to use for storing logs
redis.data_type	`list`	Data type to use in Redis (list, channel, set)
redis.password	`none`	Password for Redis authentication (if required)
redis.db	`0`	Database number to use in Redis
redis.timeout	`5`	Timeout in seconds for Redis operations

Example: Redis Output Configuration

output {
  redis {
    host => "localhost"
    port => 6379
    key => "logstash"
    data_type => "list"
    db => 0
  }
}

Note: Ensure that the Redis server is running and accessible from the Logstash instance.

10. Using TCP Output

Property	Values	Description
tcp	`output`	Defines the TCP output plugin configuration block.
host	`string`	The IP address or hostname of the TCP server to connect to.
port	`number`	The port number on the TCP server to connect to.
codec	`string`	The codec used to encode the event before sending over TCP. Default is `json`.
reconnect_interval	`number`	Number of seconds to wait before attempting to reconnect if the connection is lost. Default is `1`.

Example: Basic TCP Output Configuration

output {
  tcp {
    host => "192.168.1.100"
    port => 5000
    codec => json
  }
}

11. Using Multiple Outputs

Example: Multiple outputs configuration

{
  "pipeline": {
    "id": "main-pipeline",
    "inputs": [
      {
        "type": "file",
        "path": "/var/log/app.log"
      }
    ],
    "filters": [
      {
        "type": "grok",
        "match": {
          "message": "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"
        }
      }
    ],
    "outputs": [
      {
        "type": "elasticsearch",
        "hosts": ["localhost:9200"],
        "index": "app-logs-%{+YYYY.MM.dd}"
      },
      {
        "type": "file",
        "path": "/var/log/logstash/app-backup.log"
      }
    ]
  }
}

Property	Description	Default
type	Output plugin type (e.g., `elasticsearch`, `file`)	Required
hosts	Array of Elasticsearch hosts to send data to	Required for `elasticsearch`
index	Index name pattern to use in Elasticsearch	`logstash-%{+YYYY.MM.dd}`
path	File path to write logs to	Required for `file`

Note: When using multiple outputs, each output is processed in parallel. Ensure that your system can handle the load from multiple concurrent output operations.

12. Setting Output Workers

Property	Values	Description
output.workers	`1`	Number of output worker threads to use. Default is 1.
output.queue.size	`1024`	Size of the internal queue for output workers. Default is 1024.
output.flush_interval	`1`	Interval (in seconds) to flush output. Default is 1 second.

Example: Configuring Output Workers

output {
  elasticsearch {
    hosts => [ "localhost:9200" ]
    workers => 3
    queue_size => 2048
    flush_interval => 5
  }
}

Note: Increasing workers can improve throughput but may increase memory usage.