Skip to main content
Mapagam

Working with Logstash Inputs

1. Understanding Input Plugins

Plugin Type Description Common Use Cases Configuration Example
beats Collects data from Elastic Beats (e.g., Filebeat, Metricbeat) Log aggregation, metric collection 
input {
  beats {
    port => 5044
  }
}
tcp Receives logs over TCP connections Remote log forwarding, custom logging systems 
input {
  tcp {
    port => 5000
    codec => json
  }
}
udp Receives logs over UDP connections High-volume, low-latency logging (e.g., syslog) 
input {
  udp {
    port => 514
    codec => syslog
  }
}
file Reads logs from files on disk Local log file monitoring, legacy system integration 
input {
  file {
    path => "/var/log/myapp.log"
    start_position => "beginning"
  }
}
http Receives logs via HTTP requests Webhook-style log ingestion, API logging 
input {
  http {
    port => 8080
    codec => json
  }
}
stdin Reads logs from standard input Testing and debugging configurations 
input {
  stdin {
    codec => json
  }
}

Example: Using the Beats Input Plugin

input {
  beats {
    port => 5044
    host => "0.0.0.0"
    codec => json
  }
}

Example: Using the TCP Input Plugin

input {
  tcp {
    port => 5000
    codec => json_lines
    type => "application-logs"
  }
}
Note: Input plugins can be combined to receive logs from multiple sources simultaneously. Each input plugin supports various codecs for parsing different log formats.

2. Using File Input

Property Values Description
path /var/log/*.log Path to log files. Supports wildcards.
start_position beginning | end Where to start reading from. Default is end.
ignore_older 7d Ignore files older than this duration.
sincedb_path /var/lib/logstash/.sincedb Path to sincedb file for tracking file positions.
codec plain | json Codec to use for decoding log lines. Default is plain.

Example: Basic file input configuration

input {
  file {
    path => "/var/log/application.log"
    start_position => "beginning"
    codec => json
  }
}

Example: Using wildcards and ignore_older

input {
  file {
    path => "/var/log/*.log"
    ignore_older => "7d"
    start_position => "end"
  }
}
Note: The sincedb_path is critical for tracking file positions when Logstash restarts. If not set, it defaults to a hidden file in the Logstash directory.

3. Using Beats Input

Property Values Description
beats.port 5044 Port to listen for Beats input
beats.host 0.0.0.0 Host address to bind the input
beats.codec json Codec to decode incoming data (e.g., json, plain)
beats.idle_timeout 30s Timeout for idle connections

Example: Beats Input Configuration

{
  "input": {
    "beats": {
      "port": 5044,
      "host": "0.0.0.0",
      "codec": "json"
    }
  }
}
Note: Ensure that the Beats instance is configured to send data to this Logstash endpoint.

4. Using TCP Input

Property Values Description
port 5000 Port to listen for incoming TCP connections
host 0.0.0.0 Host address to bind the TCP listener
codec json Codec to decode incoming messages (default: plain)
type tcp Input type identifier for Logstash

Example: TCP Input Configuration

{
  "input": {
    "tcp": {
      "port": 5000,
      "host": "0.0.0.0",
      "codec": "json"
    }
  }
}
Note: Ensure the port is not blocked by a firewall and that the Logstash instance has sufficient permissions to bind to the specified host and port.

5. Using UDP Input

Property Value Description
udp input { udp { ... } } Defines a UDP input plugin for Logstash
port 514 Port to listen for UDP messages
host 0.0.0.0 Host address to bind to
codec plain Codec to use for decoding messages (e.g., json, plain)

Example: UDP Input Configuration

input {
  udp {
    port => 514
    host => "0.0.0.0"
    codec => "plain"
  }
}
Note: UDP is a connectionless protocol, so Logstash does not guarantee message delivery or order.

6. Using HTTP Input

Property Values Description
port 8080 Port to listen for HTTP requests
host localhost Host address to bind to
path /logstash Endpoint path for incoming logs
codec json Codec to decode incoming messages
ssl_enable false Enable HTTPS support

Example: Configuring HTTP Input

input {
  http {
    port => 8080
    host => "localhost"
    path => "/logstash"
    codec => json
  }
}

Example: Sending Log via HTTP

curl -X POST http://localhost:8080/logstash \
  -H "Content-Type: application/json" \
  -d '{"message": "Sample log entry", "level": "info"}'

7. Using Kafka Input

Property Value Description
kafka input Defines the Kafka input plugin configuration
bootstrap_servers localhost:9092 Comma-separated list of Kafka broker addresses
topic log-topic Kafka topic to consume messages from
group_id logstash-group Consumer group ID for Kafka
codec json Codec to use for decoding messages (e.g., json, plain)
client_id logstash-kafka-input Unique identifier for the Kafka consumer client

Example: Kafka Input Configuration

input {
  kafka {
    bootstrap_servers => "localhost:9092"
    topic => "log-topic"
    group_id => "logstash-group"
    codec => "json"
    client_id => "logstash-kafka-input"
  }
}
Note: Ensure that the Kafka brokers are accessible from the Logstash instance and that proper security settings (e.g., SASL/SSL) are configured if required.

8. Using JDBC Input

Property Values Description
jdbc.connection_string jdbc:postgresql://localhost:5432/mydb Connection string to the database
jdbc.driver_path /path/to/postgresql-42.6.0.jar Path to JDBC driver JAR file
jdbc.user myuser Database user for authentication
jdbc.password mypassword Password for database user
jdbc.query SELECT * FROM logs WHERE timestamp > :sql_last_value SQL query to fetch logs from database
jdbc.schedule * * * * * Cron-like schedule for polling (default every minute)
jdbc.last_run_metadata_path /var/lib/logstash/jdbc_last_run.json Path to store last run metadata for incremental queries

Example: JDBC Input Configuration

input {
  jdbc {
    driver_class => "org.postgresql.Driver"
    connection_string => "jdbc:postgresql://localhost:5432/mydb"
    user => "myuser"
    password => "mypassword"
    query => "SELECT * FROM logs WHERE timestamp > :sql_last_value"
    schedule => "* * * * *"
    last_run_metadata_path => "/var/lib/logstash/jdbc_last_run.json"
  }
}
Note: The :sql_last_value placeholder is automatically replaced by Logstash with the last timestamp value from previous runs to enable incremental fetching.
Warning: Ensure that the JDBC driver JAR is accessible to Logstash and compatible with your database version.

9. Using S3 Input

Property Value Description
type s3 Specifies the input plugin type as S3.
bucket my-log-bucket Name of the S3 bucket to read logs from.
region us-east-1 AWS region where the S3 bucket is located.
access_key_id AKIA... AWS access key ID for authentication.
secret_access_key ... AWS secret access key for authentication.
interval 30 How often (in seconds) to check for new logs in the bucket.
prefix logs/ Optional prefix to filter logs by path in the bucket.
codec json Codec to use for parsing log entries (e.g., json, plain).

Example: S3 Input Configuration

input {
  s3 {
    bucket => "my-log-bucket"
    region => "us-east-1"
    access_key_id => "AKIA..."
    secret_access_key => "..."
    interval => 30
    prefix => "logs/"
    codec => "json"
  }
}
Note: Ensure that the IAM role or credentials provided have appropriate permissions to read from the specified S3 bucket.

10. Using Redis Input

Property Values Description
type redis Specifies the input plugin type as Redis.
host localhost Hostname or IP address of the Redis server.
port 6379 Port number on which Redis is listening.
key logstash Redis key to use for storing logs.
data_type list Data structure used in Redis (e.g., list, channel).
batch_count 10 Number of messages to fetch in a single batch.
timeout 5 Timeout in seconds for Redis operations.

Example: Basic Redis Input Configuration

input {
  redis {
    host => "localhost"
    port => 6379
    key => "logstash"
    data_type => "list"
    batch_count => 10
    timeout => 5
  }
}
Note: Ensure Redis is running and accessible from the Logstash instance.

11. Using Stdin Input

Property Value Description
stdin {} Input plugin to read from standard input

Example: Stdin Input Configuration

input {
  stdin {
    codec => json
  }
}
Note: The stdin input plugin is useful for testing and debugging Logstash configurations in real-time.

12. Configuring Input Codec

Codec Description Usage
plain Reads input as plain text without any transformation. codec => "plain"
json Parses JSON formatted logs into structured fields. codec => "json"
grok Uses patterns to parse unstructured log data into structured fields. codec => "grok"
multiline Handles multi-line log entries by combining them into single events. codec => "multiline"
csv Parses CSV formatted logs into structured fields. codec => "csv"
json_lines Processes each line as a separate JSON object. codec => "json_lines"

Example: Using JSON Codec

input {
  file {
    path => "/var/log/application.log"
    codec => "json"
  }
}

Example: Using Grok Codec

input {
  file {
    path => "/var/log/nginx/access.log"
    codec => "grok"
    patterns_dir => ["/etc/logstash/patterns"]
    match => { message => "%{COMMON_LOG_FORMAT}" }
  }
}

Was this article helpful?

Related Cheat Sheets