Installing and Setting Up Logstash
1. Installing Logstash
| Platform | Installation Method | Notes |
|---|---|---|
| Linux (Debian/Ubuntu) | sudo apt-get update && sudo apt-get install logstash |
Requires Java 11 or higher |
| Linux (RHEL/CentOS) | sudo yum install logstash |
Requires Java 11 or higher |
| macOS | brew install logstash |
Requires Java 11 or higher |
| Windows | choco install logstash |
Requires Java 11 or higher |
| Docker | docker pull docker.elastic.co/logstash/logstash:8.15.3 |
Latest stable version as of 2026 |
Example: Running Logstash via Docker
docker run -it --rm \
-v /path/to/logstash.conf:/usr/share/logstash/pipeline/logstash.conf \
docker.elastic.co/logstash/logstash:8.15.3Note: Logstash requires Java 11 or higher to run. Ensure the
JAVA_HOME environment variable is set correctly.2. Understanding Pipeline Configuration
| Property | Description | Example |
|---|---|---|
| input | Defines the source of the data to be processed | {<br> "file": {<br> "path": "/var/log/app.log"<br> }<br>} |
| filter | Transforms and enriches the data | {<br> "grok": {<br> "match": {<br> "message": "%{TIMESTAMP}"<br> }<br> }<br>} |
| output | Defines where to send the processed data | {<br> "elasticsearch": {<br> "hosts": ["localhost:9200"]<br> }<br>} |
Example: Basic Pipeline Configuration
{<br> "pipeline": {<br> "id": "my-pipeline",<br> "input": {<br> "file": {<br> "path": "/var/log/app.log"<br> }<br> },<br> "filter": {<br> "grok": {<br> "match": {<br> "message": "%{TIMESTAMP}"<br> }<br> }<br> },<br> "output": {<br> "elasticsearch": {<br> "hosts": ["localhost:9200"]<br> }<br> }<br> }<br>}
Note: Pipeline configuration is defined in JSON format and must be valid JSON.
Warning: Incorrectly formatted pipeline configuration can cause Logstash to fail to start.
3. Creating Basic Pipeline
Example: Basic Logstash Configuration
# logstash.conf
input {
file {
path => "/var/log/myapp.log"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{MESSAGE:message}" }
}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "myapp-%{+YYYY.MM.dd}"
}
}
| Property | Description | Default |
|---|---|---|
| input | Defines the source of logs to process | None |
| filter | Transforms and enriches log data | None |
| output | Sends processed logs to destination (e.g., Elasticsearch) | None |
Note: The
grok filter is used to parse unstructured log data into structured fields.Warning: Ensure file paths exist and Logstash has read permissions.
Configuration Components
- input: Specifies log sources (e.g., files, TCP/UDP)
- filter: Processes logs using plugins (e.g., grok, date)
- output: Sends logs to destinations (e.g., Elasticsearch, stdout)
4. Configuring JVM Heap Size
| Property | Default Value | Description | Notes |
|---|---|---|---|
| jvm.heap.size | 1g |
Maximum heap size for the JVM | Recommended to set between 1GB and 8GB depending on data volume |
| jvm.heap.newsize | 128m |
Initial size of the young generation heap | Should be 1/4 to 1/2 of the total heap size |
| jvm.perm.size | 128m |
Size of the permanent generation space (Java 7 and earlier) | Not used in Java 8+; deprecated in newer versions |
Note: Ensure that the total heap size does not exceed available system memory to avoid performance degradation or out-of-memory errors.
5. Setting Up Path Settings
| Property | Default Value | Description |
|---|---|---|
| path.data | /var/lib/logstash |
Directory where Logstash stores internal data like pipeline state and configuration files. |
| path.logs | /var/log/logstash |
Directory where Logstash writes log files. |
| path.config | /etc/logstash/conf.d |
Directory where Logstash looks for configuration files (e.g., logstash.conf). |
| path.pipeline | /etc/logstash/pipeline |
Directory for pipeline configuration files (e.g., pipeline.conf). |
| path.plugins | /usr/share/logstash/plugins |
Directory for Logstash plugins (e.g., input, filter, output plugins). |
Example: Configuring Path Settings in logstash.yml
path:
data: /opt/logstash/data
logs: /opt/logstash/logs
config: /opt/logstash/configs
pipeline: /opt/logstash/pipelines
plugins: /opt/logstash/plugins6. Setting Up Environment Variables
Example: Logstash Environment Configuration
# Set environment variables for Logstash
export LS_JAVA_OPTS="-Xms1g -Xmx2g"
export LOGSTASH_HOME="/opt/logstash"
export LS_LOG_LEVEL="info"| Variable | Description | Default Value | Notes |
|---|---|---|---|
| LS_JAVA_OPTS | Java options for Logstash process | -Xms512m -Xmx1g |
Memory allocation settings |
| LOGSTASH_HOME | Installation directory of Logstash | /usr/share/logstash |
Used for configuration and plugin paths |
| LS_LOG_LEVEL | Log level for Logstash output | info |
Options: debug, info, warn, error |
| LS_SETTINGS_DIR | Directory for Logstash settings files | $LOGSTASH_HOME/config |
Overrides default config directory |
Note: Environment variables can be set in
/etc/default/logstash on Debian/Ubuntu systems or /etc/sysconfig/logstash on Red Hat/CentOS systems.
7. Starting and Stopping Services
Starting Logstash Service
- Ensure Logstash is installed and configured
- Start the service using system commands or scripts
| Command | Description | Notes |
|---|---|---|
| systemctl start logstash | Start Logstash as a systemd service | Requires root or sudo privileges |
| service logstash start | Start Logstash using the service command | Alternative for systems not using systemd |
| logstash -f /path/to/config.conf | Start Logstash manually with a config file | Useful for testing or development |
Example: Starting Logstash with systemd
sudo systemctl start logstash
sudo systemctl status logstashStopping Logstash Service
- Identify the running Logstash process
- Stop the service using system commands or scripts
| Command | Description | Notes |
|---|---|---|
| systemctl stop logstash | Stop Logstash systemd service | Gracefully shuts down the service |
| service log/logstash stop | Stop Logstash using the service command | Alternative for systems not using systemd |
| pkill -f logstash | Kill all Logstash processes by name | Forcefully terminates any running instances |
Note: When stopping Logstash, ensure all pipelines are fully flushed to avoid data loss.
8. Verifying Installation
| Check | Description | Expected Output |
|---|---|---|
| Version | Verify Logstash version | Logstash 8.12.0 |
| Service Status | Check if Logstash service is running | active (running) |
| Configuration Test | Test Logstash configuration file | Configuration OK |
Note: Ensure the Logstash user has appropriate permissions to read configuration files and access input/output sources.
Warning: If Logstash fails to start, check the logs in
/var/log/logstash/ for detailed error messages.9. Configuring Pipeline Workers
| Property | Default Value | Description |
|---|---|---|
| pipeline.workers | 2 |
Number of worker threads to use for processing events. |
| pipeline.ordered | false |
Whether to process events in order per pipeline. |
| pipeline.batch.size | 125 |
Number of events to batch together for processing. |
| pipeline.batch.delay | 50 |
Delay in milliseconds before processing a batch. |
Example: Setting Pipeline Workers
pipeline.workers: 4
pipeline.ordered: true
pipeline.batch.size: 250
pipeline.batch.delay: 100
Note: Increasing
pipeline.workers can improve throughput, but too many workers may lead to resource contention.
10. Setting Up Multiple Pipelines
Logstash supports running multiple pipelines to process different types of logs or apply distinct transformations. Each pipeline operates independently and can have its own configuration file.
| Pipeline ID | Configuration File | Description |
|---|---|---|
| main | pipelines.yml |
Default pipeline used if no specific pipeline is specified. |
| audit | audit-pipeline.conf |
Dedicated pipeline for audit logs. |
| web | web-pipeline.conf |
Handles web server logs. |
Example: Configuring Multiple Pipelines
# pipelines.yml
- pipeline.id: main
path.config: "/etc/logstash/conf.d/main.conf"
- pipeline.id: audit
path.config: "/etc/logstash/conf.d/audit.conf"
- pipeline.id: web
path.config: "/etc/logstash/conf.d/web.conf"
Example: Pipeline Configuration Files
// main.conf
input {
beats {
port => 5044
}
}
filter {
grok {
match => { "message" => "%{COMBINED_LOG_FORMAT}" }
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "main-%{+YYYY.MM.dd}"
}
}
Note: Ensure each pipeline has a unique ID and configuration file path to avoid conflicts.
Warning: Running multiple pipelines increases resource usage. Monitor CPU and memory consumption carefully.
11. Testing Configuration
Example: Test Logstash Configuration
bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/my-config.conf| Option | Description | Default |
|---|---|---|
| --config.test_and_exit | Tests the configuration file and exits | false |
| -f <path> | Path to the configuration file | none |
Note: The command will output any syntax errors or warnings found in the configuration file.
Warning: Ensure Logstash is installed and the configuration file path is correct before running the test.