Managing Logstash Plugins
1. Understanding Plugin Types
| Type | Description | Examples | Usage |
|---|---|---|---|
| input | Source of data into the Logstash pipeline | beats, file, tcp |
First stage in the pipeline; reads data from sources like files, network streams, or Beats |
| filter | Processes and transforms data within the pipeline | grok, mutate, date |
Modifies or enriches events; can parse, add, remove, or change fields |
| output | Destination of processed data in the Logstash pipeline | elasticsearch, file, http |
Writes data to external systems like Elasticsearch, files, or HTTP endpoints |
| codec | Encoding/decoding format for input and output data | json, plain, multiline |
Handles data format conversion; used to parse or serialize events |
Example: Plugin Configuration
input {
beats {
port => 5044
}
}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}" }
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "logs-%{+YYYY.MM.dd}"
}
}2. Listing Installed Plugins
| Command | Description | Notes |
|---|---|---|
| logstash-plugin list | List all installed Logstash plugins | Shows plugin name, version, and installation status |
| logstash-plugin list --verbose | List installed plugins with detailed information | Includes plugin type, dependencies, and metadata |
| logstash-plugin list --offline | List plugins without connecting to remote repositories | Useful for systems with restricted network access |
3. Installing Plugins
| Command | Description | Notes |
|---|---|---|
| logstash-plugin install <plugin-name> | Installs a Logstash plugin from the default repository | Use this for official plugins |
| logstash-plugin install --version <version> <plugin-name> | Installs a specific version of a plugin | Use to ensure compatibility or rollback |
| logstash-plugin install --offline <plugin-path> | Installs a plugin from a local file | Use for environments without internet access |
| logstash-plugin install --file <plugin-gem> | Installs a plugin from a local .gem file | Use for custom or pre-downloaded plugins |
Example: Install a specific version of a plugin
logstash-plugin install --version 4.0.0 logstash-input-http
Note: Ensure that the plugin is compatible with your Logstash version. Incompatible plugins may cause errors or instability.
4. Updating Plugins
| Command | Description | Notes |
|---|---|---|
| logstash-plugin update | Updates all installed plugins to their latest versions | Runs logstash-plugin update for each plugin |
| logstash-plugin update <plugin_name> | Updates a specific plugin to the latest version | Useful for updating individual plugins without affecting others |
Note: Ensure Logstash is not running when updating plugins to avoid conflicts.
Warning: Updating plugins may introduce breaking changes. Always test in a staging environment before updating production.
5. Removing Plugins
| Command | Description | Notes |
|---|---|---|
| logstash-plugin remove <plugin_name> | Removes a specific Logstash plugin | Replace <plugin_name> with the actual plugin name |
Note: Ensure the plugin is not in use by any pipeline before removal to avoid runtime errors.
6. Creating Custom Plugins
Creating custom Logstash plugins allows you to extend the functionality of Logstash for specific use cases. Plugins can be developed in Ruby or Java.
| Plugin Type | Description | Development Language | Use Case |
|---|---|---|---|
| Input Plugin | Reads data from sources like files, network streams, or databases. | Ruby | Collecting logs from various sources |
| Filter Plugin | Processes and transforms events in the Logstash pipeline. | Ruby | Modifying log fields or enriching data |
| Output Plugin | Sends processed events to destinations like Elasticsearch, files, or databases. | Ruby | Storing logs in Elasticsearch or other systems |
| Codec Plugin | Handles encoding and decoding of events. | Ruby | Processing logs in different formats (e.g., JSON, syslog) |
Example: Creating a Simple Filter Plugin in Ruby
module LogStash
module Filters
class CustomFilter < LogStash::Filters::Base
config_name "custom_filter"
def register
@logger.info("Custom filter registered")
end
def filter(event)
# Example: Add a custom field to the event
event.set("custom_field", "custom_value")
filter_matched(event)
end
end
end
end
Note: Custom plugins must be placed in the
logstash/plugins directory or installed via the LogStash plugin manager.
Warning: Ensure that custom plugins are tested thoroughly in a development environment before deploying to production.
Steps to Create a Custom Plugin
- Create the plugin directory structure under
logstash/plugins. - Implement the plugin logic in Ruby or Java.
- Register the plugin in LogStash's plugin registry.
- Test the plugin using sample data.
- Deploy to production after validation.
7. Using Plugin Mixins
Plugin mixins in Logstash allow you to share common functionality across multiple plugins. They are a way to reduce code duplication and improve maintainability.
| Mixin | Description | Usage |
|---|---|---|
| logstash::plugin | Base mixin for all Logstash plugins | Include in plugin definition to inherit core plugin behavior |
| logstash::codec | Mixin for codec plugins | Used to define codecs that transform data |
| logstash::filter | Mixin for filter plugins | Used to define filters that process events |
| logstash::input | Mixin for input plugins | Used to define inputs that receive data |
| logstash::output | Mixin for output plugins | Used to define outputs that send data elsewhere |
Example: Using Mixins in a Custom Plugin
require "logstash"
require "logstash/plugin"
class LogStash::Plugins::MyCustomFilter < LogStash::Plugin
include LogStash::Plugin::Filter
config :my_param, :validate => :string
def register
# Initialization logic here
end
def filter(event)
# Filter logic here
end
end
Note: Mixins provide a standardized way to define plugin behavior and ensure compatibility with Logstash's core systems.
8. Configuring Plugin Settings
| Setting | Description | Example Value |
|---|---|---|
| plugin.config | Configuration options for the plugin | { "some_param": "value" } |
| plugin.path | Path to plugin directory | /usr/share/logstash/plugins |
Example: Configuring Plugin Settings
{
"input": {
"plugin": "file",
"path": "/var/log/application.log"
},
"filter": {
"plugin": "grok",
"pattern": "%{TIMESTAMP}"
},
"output": {
"plugin": "elasticsearch",
"hosts": ["localhost:9200"]
}
}9. Testing Custom Plugins
Example: Basic Plugin Test Structure
# spec/logstash/filter/myfilter_spec.rb
require 'logstash/devutils/rspec'
require 'logstash/filter/myfilter'
describe LogStash::Filter::MyFilter do
it 'should work' do
config = {
'field' => 'message',
'pattern' => '%{TIMESTAMP} %{LEVEL} %{MESSAGE}'
}
event = LogStash::Event.new('message' => '2023-01-01 12:00:00 INFO Test message')
filter = LogStash::Filter::MyFilter.new(config)
filter.filter(event)
expect(event.get('level')).to eq('INFO')
end
end
| Test Type | Description | Command |
|---|---|---|
| Unit Tests | Test individual plugin logic without external dependencies | rake test |
| Integration Tests | Test plugin behavior with actual Logstash pipeline | rake integration |
Note: Use
LogStash::DevUtils::RSpec for consistent plugin testing patterns.
Warning: Custom plugins must pass all tests before deployment to production.
Testing Best Practices
- Write tests for all plugin configurations
- Test edge cases and error conditions
- Use fixtures for realistic input data
- Validate output format matches expectations
10. Packaging Plugins
| Field | Description | Example |
|---|---|---|
| name | Plugin name | "logstash-filter-json" |
| version | Plugin version | "3.1.0" |
| license | License type | "Apache-2.0" |
| description | Plugin description | "Parses JSON logs" |
| author | Plugin author | "Elastic" |
| gem_name | Gem name for plugin | "logstash-filter-json" |
| gem_version | Gem version | "3.1.0" |
| logstash_version | Compatible Logstash version | "7.10.0" |
| logstash_plugin_type | Plugin type (filter, input, output) | "filter" |
Example: Plugin Gemspec
Gem::Specification.new do |s|
s.name = "logstash-filter-json"
s.version = "3.1.0"
s.license = "Apache-2.0"
s.description = "Parses JSON logs"
s.author = "Elastic"
s.summary = "JSON filter for Logstash"
s.homepage = "https://github.com/elastic/logstash-filter-json"
s.require_paths = ["lib"]
s.files = Dir["lib/**/*", "spec/**/*", "Gemfile", "Rakefile", "README.md", "LICENSE"]
s.test_files = s.files.grep(%r{^spec/})
s.add_runtime_dependency "logstash-core", ">= 7.10.0"
s.add_development_dependency "logstash-devutils", "~> 2.0"
endExample: Plugin Manifest
{
"name": "logstash-filter-json",
"version": "3.1.0",
"license": "Apache-2.0",
"description": "Parses JSON logs",
"author": "Elastic",
"gem_name": "logstash-filter-json",
"gem_version": "3.1.0",
"logstash_version": "7.10.0",
"logstash_plugin_type": "filter"
}
Note: Ensure all dependencies are compatible with the target Logstash version.
Warning: Avoid including sensitive data in plugin manifests or gemspecs.
11. Publishing Plugins
To publish a Logstash plugin, you must first ensure it meets the requirements for distribution within the Elastic ecosystem.
| Requirement | Description | Status |
|---|---|---|
| Plugin Structure | Follows the standard Logstash plugin directory structure with a logstash folder and proper metadata files. |
NEW |
| Documentation | Includes a README.md with usage examples and configuration details. | NEW |
| Testing | Comprehensive tests using the Logstash testing framework. | NEW |
Example: Plugin Metadata File
{
"name": "logstash-filter-example",
"version": "1.0.0",
"description": "An example filter plugin for Logstash",
"author": "Your Name",
"email": "your.email@example.com",
"license": "Apache-2.0"
}
Note: Ensure that all dependencies are declared in the
Gemfile and that the plugin is compatible with the latest stable version of Logstash.