Working with Beats

1. Understanding Beats Family

Beat	Description	Primary Use Case	Language/Platform
Filebeat	Collects and forwards log files from servers.	Log collection and forwarding	Go
Metricbeat	Collects metrics from services and systems.	System and service monitoring	Go
Packetbeat	Captures and analyzes network traffic.	Network monitoring and analysis	Go
Auditbeat	Collects and analyzes audit logs from the Linux kernel.	Security event monitoring	Go
Heartbeat	Monitors service availability and network connectivity.	Service health checks	Go
Winlogbeat	Collects Windows event logs.	Windows log collection	Go
Containerbeat	Collects logs and metrics from containerized environments.	Container monitoring	Go

Example: Filebeat Configuration

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log

output.elasticsearch:
  hosts: ["localhost:9200"]

Note: Beats are lightweight data shippers that run on the systems where your data is generated.

2. Installing Filebeat

Platform	Installation Method	Notes
Ubuntu/Debian	`sudo apt update && sudo apt install filebeat`	Requires Elasticsearch and Kibana to be installed first
CentOS/RHEL	`sudo yum install filebeat`	Use `yum` or `dnf` depending on version
macOS	`brew install filebeat`	Available via Homebrew
Windows	`choco install filebeat`	Requires Chocolatey to be installed

Example: Verify Installation

filebeat version

Note: Ensure that the Filebeat version matches your Elasticsearch and Kibana versions for compatibility.

3. Configuring Filebeat Inputs

Input Type	Configuration	Description
file	`{ "file": { "paths": ["/var/log/*.log"], "exclude_files": [".gz$"] } }`	Reads log files from the filesystem
stdin	`{ "stdin": {} }`	Reads from standard input
syslog	`{ "syslog": { "path": "/var/run/syslog", "port": 514 } }`	Reads from syslog socket or file

Example: Filebeat input configuration

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
  exclude_files:
    - '\.gz

4. Using Filebeat Modules

Module	Description	Default Fields	Enabled by Default
apache2	Collects and parses Apache 2 access and error logs	`apache2.access.log.message`, `apache2.error.log.message`	Yes
nginx	Collects and parses Nginx access and error logs	`nginx.access.log.message`, `nginx.error.log.message`	Yes
mysql	Collects MySQL error logs and slow query logs	`mysql.error.log.message`, `mysql.slow.log.message`	No
postgresql	Collects PostgreSQL logs	`postgresql.log.message`	No
system	Collects system logs including logs from systemd, syslog, and Windows event logs	`system.log.message`	Yes

Example: Enabling the Apache2 Module

filebeat modules enable apache2

Example: Configuring the Nginx Module

filebeat.modules:
- module: nginx
  var.paths: ["/var/log/nginx/access.log"]

Note: Filebeat modules are designed to simplify log collection by providing pre-built configurations for common use cases. They can be enabled or disabled using the filebeat modules command.

Warning: When using Filebeat modules, ensure that the log paths specified in the configuration match the actual location of your logs. Misconfigured paths can lead to missing or incorrect log data.

5. Installing Metricbeat

Platform	Installation Method	Notes
Linux (Debian/Ubuntu)	`curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.12.0-amd64.deb sudo dpkg -i metricbeat-8.12.0-amd64.deb`	Use `apt` for package management after installation
Linux (RHEL/CentOS)	`curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.12.0-x86_64.rpm sudo rpm -vi metricbeat-8.12.0-x86_64.rpm`	Use `yum` or `dnf` for package management after installation
macOS	`curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.12.0-darwin-x86_64.tar.gz tar xzvf metricbeat-8.12.0-darwin-x86_64.tar.gz cd metricbeat-8.12.0-darwin-x86_64 sudo ./metricbeat install`	Install using `brew` for easier management
Windows	`curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.12.0-windows-x86_64.zip Expand-Archive .\metricbeat-8.12.0-windows-x86_64.zip -DestinationPath . cd metricbeat-8.12.0-windows-x86_64 .\metricbeat.exe install`	Run as Administrator for proper installation

Example: Basic Metricbeat Configuration

metricbeat.modules:
- module: system
  metricsets:
    - cpu
    - memory
  period: 10s

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "metricbeat-%{[agent.version]}-%{+yyyy.MM.dd}"

Note: Ensure Elasticsearch and Kibana are running before starting Metricbeat.

6. Configuring Metricbeat Modules

Module	Enabled by Default	Description	Notes
apache	`false`	Collects metrics from Apache HTTPD servers	Requires `apache.status.url` to be configured
elasticsearch	`false`	Collects metrics from Elasticsearch nodes and cluster	Supports both basic and SSL authentication
mysql	`false`	Collects metrics from MySQL database servers	Requires `mysql.host` and `mysql.port`
nginx	`false`	Collects metrics from Nginx servers	Requires `nginx.status.url` to be set
system	`true`	Collects system-level metrics (CPU, memory, disk)	Includes `system.cpu`, `system.memory`, and `system.diskio`
kafka	`false`	Collects metrics from Apache Kafka brokers	Requires `kafka.broker.host` and `kafka.broker.port`

Example: Enabling and Configuring the Elasticsearch Module

metricbeat.modules:
- module: elasticsearch
  enabled: true
  period: 10s
  hosts: ["http://localhost:9200"]
  username: "metricbeat_internal"
  password: "${ELASTICSEARCH_PASSWORD}"

Example: Enabling the System Module

metricbeat.modules:
- module: system
  enabled: true
  period: 10s
  metricsets:
    - cpu
    - memory
    - diskio

Note: The period field controls how often metrics are collected. It should be set to a reasonable value based on the desired granularity and system load.

7. Using Packetbeat

Packetbeat is a lightweight network packet analyzer that collects and analyzes network traffic data. It is part of the Elastic Stack (ELK) and helps in monitoring and analyzing network activity.

Property	Description	Default Value
packetbeat.procs	Defines the list of processes to monitor for network traffic.	`[]`
packetbeat.interfaces	Specifies the network interfaces to capture packets from.	`["any"]`
packetbeat.ports	Defines the ports to capture traffic from.	`["any"]`

Example: Basic Packetbeat Configuration

packetbeat:
  interfaces:
    device: any
  ports:
    - 80
    - 443

output:
  elasticsearch:
    hosts: ["localhost:9200"]

Note: Ensure that the user running Packetbeat has sufficient permissions to capture network packets.

Warning: Packetbeat can generate a large volume of data depending on network traffic. Monitor resource usage carefully.

8. Using Heartbeat

Heartbeat is a lightweight shipper for the ELK Stack that monitors services and infrastructure. It checks the availability and responsiveness of services and sends monitoring data to Elasticsearch or Logstash.

Property	Description	Default Value
heartbeat.checks	Defines the list of checks to run.	Empty array
heartbeat.checks.tcp.host	Host to check (e.g., `localhost:80`).	None
heartbeat.checks.tcp.port	Port to check (e.g., `80`).	None
heartbeat.checks.http.host	Host to check (e.g., `https://example.com`).	None
heartbeat.checks.http.method	HTTP method to use (e.g., `GET`, `POST`).	`GET`
heartbeat.checks.http.response.status	Expected HTTP status code (e.g., `200`).	None
heartbeat.checks.http.response.body	Expected response body content (e.g., `OK`).	None
heartbeat.period	How often to run checks (e.g., `10s`, `1m`).	`10s`

Example: Basic TCP Check Configuration

heartbeat.checks:
  - type: tcp
    hosts: ["localhost:80"]
    timeout: 16s
    period: 10s

output.elasticsearch:
  hosts: ["localhost:9200"]

Example: HTTP Check with Status Code Validation

heartbeat.checks:
  - type: http
    hosts: ["https://example.com"]
    method: GET
    response:
      status: 200
    timeout: 16s
    period: 30s

output.elasticsearch:
  hosts: ["localhost:9200"]

Note: Heartbeat supports various check types including tcp, http, icmp, and ssh. Ensure that the target services are accessible from the Heartbeat instance.

9. Using Auditbeat

Auditbeat is a lightweight shipper that collects and sends audit logs from Linux systems to the ELK stack. It monitors system-level activities such as file access, process execution, and network connections.

Field	Description	Example Value
auditbeat.modules	Defines the modules to enable in Auditbeat.	`[{ "name": "auditd", "enabled": true }]`
auditbeat.modules.name	Name of the module to use (e.g., `auditd`).	`auditd`
auditbeat.modules.enabled	Enables or disables the module.	`true`
auditbeat.modules.config	Configuration options specific to the module.	`{ "auditd": { "include": ["/var/log/audit"] } }`

Example: Basic Auditbeat Configuration

auditbeat.modules:
- name: auditd
  enabled: true
  config:
    auditd:
      include:
        - /var/log/audit

Note: Auditbeat requires root privileges to read audit logs. Ensure the user running Auditbeat has appropriate permissions.

Warning: Audit logs can be large and may impact system performance. Monitor disk usage and configure log rotation accordingly.

10. Configuring Output to Elasticsearch

Option	Values	Notes
output.elasticsearch.hosts	`["http://localhost:9200"]`	Array of Elasticsearch hosts to send data to
output.elasticsearch.index	`beats-default`	Name of the index to write events to
output.elasticsearch.username	`elastic`	Username for Elasticsearch authentication
output.elasticsearch.password	`changeme`	Password for Elasticsearch authentication
output.elasticsearch.ssl.enabled	`false`	Enable SSL/TLS for communication with Elasticsearch
output.elasticsearch.ssl.certificate_authorities	`["/path/to/ca.crt"]`	Path to CA certificate file for SSL verification

Example: Basic Elasticsearch Output Configuration

output:
  elasticsearch:
    hosts: ["http://localhost:9200"]
    index: "beats-default"
    username: "elastic"
    password: "changeme"

Example: Secure Elasticsearch Output with SSL

output:
  elasticsearch:
    hosts: ["https://localhost:9200"]
    index: "beats-logs-%{+yyyy.MM.dd}"
    ssl:
      enabled: true
      certificate_authorities: ["/etc/ssl/certs/ca.crt"]
    username: "elastic"
    password: "changeme"

11. Configuring Output to Logstash

Option	Values	Notes
output.logstash.hosts	`["localhost:5044"]`	Logstash host and port to send events to
output.logstash.protocol	`tcp` or `udp`	Network protocol to use for communication
output.logstash.index	`beats`	Index name to use in Logstash (if applicable)
output.logstash.timeout	`30s`	Timeout for sending events to Logstash
output.logstash.compression_level	`0` to `9`	Compression level for data transmission (0 = none, 9 = max)

Example: Basic Logstash Output Configuration

output.logstash:
  hosts: ["localhost:5044"]
  protocol: tcp
  index: "beats"

Example: Advanced Logstash Output with Compression

output.logstash:
  hosts: ["logstash.example.com:5044"]
  protocol: tcp
  index: "beats-%{+YYYY.MM.dd}"
  timeout: 60s
  compression_level: 6

12. Managing Beat Modules

Module	Description	Enabled by Default	Notes
auditd	Collects audit logs from Linux systems	DEPRECATED since 7.0	Replaced by `linux-auditd`
apache	Collects Apache server logs and metrics	No	Requires `apache` module config
elasticsearch	Collects logs and metrics from Elasticsearch nodes	No	Supports `elasticsearch` module config
linux-auditd	Collects audit logs from Linux systems (newer version)	No	Replaces `auditd`
nginx	Collects Nginx server logs and metrics	No	Requires `nginx` module config
system	Collects system-level metrics (CPU, memory, disk)	Yes	Default module; minimal config needed

Example: Enabling the system module

# /etc/elastic/beats/filebeat/modules.d/system.yml
- module: system
  metricsets:
    - cpu
    - memory
    - disk

Example: Enabling the apache module

# /etc/elastic/beats/filebeat/modules.d/apache.yml
- module: apache
  enabled: true
  metricsets:
    - status

Note: To enable a module, create or edit the corresponding YAML file in /etc/elastic/beats/filebeat/modules.d/ and set enabled: true.

Warning: Disabling or removing a module does not delete its data from Elasticsearch. You must manually clean up indices if needed.