Deploying on Docker

1. Understanding Docker Deployment

The ELK Stack (Elasticsearch, Logstash, Kibana) can be deployed using Docker containers to simplify deployment and management. This approach provides isolation, portability, and scalability for each component of the stack.

Component	Description	Benefits
Elasticsearch	Search and analytics engine	Fast search, scalable storage
Logstash	Data processing pipeline	Real-time data ingestion and transformation
Kibana	Visualization and dashboard tool	Interactive dashboards, data exploration

Deployment Approaches

Standalone: Each service runs independently
Docker Compose: Single configuration file for multi-container deployment
Kubernetes: Orchestration platform for containerized applications

2. Running Elasticsearch Container

Elasticsearch can be run as a Docker container with specific configurations for memory and network settings.

Example: Running Elasticsearch

docker run -d \
  --name elasticsearch \
  -p 9200:9200 \
  -p 9300:9300 \
  -e "discovery.type=single-node" \
  docker.elastic.co/elasticsearch/elasticsearch:8.15.0

Option	Values	Description
--name	`elasticsearch`	Name of the container
-p 9200:9200	`9200`	HTTP port for API access
-p 9300:9300	`9300`	Transport port for inter-node communication
-e discovery.type=single-node	`single-node`	Configures Elasticsearch to run in single-node mode

3. Running Logstash Container

Logstash can be run as a Docker container to process and transform logs before sending them to Elasticsearch.

Example: Running Logstash

docker run -d \
  --name logstash \
  -p 5044:5044 \
  -v /path/to/logstash.conf:/usr/share/logstash/pipeline/logstash.conf \
  docker.elastic.co/logstash/logstash:8.15.0

Option	Values	Description
--name	`logstash`	Name of the container
-p 5044:5044	`5044`	Input port for Logstash
-v /path/to/logstash.conf:/usr/share/logstash/pipeline/logstash.conf	`logstash.conf`	Mounts Logstash configuration file

4. Running Kibana Container

Kibana can be run as a Docker container to provide visualization and dashboard capabilities for the ELK Stack.

Example: Running Kibana

docker run -d \
  --name kibana \
  -p 5601:5601 \
  -e "ELASTICSEARCH_HOSTS=[\"http://elasticsearch:9200\"]" \
  docker.elastic.co/kibana/kibana:8.15.0

Option	Values	Description
--name	`kibana`	Name of the container
-p 5601:5601	`5601`	Port for Kibana UI access
-e ELASTICSEARCH_HOSTS=["http://elasticsearch:9200"]	`http://elasticsearch:9200`	Configures Kibana to connect to Elasticsearch

5. Using Docker Compose

Docker Compose allows defining and running multi-container Docker applications using a YAML file.

Example: docker-compose.yml

version: "3.8"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
    ports:
      - "9200:9200"
      - "9300:9300"

  logstash:
    image: docker.elastic.co/logstash/logstash:8.15.0
    container_name: logstash
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5044:5044"

  kibana:
    image: docker.elastic.co/kibana/kibana:8.15.0
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=["http://elasticsearch:9200"]
    ports:
      - "5601:5601"

Note: Ensure that the logstash.conf file exists in the same directory as the compose file.

6. Configuring Persistent Volumes

Persistent volumes ensure that data is retained even when containers are stopped or removed.

Example: Using Volumes in Docker Compose

version: "3.8"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.0
    container_name: elasticsearch
    volumes:
      - esdata:/usr/share/elasticsearch/data
    environment:
      - discovery.type=single-node

  logstash:
    image: docker.elastic.co/logstash/logstash:8.15.0
    container_name: logstash
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
      - logstashdata:/usr/share/logstash/data

volumes:
  esdata:
  logstashdata:

Volume	Description	Purpose
esdata	Elasticsearch data directory	Persists Elasticsearch index and configuration data
logstashdata	Logstash data directory	Persists Logstash pipeline and configuration data

7. Setting Environment Variables

Environment variables are used to configure components like Elasticsearch, Logstash, and Kibana.

Example: Setting Environment Variables

docker run -d \
  --name elasticsearch \
  -e "ES_JAVA_OPTS=-Xms1g -Xmx1g" \
  -e "discovery.type=single-node" \
  docker.elastic.co/elasticsearch/elasticsearch:8.15.0

Variable	Values	Description
ES_JAVA_OPTS	`-Xms1g -Xmx1g`	Java heap size settings for Elasticsearch
discovery.type	`single-node`	Configures Elasticsearch to run in single-node mode

8. Configuring Container Resources

Resource limits and reservations can be set to control how much CPU and memory each container uses.

Example: Resource Constraints

version: "3.8"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.0
    container_name: elasticsearch
    deploy:
      resources:
        limits:
          memory: 2G
        reservations:
          memory: 1G

Resource	Configuration	Description
memory	`2G`	Maximum memory usage for the container
cpu	`1.5`	CPU shares allocated to the container

9. Using Docker Networks

Docker networks provide isolated communication channels between containers.

Example: Creating a Custom Network

docker network create elk-network

docker run -d \
  --name elasticsearch \
  --network elk-network \
  docker.elastic.co/elasticsearch/elasticsearch:8.15.0

docker run -d \
  --name kibana \
  --network elk-network \
  docker.elastic.co/kibana/kibana:8.15.0

Network	Description	Benefits
elk-network	Custom Docker network for ELK stack	Isolated communication between containers

10. Configuring Health Checks

Health checks ensure that containers are running correctly and can be used to trigger automatic restarts.

Example: Health Check in Docker Compose

version: "3.8"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.0
    container_name: elasticsearch
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=30s"]
      interval: 30s
      timeout: 10s
      retries: 3

Property	Values	Description
test	`curl -f http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=30s`	Command to check Elasticsearch health
interval	`30s`	How often to run the health check
timeout	`10s`	Maximum time to wait for a response
retries	`3`	Number of attempts before marking as unhealthy

11. Managing Docker Logs

Docker logs can be managed using various drivers to control where and how logs are stored.

Example: Log Driver Configuration

version: "3.8"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.0
    container_name: elasticsearch
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

Driver	Options	Description
json-file	`max-size: "10m"`	Stores logs in JSON format with size limits
syslog	`syslog-address: "tcp://localhost:514"`	Sends logs to a syslog server

12. Using Docker Secrets

Docker secrets provide a way to securely store and manage sensitive data like passwords or API keys.

Example: Using Docker Secrets

version: "3.8"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.0
    container_name: elasticsearch
    secrets:
      - es_password

secrets:
  es_password:
    file: ./es-password.txt

Secret	File	Description
es_password	`./es-password.txt`	Stores Elasticsearch password securely