Skip to main content

Deploying on Kubernetes

1. Understanding Kubernetes Deployment

Example: Basic Kubernetes Deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
        ports:
        - containerPort: 9200
Field Description Values
apiVersion API version to use for the deployment apps/v1
kind Type of Kubernetes resource Deployment
metadata.name Name of the deployment String
spec.replicas Number of desired pods Integer
spec.selector.matchLabels Label selector for pods Map of labels
spec.template.metadata.labels Labels to apply to pods Map of labels
spec.template.spec.containers List of containers in the pod Array of container specs
Note: The deployment ensures that the specified number of pods are running at all times.
Warning: Ensure that the container image version matches the desired ELK Stack version.

2. Using ECK

Example: Deploying Elasticsearch with ECK

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 8.15.0
  nodeSets:
  - name: default
    count: 3
    config:
      node.roles: [master, data, ingest]
Field Description Values
apiVersion API version of the ECK resource elasticsearch.k8s.elastic.co/v1
kind Type of ECK resource Elasticsearch
metadata.name Name of the Elasticsearch cluster elasticsearch
spec.version Version of Elasticsearch to deploy 8.15.0
spec.nodeSets.name Name of the node set default
spec.nodeSets.count Number of nodes in the set 3
spec.nodeSets.config.node.roles Roles assigned to nodes [master, data, ingest]
Note: ECK (Elastic Cloud on Kubernetes) simplifies deployment and management of Elasticsearch, Logstash, and Kibana on Kubernetes.
Warning: Ensure sufficient resources (CPU, memory) are allocated to nodes to avoid performance issues.

Key Features of ECK

  • Automated Elasticsearch cluster lifecycle management
  • Easy deployment and scaling of Elasticsearch clusters
  • Integration with Kubernetes-native tools like Helm and Kustomize
  • Support for Elasticsearch security features (TLS, authentication)

3. Installing ECK Operator

Example: Install ECK Operator using Helm

helm repo add elastic https://helm.elastic.co
helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace
Step Description Action
1 Add Elastic Helm repository helm repo add elastic https://helm.elastic.co
2 Install ECK Operator helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace
Note: Ensure the namespace elastic-system exists or is created during installation.
Warning: ECK operator requires Kubernetes 1.20 or higher.

Installation Summary

  • NEW Helm chart for ECK operator is available at elastic/eck-operator
  • Operator will be installed in the elastic-system namespace
  • Supports Kubernetes 1.20+

4. Deploying Elasticsearch Cluster

Example: Basic Elasticsearch Deployment

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
spec:
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:8.11.3
        ports:
        - containerPort: 9200
        env:
        - name: discovery.type
          value: single-node
        - name: ES_JAVA_OPTS
          value: -Xms1g -Xmx1g
        volumeMounts:
        - name: elasticsearch-data
          mountPath: /usr/share/elasticsearch/data
      volumes:
      - name: elasticsearch-data
        emptyDir: {}
Property Description Default Value Notes
discovery.type Discovery type for cluster formation single-node Use single-node for development, zen or gossip in production
ES_JAVA_OPTS Java options for Elasticsearch -Xms1g -Xmx1g Adjust memory limits based on cluster size
replicas Number of Elasticsearch nodes 1 For production, use 3 or more nodes for high availability
Note: Ensure that the elasticsearch-data volume is backed by persistent storage in production environments.
Warning: Elasticsearch requires sufficient memory and CPU resources. Misconfiguration can lead to node failures or performance degradation.

5. Deploying Kibana

Property Value Description
image docker.elastic.co/kibana/kibana:8.15.3 Kibana Docker image to use
service.port 5601 Port to expose Kibana service
config.host "0.0.0.0" Host to bind Kibana server
elasticsearch.hosts ["http://elasticsearch:9200"] URLs of Elasticsearch instances to connect to

Example: Kibana Deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:8.15.3
        env:
        - name: ELASTICSEARCH_HOSTS
          value: "http://elasticsearch:9200"
        ports:
        - containerPort: 5601

Example: Kibana Service YAML

apiVersion: v1
kind: Service
metadata:
  name: kibana-service
spec:
  selector:
    app: kibana
  ports:
  - port: 5601
    targetPort: 5601
  type: LoadBalancer
Note: Ensure Elasticsearch is running and accessible before deploying Kibana.

6. Deploying Beats

Example: Deploying Filebeat on Kubernetes

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: filebeat
  template:
    metadata:
      labels:
        app: filebeat
    spec:
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:8.15.0
        args:
        - -c
        - /etc/filebeat/filebeat.yml
        - -e
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat/filebeat.yml
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: logs
          mountPath: /var/log/containers
      volumes:
      - name: config
        configMap:
          name: filebeat-config
      - name: data
        hostPath:
          path: /var/lib/filebeat
      - name: logs
        hostPath:
          path: /var/log/containers
Property Description Default
image Filebeat Docker image to use docker.elastic.co/beats/filebeat:8.15.0
args Command-line arguments to pass to Filebeat ["-c", "/etc/filebeat/filebeat.yml", "-e"]
volumeMounts Mounts for configuration and data directories None
Note: Ensure that the filebeat.yml configMap is properly configured to point logs to Elasticsearch or Logstash.
Warning: Using DaemonSet ensures that Filebeat runs on every node in the cluster. Make sure to configure appropriate resource limits to avoid overloading nodes.

7. Configuring StatefulSets

Property Description Example
serviceName Name of the headless service to be used for network identity elasticsearch-svc
replicas Number of pods in the StatefulSet 3
podManagementPolicy Controls how pods are created and updated (e.g., Parallel or OrderedReady) OrderedReady
updateStrategy.type Update strategy for pods (e.g., RollingUpdate) RollingUpdate

Example: Basic Elasticsearch StatefulSet Configuration

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
spec:
  serviceName: elasticsearch-svc
  replicas: 3
  podManagementPolicy: OrderedReady
  updateStrategy:
    type: RollingUpdate
  template:
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:8.15.0
        ports:
        - containerPort: 9200
Note: StatefulSets are ideal for stateful applications like Elasticsearch where stable network identifiers and persistent storage are required.

8. Setting Resource Limits

Resource Type Property Default Value Description
elasticsearch.resources.limits.cpu 1 1 CPU core Maximum CPU allocation for Elasticsearch pods
elasticsearch.resources.limits.memory 2Gi 2 GiB RAM Maximum memory allocation for Elasticsearch pods
logstash.resources.limits.cpu 1 1 CPU core Maximum CPU allocation for Logstash pods
logstash.resources.limits.memory 1Gi 1 GiB RAM Maximum memory allocation for Logstash pods
kibana.resources.limits.cpu 1 1 CPU core Maximum CPU allocation for Kibana pods
kibana.resources.limits.memory 1Gi 1 GiB RAM Maximum memory allocation for Kibana pods

Example: Resource Limits Configuration

elasticsearch:
  resources:
    limits:
      cpu: "2"
      memory: 4Gi
logstash:
  resources:
    limits:
      cpu: "1"
      memory: 2Gi
kibana:
  resources:
    limits:
      cpu: "1"
      memory: 1Gi
Note: Setting appropriate resource limits prevents pods from consuming excessive resources and ensures stable cluster performance.

9. Using Init Containers

Example: Elasticsearch init container to set file permissions

apiVersion: v1
kind: Pod
metadata:
  name: elasticsearch-pod
spec:
  initContainers:
  - name: elasticsearch-init
    image: busybox
    command:
    - sh
    - -c
    - |
      echo "Setting file permissions for Elasticsearch..."
      chown -R 1000:0 /usr/share/elasticsearch/data
      chmod -R 750 /usr/share/elasticsearch/data
    volumeMounts:
    - name: elasticsearch-data
      mountPath: /usr/share/elasticsearch/data
  containers:
  - name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    ports:
    - containerPort: 9200
    volumeMounts:
    - name: elasticsearch-data
      mountPath: /usr/share/elasticsearch/data
  volumes:
  - name: elasticsearch-data
    emptyDir: {}
Property Description Default Value
initContainers List of init containers to run before the main container starts. None
name Name of the init container. None
image Docker image to use for the init container. None
command Command to run inside the init container. None
volumeMounts Volume mounts to make available inside the init container. None
Note: Init containers run in sequence and must complete successfully before the main containers start.
Warning: Ensure that init containers do not interfere with the main container's lifecycle or data persistence.

10. Configuring Health Checks

Check Type Configuration Description
Liveness Probe livenessProbe: Indicates if the pod is running correctly. If it fails, Kubernetes will restart the pod.
Readiness Probe readinessProbe: Determines if the pod is ready to serve traffic. If it fails, the pod will not receive traffic.

Example: Liveness Probe Configuration

livenessProbe:
  httpGet:
    path: /health
    port: 9200
  initialDelaySeconds: 30
  periodSeconds: 10

Example: Readiness Probe Configuration

readinessProbe:
  httpGet:
    path: /ready
    port: 9200
  initialDelaySeconds: 5
  periodSeconds: 5
Note: Ensure that the health endpoints (/health, /ready) are properly implemented in your ELK components to return appropriate HTTP status codes (e.g., 200 for success).

11. Using Persistent Volume Claims

Property Description Example
persistentVolumeClaim Defines the PVC to use for storing data elasticsearch-pvc
storageClassName Storage class to use for dynamic provisioning fast-ssd
resources.requests.storage Amount of storage requested 100Gi

Example: Define PVC for Elasticsearch

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: elasticsearch-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: fast-ssd

Example: Mount PVC in Elasticsearch StatefulSet

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
spec:
  template:
    spec:
      containers:
      - name: elasticsearch
        volumeMounts:
        - name: elasticsearch-storage
          mountPath: /usr/share/elasticsearch/data
      volumes:
      - name: elasticsearch-storage
        persistentVolumeClaim:
          claimName: elasticsearch-pvc

12. Implementing Pod Disruption Budgets

Example: Basic PodDisruptionBudget Configuration

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: elasticsearch-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: elasticsearch
Field Description Values
apiVersion API version of the PDB resource policy/v1
kind Kind of the Kubernetes resource PodDisruptionBudget
metadata.name Name of the PDB String
spec.minAvailable Minimum number of pods that must remain available during disruptions Integer or Percentage (e.g., 2, 50%)
spec.maxUnavailable Maximum number of pods that can be unavailable during disruptions Integer or Percentage (e.g., 1, 25%)
spec.selector Label selector to identify pods managed by this PDB Object with matchLabels
Note: Use minAvailable or maxUnavailable, not both. If neither is specified, the default is maxUnavailable: 0.

Example: PDB with maxUnavailable

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: kibana-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: kibana

Best Practices for PDBs in ELK Stack

  • Ensure that minAvailable or maxUnavailable values align with the desired availability SLA.
  • Use PDBs to prevent disruptions during rolling updates or node drains in Elasticsearch and Kibana deployments.
  • Do not set maxUnavailable to 0 for stateful services like Elasticsearch, as it can prevent any disruptions.

Was this article helpful?