Configuring Health Checks

1. Implementing Active Health Checks

Property	Description	Default Value	Notes
interval	Time between health check probes	`5s`	Configurable in seconds
timeout	Timeout for each probe	`3s`	Must be less than interval
failureThreshold	Number of consecutive failures to mark as unhealthy	`3`	Adjustable for sensitivity
successThreshold	Number of consecutive successes to mark as healthy	`1`	Used for recovery from failure
initialDelay	Time to wait before starting health checks	`0s`	Useful for startup delays

Example: Active Health Check Configuration

healthCheck:
  active:
    interval: 5s
    timeout: 3s
    failureThreshold: 3
    successThreshold: 1
    initialDelay: 0s

Note: Active health checks are typically used in conjunction with passive checks to provide comprehensive service status monitoring.

2. Configuring Passive Health Checks

Passive health checks in service meshes rely on observing traffic patterns and response codes to determine the health of services. These checks do not actively probe services but instead infer their status based on observed behavior.

Configuration Option	Description	Default Value	Notes
passiveHealthCheck.enabled	Enables or disables passive health checks	`false`	Set to `true` to enable passive checks
passiveHealthCheck.interval	Time interval between passive checks	`5s`	Adjust based on service traffic volume
passiveHealthCheck.unhealthyThreshold	Number of consecutive failures to mark a service as unhealthy	`3`	Higher values reduce flapping
passiveHealthCheck.healthyThreshold	Number of consecutive successes to mark a service as healthy	`1`	Used to re-enable a previously unhealthy service
passiveHealthCheck.httpStatusCodes	HTTP status codes that indicate a failed request	`[500, 503]`	Customize based on application needs

Example: Enabling Passive Health Checks

passiveHealthCheck:
  enabled: true
  interval: 10s
  unhealthyThreshold: 5
  healthyThreshold: 2
  httpStatusCodes:
    - 500
    - 503
    - 429

Note: Passive health checks are particularly useful in environments with high traffic where active probing might introduce unnecessary load or latency.

Warning: Misconfiguring thresholds can lead to false positives or negatives, potentially causing cascading failures in the mesh.

3. Setting Health Check Intervals

Property	Values	Description
interval	`30s`	Time between health checks
timeout	`5s`	Maximum time to wait for a response
failureThreshold	`3`	Number of consecutive failures before marking as unhealthy

Example: Configuring Health Check Intervals

healthCheck:
  interval: 30s
  timeout: 5s
  failureThreshold: 3

4. Configuring Health Check Timeouts

Property	Description	Default Value	Notes
timeout	Timeout duration for health checks	`5s`	Duration after which a health check is considered failed
interval	Interval between health check executions	`10s`	Frequency of health check runs
failureThreshold	Number of consecutive failures before marking a service as unhealthy	`3`	Threshold for determining service health status

Example: Configuring Health Check Timeout in Istio

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: example-service
spec:
  host: example-service
  trafficPolicy:
    healthCheck:
      interval: 10s
      timeout: 5s
      failureThreshold: 3

5. Implementing HTTP Health Checks

Property	Value	Description
httpGet.path	`/health`	HTTP path to check for health status
httpGet.port	`8080`	Port to send the HTTP request to
httpGet.host	`localhost`	Host to send the HTTP request to
httpGet.scheme	`HTTP`	Scheme to use for the HTTP request (HTTP or HTTPS)
httpGet.httpHeaders	`[{name: "User-Agent", value: "kube-probe"}]`	Custom headers to include in the HTTP request

Example: HTTP Health Check Configuration

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
  - name: example-container
    image: nginx
    livenessProbe:
      httpGet:
        path: /health
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 30
      periodSeconds: 10

Note: The httpGet health check is ideal for services that expose a dedicated health endpoint.

Warning: Ensure that the path specified in httpGet.path is accessible and returns a 2xx status code for healthy services.

6. Configuring gRPC Health Checks

Property	Value	Description
grpc.health.v1.Health	`grpc.health.v1.Health`	gRPC service for health checking
service.health.check	`true`	Enable gRPC health checks
grpc.health.timeout	`5s`	Timeout for gRPC health check requests

Example: gRPC Health Check Configuration

apiVersion: v1
kind: Service
metadata:
  name: grpc-service
spec:
  selector:
    app: grpc-app
  ports:
  - port: 50051
    targetPort: 50051
  healthCheck:
    grpc:
      port: 50051
      service: grpc.health.v1.Health

Note: gRPC health checks require the grpc.health.v1.Health service to be implemented and exposed by the gRPC server.

7. Implementing TCP Health Checks

Property	Value	Description
tcpCheck	`{ port: 8080 }`	Defines the TCP port to check for connectivity
tcpCheck.interval	`5s`	How often to perform the TCP check
tcpCheck.timeout	`2s`	Timeout for each TCP check attempt
tcpCheck.failureThreshold	`3`	Number of consecutive failures before marking as unhealthy

Example: TCP Health Check Configuration

tcpCheck:
  port: 8080
  interval: 5s
  timeout: 2s
  failureThreshold: 3

Note: TCP health checks are useful for services that do not support HTTP/HTTPS endpoints but still need to be monitored for availability.

8. Setting Healthy/Unhealthy Thresholds

Threshold Type	Description	Default Value	Notes
healthyThreshold	Number of consecutive successful health checks required to mark a previously unhealthy endpoint as healthy.	`1`	Must be a positive integer. Increasing this value increases resilience to transient failures.
unhealthyThreshold	Number of consecutive failed health checks required to mark a healthy endpoint as unhealthy.	`3`	Higher values provide more tolerance for intermittent failures but may delay detection of real issues.

Example: Configuring Thresholds in Istio

apiVersion: networking.istio.io/v1beta1
kind: HTTPRoute
metadata:
  name: example-route
spec:
  host: example.com
  rules:
  - backend:
      service:
        name: example-service
        port: 80
    healthCheck:
      healthyThreshold: 2
      unhealthyThreshold: 5

9. Managing Health Check Endpoints

Endpoint	Description	Method	Notes
/health/live	Indicates if the service is alive	`GET`	Used for liveness probes in Kubernetes
/health/ready	Indicates if the service is ready to serve traffic	`GET`	Used for readiness probes in Kubernetes
/health/alive	Alias for liveness check	`GET`	Deprecated in favor of `/health/live`

Example: Configuring Health Checks in Istio

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  ports:
  - port: 8080
    targetPort: 8080
  selector:
    app: my-app
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-service-dr
spec:
  host: my-service
  trafficPolicy:
    healthCheck:
      interval: 5s
      timeout: 2s
      unhealthyThreshold: 3
      healthyThreshold: 2
      path: /health/ready

Note: The path field in the health check configuration should match the actual endpoint exposed by your application.

Warning: Misconfigured health checks can lead to service disruptions. Ensure that the endpoints are lightweight and do not impact application performance.

10. Troubleshooting Health Check Failures

Example: Debugging a failing liveness probe

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: app
    image: myapp:latest
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 3
      failureThreshold: 3

Field	Description	Default Value	Notes
initialDelaySeconds	Number of seconds after container starts before probe begins	0	Useful for slow-starting applications
periodSeconds	How often (in seconds) to execute the probe	10	Must be >= timeoutSeconds
timeoutSeconds	Number of seconds after which the probe times out	1	Must be < periodSeconds
failureThreshold	Number of consecutive failures to mark container as unhealthy	3	Higher values allow for transient issues

Note: If a health check fails, Kubernetes will restart the container. Ensure your application can handle restarts gracefully.

Warning: Setting timeoutSeconds too low can cause false positives during normal operation.

Common Causes of Health Check Failures

Application not ready: Service is up but not serving requests
Incorrect endpoint: Path or port does not match actual health check
Network issues: Pod cannot reach the health check endpoint
Resource exhaustion: CPU or memory limits causing delays
Timeouts: Probe timeout is too short for application response

Health Check Types Comparison

HTTP Get: Checks response from a URL endpoint
TCPSocket: Validates port connectivity only
Exec: Runs a command inside the container

11. Implementing Custom Health Check Logic

Property	Description	Example
customCheck	Defines a custom health check logic using a script or function	`customCheck: (context) => { return context.status === 'healthy'; }`
timeout	Timeout for custom health check execution in milliseconds	`timeout: 5000`
interval	Interval between custom health check executions in seconds	`interval: 30`

Example: Custom Health Check Implementation

const customHealthCheck = {
  name: 'database-health',
  check: (context) => {
    const dbStatus = context.database.status;
    return dbStatus === 'connected';
  },
  timeout: 3000,
  interval: 60
};

module.exports = customHealthCheck;

Note: Custom health checks must return a boolean value indicating the status of the component being checked.

Warning: Ensure that custom health check logic does not introduce performance bottlenecks or excessive load on the system.

12. Configuring Health Check Events

Event Type	Description	Configuration	Notes
healthCheckEvent	Triggers when a health check fails or recovers	`"healthCheckEvent"`	Used for alerting and monitoring systems
healthCheckFailure	Event triggered on failed health check	`"healthCheckFailure"`	Can be used to initiate failover or alerting
healthCheckRecovery	Event triggered when a previously failed check recovers	`"healthCheckRecovery"`	Useful for logging and status tracking

Example: Configuring Health Check Events

apiVersion: v1
kind: Service
metadata:
  name: my-service
  annotations:
    traffic.sidecar.istio.io/healthCheckEvent: "true"
spec:
  ports:
  - port: 80
    targetPort: 8080

Note: Health check events can be used to trigger external monitoring systems or alerting mechanisms.