Implementing Circuit Breaking

1. Configuring Maximum Connections

Property	Values	Description
maxConnections	`1000`	Maximum number of concurrent connections to a service endpoint
maxPendingRequests	`100`	Maximum number of pending requests to a service endpoint
maxRetries	`3`	Maximum number of retries for failed requests

Example: Configuring maxConnections in Istio

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: example-destination
spec:
  host: example-service
  trafficPolicy:
    connectionPool:
      http:
        maxConnections: 1000
        maxPendingRequests: 100
      tcp:
        maxConnections: 1000

2. Setting Maximum Pending Requests

Property	Values	Description
maxPendingRequests	`100`	Maximum number of pending requests allowed before circuit breaker opens
maxConcurrentRequests	`200`	Maximum concurrent requests allowed to the service

Example: Configuring maxPendingRequests in Istio

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: example-service
spec:
  host: example-service
  trafficPolicy:
    circuitBreaker:
      simpleCb:
        maxPendingRequests: 100
        maxConcurrentRequests: 200

3. Configuring Maximum Requests

Property	Values	Description
maxRequests	`100`	Maximum number of requests allowed before circuit opens
maxPendingRequests	`10`	Maximum number of pending requests while circuit is open

Example: Configuring maxRequests in Istio

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: example-service
spec:
  host: example-service
  trafficPolicy:
    circuitBreaker:
      simpleCb:
        maxRequests: 100
        maxPendingRequests: 10

4. Setting Maximum Requests Per Connection

Property	Values	Description
maxRequestsPerConnection	`1000`	Maximum number of requests allowed per connection before closing and reopening the connection.
maxRequestsPerConnection (per route)	`500`	Override for specific routes to limit requests per connection.

Example: Configuring maxRequestsPerConnection in Istio

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: productpage-policy
spec:
  host: productpage
  trafficPolicy:
    connectionPool:
      http:
        maxRequestsPerConnection: 1000

Note: Setting maxRequestsPerConnection helps prevent connection-level resource exhaustion and improves resilience in high-traffic environments.

5. Configuring Maximum Retries

Property	Values	Description
retries.max	`3`	Maximum number of retry attempts for a failed request
retries.backoff	`exponential`	Backoff strategy for retry delays (linear, exponential)
retries.perRetryTimeout	`2s`	Timeout for each individual retry attempt

Example: Retry Configuration in Istio

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: backend-retry-config
spec:
  host: backend-service
  trafficPolicy:
    retryPolicy:
      maxRetries: 3
      perRetryTimeout: 2s
      retryOn: "5xx,connect-failure"

6. Setting Outlier Detection

Property	Description	Default Value	Notes
consecutive5xx	Number of 5xx errors to mark a host as unhealthy	5	Can be overridden per route or cluster
timeout	Time window to consider outliers (in seconds)	60	Used for outlier detection logic
interval	Interval between outlier detection checks (in seconds)	10	Frequency of outlier status updates
maxEjectionPercent	Maximum percentage of hosts to eject from the cluster	10%	Prevents complete cluster ejection

Example: Outlier Detection Configuration

outlier_detection:
  consecutive5xx: 5
  timeout: 60
  interval: 10
  maxEjectionPercent: 10

Note: Outlier detection helps maintain service reliability by removing unhealthy hosts from the load balancing pool.

7. Configuring Consecutive Errors Threshold

Property	Values	Description
circuitBreaker.consecutiveErrors	`5`	Number of consecutive failures before opening the circuit
circuitBreaker.timeout	`30s`	Timeout for requests to the backend service
circuitBreaker.retryAttempts	`3`	Number of retry attempts before circuit opens

Example: Configuring Consecutive Errors Threshold in Istio

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-destination-rule
spec:
  host: my-service
  trafficPolicy:
    circuitBreaker:
      simpleCb:
        consecutiveErrors: 5
        timeout: 30s
        retryAttempts: 3

8. Setting Ejection Duration

Property	Description	Default Value	Notes
ejectionDuration	Duration for which a host is ejected from the circuit breaker	`30s`	Host remains ejected for this duration before being re-evaluated

Example: Configuring Ejection Duration

circuits:
  - name: my-circuit
    ejectionDuration: 60s

9. Configuring Max Ejection Percentage

Property	Values	Description
maxEjectionPercentage	`10`	Maximum percentage of hosts that can be ejected from the circuit breaker.
maxEjectionPercentage	`25`	NEW Default value in latest versions.

Example: Configuring maxEjectionPercentage in a Service Mesh

apiVersion: networking.istio.io/v1beta1
kind: CircuitBreaker
metadata:
  name: my-circuit-breaker
spec:
  destination:
    host: example-service
  circuitBreaker:
    simpleCb:
      maxEjectionPercentage: 25

10. Monitoring Circuit Breaker State

Property	Description	Values	Notes
circuitBreaker.state	Current state of the circuit breaker	`OPEN`, `HALF_OPEN`, `CLOSED`	Indicates whether the circuit is blocking requests, allowing a single request, or permitting all requests
circuitBreaker.failureCount	Number of consecutive failures since last success	Integer	Used to determine when to transition to OPEN state
circuitBreaker.resetTimeout	Time in milliseconds before transitioning to HALF_OPEN	Integer (ms)	Configurable threshold for circuit breaker reset behavior
circuitBreaker.requestVolumeThreshold	Minimum number of requests before circuit breaker can open	Integer	Defaults to 20 requests before considering failure rate
circuitBreaker.failureThreshold	Failure rate threshold to open circuit	Percentage (e.g., `50`)	Defaults to 50% of requests failing

Example: Circuit Breaker State in Prometheus Metrics

circuit_breaker_state{service="user-service", state="CLOSED"} 1
circuit_breaker_failure_count{service="user-service"} 3
circuit_breaker_reset_timeout{service="user-service"} 5000

Note: The circuit breaker state can be monitored via service mesh telemetry systems like Prometheus or Grafana to track health and performance of services under load.

11. Implementing Circuit Breaker Metrics

Example: Configuring Prometheus Metrics for Circuit Breaker

apiVersion: v1
kind: Service
metadata:
  name: circuit-breaker-metrics
  labels:
    app: service-mesh
spec:
  selector:
    app: istio-pilot
  ports:
  - port: 15014
    name: metrics
    protocol: TCP

Metric Name	Description	Unit	Labels
istio_circuit_breaker_calls_total	Total number of calls made through circuit breaker	counter	`destination_service`, `source_service`
istio_circuit_breaker_errors_total	Total number of errors encountered by circuit breaker	counter	`destination_service`, `source_service`
istio_circuit_breaker_opened_total	Total number of times circuit breaker opened	counter	`destination_service`, `source_service`
istio_circuit_breaker_half_opened_total	Total number of times circuit breaker transitioned to half-open state	counter	`destination_service`, `source_service`

Note: These metrics are exposed via Prometheus endpoint at /metrics and can be scraped by Prometheus server for monitoring.

Example: Grafana Dashboard Configuration

{
  "dashboard": {
    "title": "Service Mesh Circuit Breaker Metrics",
    "panels": [
      {
        "title": "Circuit Breaker Errors",
        "targets": [
          {
            "expr": "rate(istio_circuit_breaker_errors_total[5m])"
          }
        ]
      },
      {
        "title": "Circuit Breaker Opened",
        "targets": [
          {
            "expr": "rate(istio_circuit_breaker_opened_total[5m])"
          }
        ]
      }
    ]
  }
}

12. Troubleshooting Circuit Breaker Issues

Issue	Description	Common Causes	Solution
Circuit trips too frequently	Unexpectedly opens and closes rapidly	High latency or frequent timeouts, misconfigured thresholds	Review timeout and error threshold settings; adjust to realistic values
Circuit stays open permanently	Never recovers despite healthy upstream services	Incorrect reset timeout, persistent errors in upstream	Adjust circuit reset timeout; verify upstream service health
Latency spikes not triggering circuit	High latency does not cause circuit to open	Incorrect timeout settings, traffic not reaching circuit breaker	Verify latency thresholds and ensure all traffic is routed through the circuit breaker
Unstable traffic patterns	Fluctuating success/failure rates cause circuit instability	High variance in request patterns, low error thresholds	Increase error thresholds or use adaptive circuit breaker logic

Example: Circuit Breaker Configuration with Logging

resilience:
  circuitBreaker:
    name: api-service-circuit-breaker
    timeout: 5s
    maxRetries: 3
    failureThreshold: 50
    resetTimeout: 30s
    logging:
      enabled: true
      level: INFO

Note: Use logging to monitor circuit state transitions and error patterns for effective troubleshooting.

Common Troubleshooting Steps

Check circuit breaker configuration parameters (timeout, thresholds)
Verify that traffic is correctly routed through the circuit breaker
Analyze logs for error patterns and latency spikes
Ensure upstream services are stable and responsive
Adjust reset timeout to allow for proper recovery