Skip to main content
Mapagam

Configuring Request Retries

1. Setting Maximum Retry Attempts

Property Values Description
retry.maxAttempts 3 Maximum number of retry attempts for a failed request
retry.backoffPolicy exponential, constant Backoff strategy for retry delays
retry.perTryTimeout 2s Timeout for each individual retry attempt

Example: Configuring Retry Attempts in Istio

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example-service
  http:
  - route:
    - destination:
        host: example-service
    retries:
      attempts: 3
      perTryTimeout: 2s
      backoffPolicy: exponential

2. Configuring Retry Timeout

Property Values Description
retry.timeout 30s Maximum time to wait for a retry attempt before giving up
retry.backoff exponential Backoff strategy for retry delays (e.g., linear, exponential)
retry.maxAttempts 3 Maximum number of retry attempts before failing the request

Example: Retry Configuration in Istio

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example-service
  http:
  - route:
    - destination:
        host: example-service
    retries:
      attempts: 3
      perTryTimeout: 2s
      backoffPolicy: exponential
Note: Setting perTryTimeout to a value less than the overall timeout ensures retries don't indefinitely block requests.

3. Setting Retry Conditions

Condition Description Default Notes
retryOn Specifies the conditions under which a request should be retried 5xx Can include 5xx, gateway-error, connect-failure, refused-stream
retryOn.httpStatuses HTTP status codes to retry on None Array of HTTP status codes (e.g., [503, 504])
retryOn.httpHeaders HTTP headers to check for retry conditions None Object with header name and value to match
retryOn.httpMethods HTTP methods to retry on None Array of HTTP methods (e.g., ["GET", "POST"])

Example: Retry on 503 and 504 Status Codes

retryPolicy:
  retryOn:
    httpStatuses:
      - 503
      - 504

Example: Retry on Gateway Errors and Connect Failures

retryPolicy:
  retryOn:
    gateway-error: true
    connect-failure: true
Note: Retry conditions can be combined to create more complex retry logic.

4. Implementing Per-Try Timeout

Property Values Description
perTryTimeout 500ms, 1s, 2.5s Timeout for each individual retry attempt
perTryTimeout.unit ms, s, m Time unit for perTryTimeout value

Example: Setting perTryTimeout in Istio

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example-service
  http:
  - route:
    - destination:
        host: example-service
    retries:
      attempts: 3
      perTryTimeout: 2s

5. Configuring Retry Budget

Property Description Default Value Notes
retryBudget.minRetriesPerSecond Minimum number of retries per second to allow 10 Used to prevent bursty retry behavior
retryBudget.maxRetryRatio Maximum ratio of retries to requests allowed 0.2 Enforces a cap on retry traffic
retryBudget.retryWindow Time window (in seconds) to calculate retry ratio 60 Defines the sliding window for retry rate calculation

Example: Retry Budget Configuration

retryBudget:
  minRetriesPerSecond: 5
  maxRetryRatio: 0.15
  retryWindow: 30
Note: The retry budget is enforced at the proxy level and applies to all outbound requests.

6. Implementing Exponential Backoff

Property Values Description
retryPolicy exponentialBackoff Specifies the retry policy to use for request retries
baseInterval 50ms Initial delay before the first retry attempt
maxInterval 5s Maximum delay between retry attempts
maxAttempts 3 Maximum number of retry attempts before giving up

Example: Retry Policy Configuration

retryPolicy:
  exponentialBackoff:
    baseInterval: 50ms
    maxInterval: 5s
    maxAttempts: 3
Note: Exponential backoff increases the delay between retries exponentially to avoid overwhelming the system.
Warning: Setting maxInterval too high can lead to long wait times for clients.

7. Setting Retry Headers

Header Description Values Notes
Retry-After Indicates how long to wait before retrying a request 30 (seconds) Used by clients to determine backoff time
X-Retry-Count Number of retries attempted so far 2 Custom header, not part of HTTP spec
X-Retry-After Time in seconds to wait before retrying 5 Used in service mesh implementations for retry logic

Example: Configuring Retry Headers in Istio

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example-service
  http:
  - route:
    - destination:
        host: example-service
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: 5xx,connect-failure
    headers:
      response:
        add:
          Retry-After: "30"
          X-Retry-Count: "0"
Note: Retry headers are often added by the service mesh itself during retry logic, but can also be customized via VirtualService configuration.

8. Implementing Idempotent Retries

Property Description Values Notes
retryPolicy Defines retry policy configuration idempotent Enables idempotent retry behavior
maxRetries Maximum number of retry attempts 3 Default value for idempotent retries
backoff Backoff strategy for retries exponential Exponential backoff with jitter

Example: Configuring Idempotent Retries

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example-service
  http:
  - route:
    - destination:
        host: example-service
    retries:
      attempts: 3
      perTryTimeout: 2s
      backoff:
        baseInterval: 10ms
        maxInterval: 1s
Note: Idempotent retries are safe to use for operations that can be safely retried without side effects.

9. Configuring Retry on Connect Failure

Property Description Default Value Notes
retryPolicy Defines retry policy for connection failures {} Object containing retry configuration
retryPolicy.retryOn Specifies conditions under which to retry "connect-failure" Can include 5xx, gateway-error, etc.
retryPolicy.attempts Number of retry attempts 3 Maximum number of retries before giving up
retryPolicy.perTryTimeout Timeout per retry attempt 2s Must be less than overall timeout

Example: Retry on Connect Failure

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example-service
  http:
  - route:
    - destination:
        host: example-service
    retryPolicy:
      retryOn: "connect-failure"
      attempts: 3
      perTryTimeout: 2s
Note: Retry on connect failure is particularly useful in microservices environments where transient network issues can occur.

10. Managing Retry Policies

Policy Description Configuration Notes
retryPolicy Defines retry behavior for failed requests { "retryPolicy": { "attempts": 3, "backoff": "exponential", "conditions": ["5xx", "timeout"] } } Applies to all outbound requests
attempts Number of retry attempts 3 Default is 0 (no retries)
backoff Backoff strategy for retries "exponential" Options: constant, exponential
conditions Failure conditions to trigger retries ["5xx", "timeout"] Supports HTTP status codes and timeouts

Example: Retry Policy Configuration

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example-service
  http:
  - route:
    - destination:
        host: example-service
    retries:
      attempts: 3
      backoff:
        baseInterval: 100ms
        maxInterval: 1s
      conditions:
      - "5xx"
      - "timeout"
Note: Retry policies apply to all outbound requests from a service. Configure carefully to avoid cascading failures.

11. Configuring Retry Hedging

Property Description Default Value Notes
retryPolicy Defines the retry policy configuration for requests null Can be configured to specify retry behavior
retryPolicy.http HTTP-specific retry configuration {} Used to define HTTP status codes and conditions for retries
retryPolicy.http.retryOn HTTP status codes to retry on [] Array of status codes to trigger retries (e.g., [503])
retryPolicy.http.numRetries Number of retry attempts 3 Maximum number of retries before giving up
retryPolicy.http.perRetryTimeout Timeout per retry attempt 2s Time to wait before retrying a request
retryPolicy.http.retryableStatusCodes HTTP status codes that are retryable [503] Defines which status codes should trigger a retry

Example: Retry Policy Configuration

retryPolicy:
  http:
    retryOn: [503]
    numRetries: 3
    perRetryTimeout: 2s
    retryableStatusCodes: [503]
Note: Retry hedging is useful in environments where transient failures are common, such as microservices under high load.

12. Troubleshooting Retry Behavior

Example: Configuring Retry Policy with Debug Logging

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example-service
  http:
  - route:
    - destination:
        host: example-service
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: "5xx,connect-failure"
Field Description Notes
retries.attempts Number of retry attempts Default is 0 (no retries)
retries.perTryTimeout Timeout for each retry attempt Must be <= global timeout
retries.retryOn Conditions to trigger retries Comma-separated list of conditions
Note: Retry conditions include 5xx, connect-failure, gateway-error, and reset.

Common Retry Conditions

  • 5xx: Server-side errors (500, 503, etc.)
  • connect-failure: Connection timeouts or failures
  • gateway-error: Gateway or upstream errors
  • reset: Connection resets by the client or server
Client --> Service Mesh --> Upstream Service
         |
         |--- Retry Logic:
         |    - If 5xx error, retry up to 3 times
         |    - Each retry has 2s timeout

Was this article helpful?

Related Cheat Sheets