Implementing Fault Injection

1. Injecting HTTP Delays

Property	Value	Description
delay.fixedDelay	`500ms`	Fixed delay to inject before forwarding the request
delay.percentage	`100`	Percentage of requests to apply the delay to

Example: Injecting HTTP Delay

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: delay-example
spec:
  hosts:
  - api-service
  http:
  - route:
    - destination:
        host: api-service
    fault:
      delay:
        fixedDelay: 500ms
        percentage:
          value: 100

2. Injecting HTTP Aborts

Property	Value	Description
abort.http.status	`503`	HTTP status code to return for aborted requests
abort.http.percentage	`10`	Percentage of requests to abort (0-100)

Example: Injecting HTTP Aborts

apiVersion: networking.istio.io/v1beta1
kind: FaultInjection
metadata:
  name: http-abort-fault
spec:
  destination:
    host: backend-service
  http:
  - abort:
      httpStatus: 503
      percentage:
        value: 10

3. Configuring Fault Injection Percentage

Property	Values	Description
fault.injection.percentage	`0-100`	Percentage of requests to inject faults into. 0 = no faults, 100 = all requests.

Example: Configuring 20% Fault Injection

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 8080
  faultInjection:
    percentage: 20

Note: The fault injection percentage is applied at the service level and can be overridden per route or operation.

4. Implementing Header-Based Faults

Header Name	Values	Description	Usage
x-fault-delay	`500ms`	Introduces a delay in milliseconds before the request is processed.	Simulates network latency or service unavailability.
x-fault-abort	`503`	Aborts the request with a specified HTTP status code.	Simulates service failure or overload.
x-fault-rate	`0.1`	Defines the rate at which faults are injected (0.0 to 1.0).	Controls the probability of fault injection.

Example: Injecting a Delay Based on Header

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example-service
  http:
  - route:
    - destination:
        host: example-service
    fault:
      delay:
        percentage:
          value: 100
        fixedDelay: 500ms

Example: Aborting Requests with Custom Header

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example-service
  http:
  - route:
    - destination:
        host: example-service
    fault:
      abort:
        percentage:
          value: 50
        httpStatus: 503

Note: Header-based fault injection is useful for simulating real-world conditions in a controlled environment, such as network delays or service outages.

5. Setting Delay Duration

Property	Values	Description
delay.fixedDelay	`500ms`	Fixed delay duration to inject before forwarding requests
delay.percentage	`100`	Percentage of requests to apply the delay to
delay.jitter	`100ms`	Random variation added to the fixed delay (optional)

Example: Configuring Fixed Delay

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-delay
spec:
  hosts:
  - backend
  http:
  - route:
    - destination:
        host: backend
    fault:
      delay:
        fixedDelay: 500ms
        percentage:
          value: 100

Example: Configuring Delay with Jitter

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-delay-jitter
spec:
  hosts:
  - backend
  http:
  - route:
    - destination:
        host: backend
    fault:
      delay:
        fixedDelay: 300ms
        jitter: 100ms
        percentage:
          value: 50

Note: Delay durations are specified in ns, us, ms, or s. The jitter parameter adds randomness to the delay, simulating real-world network variability.

6. Configuring Abort Status Codes

Property	Value	Description
abort.httpStatus	`503`	HTTP status code to return when aborting a request
abort.percentage	`50`	Percentage of requests to abort (0-100)

Example: Configuring Abort Status Codes

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example-service
  http:
  - route:
    - destination:
        host: example-service
    fault:
      abort:
        httpStatus: 503
        percentage: 50

7. Testing Service Resilience

Example: Injecting Delay to Test Resilience

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: frontend-vs
spec:
  hosts:
  - frontend
  http:
  - route:
    - destination:
        host: frontend
    fault:
      delay:
        percentage:
          value: 100
        fixedDelay: 5s

Property	Description	Values	Notes
fault	Defines fault injection configuration	`object`	Root object for fault injection rules
delay	Configures delay injection	`object`	Applies delays to requests
percentage.value	Percentage of requests to apply delay	`0-100`	Integer value between 0 and 100
fixedDelay	Fixed delay duration	`duration`	Format: `5s`, `100ms`

Note: Delay injection helps test how services handle latency and timeout scenarios.

Example: Injecting Abort to Test Error Handling

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: backend-vs
spec:
  hosts:
  - backend
  http:
  - route:
    - destination:
        host: backend
    fault:
      abort:
        percentage:
          value: 50
        httpStatus: 503

Property	Description	Values	Notes
abort	Configures abort injection	`object`	Applies HTTP status code errors to requests
percentage.value	Percentage of requests to abort	`0-100`	Integer value between 0 and 100
httpStatus	HTTP status code to return	`4xx/5xx`	Valid HTTP status code (e.g., `503`)

Warning: Fault injection should only be used in testing environments to avoid impacting production traffic.

8. Implementing gRPC Fault Injection

Field	Description	Example
delay	Specifies the delay to inject before forwarding the request	`"fixedDelay": "50ms"`
abort	Specifies the percentage of requests to abort	`"percentage": 10, "httpStatus": 503`
percentage	Percentage of requests to apply fault injection to	`"percentage": 50`

Example: gRPC Delay Fault Injection

apiVersion: networking.istio.io/v1alpha3
kind: FaultInjection
metadata:
  name: grpc-delay-fault
spec:
  destination:
    host: grpc-service
  delay:
    fixedDelay: 50ms
    percentage:
      value: 100

Example: gRPC Abort Fault Injection

apiVersion: networking.istio.io/v1alpha3
kind: FaultInjection
metadata:
  name: grpc-abort-fault
spec:
  destination:
    host: grpc-service
  abort:
    percentage:
      value: 50
    httpStatus: 503

Note: gRPC fault injection is supported in Istio through the FaultInjection CRD and requires proper configuration of traffic policies.

9. Managing Fault Injection Rules

Rule Property	Description	Example Value	Notes
source	Specifies the source service or namespace for applying fault rules	`frontend`	Can be a service name or namespace pattern
destination	Specifies the destination service or namespace to inject faults into	`backend`	Can be a service name or namespace pattern
percentage	Percentage of requests to apply fault injection to	`50`	Value between 0 and 100
faultType	Type of fault to inject (e.g., delay, abort)	`delay`	Supported values: `delay`, `abort`
delayDuration	Duration of delay in milliseconds (only for `delay` fault)	`1000`	Applies to `delay` fault type only
abortCode	HTTP status code to abort with (only for `abort` fault)	`503`	Applies to `abort` fault type only

Example: Configuring Fault Injection Rule

apiVersion: networking.istio.io/v1beta1
kind: FaultInjectionPolicy
metadata:
  name: fault-injection-rule
spec:
  source:
    name: frontend
  destination:
    name: backend
  percentage: 50
  faultType: delay
  delayDuration: 1000

Example: Fault Injection with Abort

apiVersion: networking.istio.io/v1beta1
kind: FaultInjectionPolicy
metadata:
  name: abort-fault-rule
spec:
  source:
    name: frontend
  destination:
    name: backend
  percentage: 30
  faultType: abort
  abortCode: 503

Note: Fault injection rules are applied at the service mesh level and can be managed via Kubernetes CRDs or through the service mesh control plane.

10. Debugging with Fault Injection

Property	Description	Values	Notes
faultType	Type of fault to inject	`delay`, `abort`	Controls the kind of fault to simulate
delay.duration	Duration of delay in milliseconds	`100ms`, `500ms`	Applies to `delay` faults
abort.httpStatus	HTTP status code to return	`500`, `404`	Applies to `abort` faults
abort.percentage	Percentage of requests to abort	`50%`, `100%`	Controls fault injection rate

Example: Injecting a Delay Fault

apiVersion: networking.istio.io/v1alpha3
kind: FaultInjection
metadata:
  name: delay-fault
spec:
  destination:
    host: service-a.default.svc.cluster.local
  delay:
    percentage:
      value: 100
    fixedDelay: 200ms

Example: Injecting an Abort Fault

apiVersion: networking.istio.io/v1alpha3
kind: FaultInjection
metadata:
  name: abort-fault
spec:
  destination:
    host: service-b.default.svc.cluster.local
  abort:
    percentage:
      value: 50
    httpStatus: 503

Note: Fault injection is a powerful debugging tool in service meshes like Istio. It allows you to simulate various failure conditions to test the resilience of your services.

Warning: Fault injection should only be used in non-production environments to avoid impacting real user traffic.

11. Implementing Targeted Fault Injection

Property	Description	Example
target	Specifies the target service or endpoint for fault injection	`target: "user-service"`
faultType	Type of fault to inject (e.g., delay, abort)	`faultType: "delay"`
percentage	Percentage of requests to apply the fault (0-100)	`percentage: 50`
delay.duration	Duration of delay in milliseconds (only for delay faults)	`delay.duration: 1000`
abort.httpStatus	HTTP status code to abort with (only for abort faults)	`abort.httpStatus: 503`

Example: Targeted Delay Fault Injection

faults:
  - target: "user-service"
    faultType: "delay"
    percentage: 30
    delay:
      duration: 2000

Example: Targeted Abort Fault Injection

faults:
  - target: "payment-service"
    faultType: "abort"
    percentage: 10
    abort:
      httpStatus: 500

Note: Fault injection should be enabled only in testing or staging environments to avoid impacting production traffic.

12. Configuring Fault Injection Timeouts

Property	Value	Description
fault.timeout	`5s`	Timeout duration for fault injection operations
fault.delay	`1s`	Delay before injecting fault
fault.duration	`10s`	Duration for which fault is active

Example: Configuring Fault Injection Timeout in YAML

fault:
  timeout: 5s
  delay: 1s
  duration: 10s