Skip to main content
Mapagam

Implementing Canary Deployments

1. Understanding Canary Deployment Pattern

The canary deployment pattern is a release strategy that gradually rolls out a new version of an application to a small subset of users or infrastructure before fully deploying it. This approach minimizes risk by allowing teams to observe the behavior of the new version in production with a controlled audience.

Aspect Description Benefits
Gradual Rollout Introduces new version to a small percentage of traffic (e.g., 5%) Reduces risk of widespread issues
Traffic Splitting Uses service mesh to control traffic distribution between versions Enables A/B testing and feature toggling
Monitoring Tracks metrics like latency, error rates, and throughput during rollout Provides early detection of regressions or performance issues
Rollback Capability Canary can be quickly reverted if problems arise Ensures system stability and user experience

Example: Traffic Splitting in Istio

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 90
    - destination:
        host: reviews
        subset: v2
      weight: 10
Note: The v2 subset represents the new version being tested with 10% of traffic.

Key Concepts

  • Traffic Management: Service mesh enables fine-grained control over traffic routing.
  • Version Control: Different versions of services can coexist and be gradually promoted.
  • Metrics & Observability: Real-time monitoring is crucial for making informed decisions during deployment.

2. Configuring Traffic Splitting

Property Description Example
traffic.split Defines the percentage of traffic to route to each version { "v1": 90, "v2": 10 }
strategy Algorithm used to split traffic (e.g., weight, header-based) "weight"

Example: Weight-based Traffic Splitting

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: my-service
spec:
  hosts:
  - my-service
  http:
  - route:
    - destination:
        host: my-service-v1
        port:
          number: 80
      weight: 90
    - destination:
        host: my-service-v2
        port:
          number: 80
      weight: 10
Note: Traffic splitting can be combined with other routing rules like header or cookie-based routing.

3. Setting Up Canary Subsets

Property Description Example
subset.name Name of the canary subset canary-v2
subset.labels Labels to identify the canary version {"version": "v2"}
subset.trafficWeight Percentage of traffic routed to this subset 10

Example: Define Canary Subset in Istio

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp-virtual-service
spec:
  hosts:
  - myapp
  http:
  - route:
    - destination:
        host: myapp
        subset: stable
      weight: 90
    - destination:
        host: myapp
        subset: canary
      weight: 10

Example: Define Destination Rule with Subsets

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: myapp-destination-rule
spec:
  host: myapp
  subsets:
  - name: stable
    labels:
      version: v1
  - name: canary
    labels:
      version: v2

4. Implementing Gradual Traffic Shift

Method Description Example Notes
weight-based routing Gradually shift traffic by assigning weights to different versions of a service 
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: my-app
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: my-service-vs
spec:
  hosts:
    - my-service
  http:
    - route:
        - destination:
            host: my-service-v1
            port:
              number: 80
          weight: 90
        - destination:
            host: my-service-v2
            port:
              number: 80
          weight: 10
 Use weights to control traffic distribution between versions
percentage-based routing Route a percentage of traffic to a new version for gradual rollout 
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: my-service-vs
spec:
  hosts:
    - my-service
  http:
    - route:
        - destination:
            host: my-service-v1
            port:
              number: 80
          weight: 75
        - destination:
            host: my-service-v2
            port:
              number: 80
          weight: 25
 Start with a small percentage and increase over time
traffic shifting with delay Introduce delays or timeouts to control traffic flow 
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: my-service-vs
spec:
  hosts:
  - my-service
  http:
    - route:
        - destination:
            host: my-service-v1
            port:
              number: 80
          weight: 95
        - destination:
            host: my-service-v2
            port:
              number: 80
          weight: 5
      timeout:
        duration: 5s
 Use timeouts to prevent cascading failures during shift

Example: Gradual Rollout with Weight Adjustment

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api-virtual-service
spec:
  hosts:
    - api-service
  http:
    - route:
        - destination:
            host: api-service-v1
            port:
              number: 80
          weight: 90
        - destination:
            host: api-service-v2
            port:
              number: 80
          weight: 10
Note: Gradual traffic shift allows for safer rollouts by minimizing impact on users during version upgrades.
Warning: Ensure that monitoring and alerting are in place to detect anomalies during traffic shifts.

5. Configuring Header-Based Canary Routing

Property Value Description
header.name version Name of the HTTP header used for routing
header.value v2 Value of the header that triggers canary traffic
weight 10% Percentage of traffic to route to the canary version

Example: Header-Based Canary Rule

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example-service
  http:
  - route:
    - destination:
        host: example-service-primary
        port:
          number: 80
      weight: 90
    - destination:
        host: example-service-canary
        port:
          number: 80
      headers:
        request:
          add:
            version: v2
      weight: 10
Note: The version header must be present in the incoming request for the canary routing to activate.

6. Implementing Automated Canary Analysis

Example: Configuring Prometheus for Canary Metrics

scrape_configs:
  - job_name: 'canary-app'
    metrics_path: /metrics
    static_configs:
      - targets: ['canary-app:8080']
Field Description Values Notes
job_name Unique identifier for the job canary-app Used for labeling metrics in Prometheus
metrics_path Path to scrape metrics from /metrics Must match the endpoint exposed by the service
static_configs Static list of targets to scrape ['canary-app:8080'] Can include multiple services or versions
Note: Ensure that the service exposes a metrics endpoint compatible with Prometheus for accurate analysis.

Example: Setting Up Alerting Rules

groups:
  - name: canary-alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{job="canary-app",status="5xx"}[5m]) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected in canary deployment"
Field Description Values Notes
alert Name of the alert rule HighErrorRate Used to identify specific issues in canary analysis
expr Prometheus query expression rate(http_requests_total{job="canary-app",status="5xx"}[5m]) > 0.05 Triggers if error rate exceeds 5% over 5 minutes
for Duration to wait before firing the alert 2m Prevents flapping alerts due to temporary spikes
severity Severity level of the alert critical Used for prioritizing incident response

Key Considerations for Automated Canary Analysis

  • Define clear success criteria (e.g., latency, error rate thresholds)
  • Use metrics that align with business KPIs (e.g., conversion rate, user satisfaction)
  • Implement alerting rules to detect anomalies quickly
  • Ensure rollback mechanisms are in place if analysis fails

7. Managing Canary Rollback

Rollback Strategy Description Notes
Automatic Rollback Revert to previous version if failure thresholds are met Configurable via failureThreshold in rollout policies
Manual Rollback Requires operator intervention to revert to stable version Used in high-risk environments or compliance-sensitive systems
Gradual Rollback Slowly shift traffic back to stable version while monitoring Reduces risk of cascading failures during rollback

Example: Configuring Automatic Rollback

apiVersion: flagger.app/v1
kind: Canary
metadata:
  name: app-canary
spec:
  service:
    port: 80
  analysis:
    interval: 1m
    threshold: 10
    failureThreshold: 3
    metrics:
    - name: request-success-rate
      threshold: 99
      interval: 1m
    - name: request-duration
      threshold: 500
      interval: 1m
Note: The failureThreshold parameter defines how many failed iterations before triggering an automatic rollback. Adjust based on your system's tolerance for instability.

Canary Rollback Process

  1. Monitor canary deployment for failure metrics (e.g., error rate, latency)
  2. If failureThreshold is exceeded, initiate rollback
  3. Redirect all traffic back to the stable version
  4. Log rollback event and notify operators if configured

Rollback Strategies Comparison

  • Automatic: Fast but less controlled; suitable for low-risk services
  • Manual: More control but slower; ideal for mission-critical systems
  • Gradual: Balanced approach; reduces impact of rollback on users

8. Configuring Canary Duration

Property Values Description
canary.duration 30s, 5m, 1h Duration for which the canary deployment remains active before promoting or rolling back
canary.stepDuration 5s, 1m Interval between traffic shifts during the canary process

Example: Configuring Canary Duration in Istio

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: example-vs
spec:
  hosts:
  - example.com
  http:
  - route:
    - destination:
        host: example-primary
        subset: v1
      weight: 90
    - destination:
        host: example-canary
        subset: v2
      weight: 10
    trafficPolicy:
      connectionPool:
        http:
          idleTimeout: 30s
      outlierDetection:
        baseEjectionTime: 15m
    retries:
      attempts: 3
      perTryTimeout: 2s
Note: The canary.duration setting determines how long the canary version is exposed to traffic before being promoted or discarded.

9. Monitoring Canary Metrics

Metric Description Collection Method Notes
request_count Total number of requests handled by the canary deployment Service mesh telemetry (e.g., Prometheus) Use for tracking traffic distribution
error_rate Percentage of failed requests in canary Service mesh metrics High error rates indicate issues in canary
latency_p95 95th percentile request latency Service mesh tracing Monitor for performance regressions
traffic_percentage Percentage of traffic routed to canary Service mesh routing rules Adjust based on monitoring results

Example: Configuring Prometheus for Canary Metrics

scrape_configs:
  - job_name: 'canary-service'
    static_configs:
      - targets: ['canary-service:8080']
Note: Ensure metrics are collected with sufficient granularity to distinguish between stable and unstable canary versions.
Warning: Avoid over-monitoring which can introduce latency and resource overhead in the service mesh.

10. Promoting Canary to Production

Step Description Action Notes
1 Verify canary stability Check metrics and logs for any anomalies or errors in the canary deployment. Ensure no performance degradation or error rates are increasing.
2 Adjust traffic distribution Increase the percentage of traffic routed to the canary version. Gradually shift traffic to the new version using service mesh rules (e.g., Istio's RouteRules).
3 Monitor production impact Continuously monitor the production environment for any side effects. Use observability tools to track latency, error rates, and resource usage.
4 Final traffic shift Redirect 100% of traffic to the canary version. This step promotes the canary to full production status.
5 Validate production behavior Confirm that the canary version behaves as expected in full production. Run end-to-end tests or user acceptance tests if applicable.

Example: Promoting Canary with Istio

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example-service
  http:
  - route:
    - destination:
        host: example-service-canary
        subset: canary
      weight: 0
    - destination:
        host: example-service-stable
        subset: stable
      weight: 100
Note: The traffic shift should be gradual to avoid overwhelming the canary or causing production instability.
Warning: Ensure rollback plans are in place before fully promoting the canary to production.

11. Implementing Canary with Flagger

Example: Flagger Deployment Configuration

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: app-canary
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app-deployment
  progressDeadlineSeconds: 600
  service:
    port: 80
    targetPort: 8080
  analysis:
    interval: 1m
    threshold: 1
    range: 10
    metrics:
    - name: request-success-rate
      threshold: 99
      interval: 1m
    - name: request-latency-p95
      threshold: 500
      interval: 1m
Field Description Values
apiVersion API version of the Flagger resource flagger.app/v1beta1
kind Kind of the Flagger resource Canary
metadata.name Name of the canary deployment String
metadata.namespace Namespace where the canary is deployed String
spec.targetRef Reference to the target deployment Object with apiVersion, kind, name
spec.progressDeadlineSeconds Timeout for the canary deployment to progress Integer (seconds)
spec.service.port Port of the service to be used for canary analysis Integer
spec.service.targetPort Target port of the service to be used for canary analysis Integer or string
spec.analysis.interval Interval between analysis steps Duration (e.g., 1m)
spec.analysis.threshold Threshold for canary success or failure Integer
spec.analysis.range Number of steps to analyze before making a decision Integer
Note: Flagger uses istioctl to inject sidecars into pods and manages traffic routing via Istio's VirtualService and DestinationRule.

Example: Flagger Metrics Configuration

spec:
  analysis:
    interval: 1m
    threshold: 1
    range: 10
    metrics:
    - name: request-success-rate
      threshold: 99
      interval: 1m
    - name: request-latency-p95
      threshold: 500
      interval: 1m

Canary Deployment Steps with Flagger

  • Flagger injects a canary deployment alongside the original primary deployment.
  • Flagger gradually increases traffic to the canary using Istio's VirtualService.
  • Flagger analyzes metrics to determine if the canary is successful or not.
  • If successful, Flagger promotes the canary to the primary deployment.
  • If failed, Flagger rolls back to the previous stable version.

12. Managing Canary Deployment Strategies

Strategy Description Use Case Notes
Percentage-based Gradually increase traffic to the new version based on a percentage. Low-risk rollouts where gradual traffic shift is preferred. Can be combined with metrics for automated scaling.
Label-based Route traffic based on labels or tags assigned to services. Advanced routing where specific user segments are targeted. Requires service mesh to support label-based routing.
Header-based Route traffic based on HTTP headers (e.g., user-agent, region). Targeting specific client types or environments. Useful for A/B testing or feature flags.
Method-based Route traffic based on HTTP method (GET, POST, etc.). Testing specific API endpoints or methods. Can be used to isolate traffic for performance testing.

Example: Percentage-based Canary Deployment

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example-service
  http:
  - route:
    - destination:
        host: example-service-v1
        port:
          number: 80
      weight: 90
    - destination:
        host: example-service-v2
        port:
          number: 80
      weight: 10
Note: The weights in the VirtualService define how much traffic is routed to each version. Adjusting these values allows for fine-grained control over the deployment rollout.

Was this article helpful?

Related Cheat Sheets