Skip to main content
Mapagam

Implementing Progressive Delivery

1. Understanding Progressive Delivery Patterns

Pattern Description Use Case Benefits
Canary Release Gradually roll out a new version to a small subset of users before full deployment Reducing risk of breaking changes in production Minimizes impact of failures, allows A/B testing
Blue-Green Deployment Run two identical production environments, switching traffic between them Quick rollback capability in case of issues Zero-downtime deployments, easy rollback
A/B Testing Split traffic between different versions to compare performance or user experience Evaluating feature effectiveness before full rollout Data-driven decision making, reduced risk of poor user experience
Feature Toggles Enable or disable features dynamically without redeploying code Gradual feature activation, easy rollback Flexible release strategy, reduced deployment frequency
Shadow Deployment Send production traffic to a copy of the system in parallel with live version Testing new features without affecting users Safe experimentation, performance validation

Example: Canary Release with Istio

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 90
    - destination:
        host: reviews
        subset: v2
      weight: 10
Note: The weights in the VirtualService define how much traffic is sent to each version of the service.

2. Integrating with Flagger

Feature Description Configuration Notes
flagger.name Name of the Flagger deployment flagger Default value used by Flagger
flagger.namespace Namespace where Flagger is deployed flagger-system Typically flagger-system
flagger.metrics.pushgateway Pushgateway endpoint for metrics http://pushgateway.flagger-system.svc.cluster.local:9091 Used for Prometheus metrics collection

Example: Flagger Deployment Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flagger
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flagger
  template:
    metadata:
      labels:
        app: flagger
    spec:
      containers:
      - name: flagger
        image: flagger/flagger:1.20.0
        args:
        - --configmap=flagger-config
        - --metrics.pushgateway=http://pushgateway.flagger-system.svc.cluster.local:9091

Example: Flagger Canary Configuration

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: app-canary
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app-deployment
  progressDeadlineSeconds: 600
  service:
    port: 80
    targetPort: 8080
  analysis:
    interval: 1m
    threshold: 1
    metrics:
    - name: request-success-rate
      threshold: 99
      interval: 1m
Note: Flagger uses canary resources to define progressive delivery strategies. It integrates with Istio or Linkerd for traffic routing and metrics collection.
Warning: Ensure that your service mesh supports canary resource definitions and has proper metrics exposure for Flagger to function correctly.

3. Integrating with Argo Rollouts

Feature Description Integration Details
Rollout Strategy Progressive delivery strategy using canary deployments Uses Argo Rollouts to manage rollout phases with traffic shifting
Traffic Management Gradual traffic shift to new versions Utilizes service mesh's traffic routing capabilities for canary releases
Metrics Collection Collects metrics for canary analysis Integrates with Prometheus and Grafana via service mesh telemetry

Example: Argo Rollout Configuration

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: example-rollout
spec:
  replicas: 5
  strategy:
    canary:
      steps:
        - setWeight: 20
        - pause: {}
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
    spec:
      containers:
        - name: example-container
          image: example-image:v1.0

Example: Service Mesh Integration

apiVersion: v1
kind: Service
metadata:
  name: example-service
  annotations:
    traffic.sidecar.istio.io/includeInboundPorts: "8080"
spec:
  selector:
    app: example-app
  ports:
    - port: 80
      targetPort: 8080
Note: Ensure that the service mesh is configured to support traffic shifting and canary analysis for seamless integration with Argo Rollouts.

4. Configuring Automated Progressive Rollouts

Configuration Option Description Values Notes
strategy Rollout strategy to use canary, blue-green, rolling Defines the deployment pattern
trafficWeight Percentage of traffic to route to new version 0-100 Used in canary and blue-green strategies
delay Time to wait before proceeding to next step duration Format: 5s, 1m, 2h

Example: Canary Rollout Configuration

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: example-rollout
spec:
  replicas: 5
  strategy:
    canary:
      steps:
        - setWeight: 20
          pause:
            duration: 30s
        - setWeight: 40
          pause:
            duration: 1m
        - setWeight: 60
          pause:
            duration: 2m
        - setWeight: 80
          pause:
            duration: 5m
        - setWeight: 100

Example: Blue-Green Deployment

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: blue-green-rollout
spec:
  replicas: 5
  strategy:
    blueGreen:
      activeService: active-svc
      previewService: preview-svc
Note: Automated rollouts require proper monitoring and rollback mechanisms to ensure service stability.
Warning: Misconfigured traffic weights can lead to service degradation or downtime.

5. Implementing Metric-Based Promotion

Metric Description Threshold Usage
request-success-rate Percentage of successful requests over a time window >= 95% Ensures service stability before promoting
latency-p95 95th percentile of request latency <= 200ms Monitors performance impact during promotion
error-rate Rate of failed requests per second <= 0.1% Tracks error trends to prevent promotion of unstable versions
throughput Number of requests processed per second >= 1000 req/s Validates service capacity during promotion

Example: Prometheus Query for Success Rate

sum(rate(http_requests_total{status="2xx"}[5m])) / sum(rate(http_requests_total[5m])) * 100

Example: Configuring Metric-Based Promotion in Istio

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: metric-based-promotion
spec:
  host: service-a
  trafficPolicy:
    outlierDetection:
      consecutiveErrors: 5
      interval: 10s
      baseEjectionTime: 30s
    connectionPool:
      http:
        maxRequestsPerConnection: 100
Note: Metric-based promotion requires a robust monitoring system like Prometheus or Grafana to collect and analyze service metrics in real time.
Warning: Setting thresholds too leniently can lead to unstable versions being promoted, while overly strict thresholds may prevent necessary updates from reaching production.

6. Managing Progressive Rollback

Feature Description Configuration Notes
rollback.strategy Defines the rollback approach for progressive delivery canary, blue-green, recreate Default is canary
rollback.threshold Failure threshold to trigger rollback 50%, 100% Can be set to percentage or count
rollback.delay Time to wait before initiating rollback 30s, 5m Useful for manual intervention window

Example: Rollback Configuration in Istio

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: example-vs
spec:
  hosts:
  - example.com
  http:
  - route:
    - destination:
        host: example-v1
    retries:
      attempts: 3
      perTryTimeout: 2s
    fault:
      abort:
        percent: 10
        code: 503
    rollback:
      strategy: canary
      threshold: 50%
      delay: 30s
Note: Rollback behavior depends on the service mesh implementation (e.g., Istio, Linkerd). Ensure that rollback policies are aligned with your deployment strategy.
Warning: Misconfigured rollback thresholds can lead to cascading failures or premature rollbacks during high-traffic periods.

7. Configuring Analysis Templates

Field Description Example
apiVersion API version of the analysis template flagger.app/v1beta1
kind Kind of the resource (must be AnalysisTemplate) AnalysisTemplate
metadata.name Name of the analysis template my-template
spec.metrics List of metric providers to query [{name: "request-success-rate", threshold: 95}]
spec.metrics.name Unique identifier for the metric request-success-rate
spec.metrics.threshold Threshold value for the metric (percentage) 95
spec.metrics.interval How often to query the metric (e.g., 1m) 1m

Example: Basic Analysis Template

apiVersion: flagger.app/v1beta1
kind: AnalysisTemplate
metadata:
  name: my-template
spec:
  metrics:
    - name: request-success-rate
      threshold: 95
      interval: 1m
Note: Analysis templates are used by Analysis and Canary resources to define how to evaluate metrics during progressive delivery.

8. Implementing Traffic Shifting Strategies

Strategy Description Use Case Implementation
Percentage-based shifting Routes a percentage of traffic to a new version. Gradual rollout of new features or versions. 
{
  "route": [
    {
      "destination": {
        "host": "service-v1",
        "subset": "v1"
      },
      "weight": 90
    },
    {
      "destination": {
        "host": "service-v2",
        "subset": "v2"
      },
      "weight": 10
    }
  ]
}
Canary release Gradually increases traffic to a new version based on success criteria. Testing new versions in production with minimal risk. 
{
  "route": [
    {
      "destination": {
        "host": "service-v1",
        "subset": "v1"
      },
      "weight": 95
    },
    {
      "destination": {
        "host": "service-v2",
        "subset": "v2"
      },
      "weight": 5
    }
  ],
  "trafficPolicy": {
    "loadBalancer": {
      "simple": "LEAST_CONN"
    }
  }
}
Blue-Green Deployment Routes all traffic to one version (blue) and switches to another (green) when ready. Zero-downtime deployments with full rollback capability. 
{
  "route": [
    {
      "destination": {
        "host": "service-blue",
        "subset": "v1"
      },
      "weight": 100
    }
  ]
}

Example: Percentage-based shifting with Istio

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 90
    - destination:
        host: reviews
        subset: v2
      weight: 10
Note: Traffic shifting strategies are often used in conjunction with circuit breakers and health checks to ensure stable delivery.

9. Managing Multi-Stage Deployments

Example: Staged Deployment Configuration

apiVersion: v1
kind: Service
metadata:
  name: frontend-service
spec:
  selector:
    app: frontend
  ports:
    - port: 80
      targetPort: 8080

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: frontend-virtual-service
spec:
  hosts:
    - frontend.example.com
  http:
    - route:
        - destination:
            host: frontend-service
            subset: v1
          weight: 90
        - destination:
            host: frontend-service
            subset: v2
          weight: 10
Stage Description Configuration Status
Canary Gradual rollout to a small percentage of traffic weight: 10 NEW
Blue-Green Full switch between two versions weight: 100 for one version Standard
A/B Testing Split traffic for user segmentation weight: 50/50 EXPERIMENTAL
Note: Multi-stage deployments help reduce risk by gradually introducing new features or versions to users.

Deployment Strategy Summary

  • Canary: Start with 10% of traffic to new version
  • Blue-Green: Switch all traffic after validation
  • A/B Testing: Split traffic based on user segments

10. Monitoring Progressive Delivery Metrics

Metric Description Typical Tools Notes
Request Success Rate Percentage of successful requests over time Prometheus, Grafana Indicates service reliability and health
Latency (P95/P99) 95th and 99th percentile response times Prometheus, Grafana, Datadog Helps identify performance bottlenecks
Error Rate Rate of failed requests (5xx, 4xx) Prometheus, Grafana Used to detect issues in traffic routing or service availability
Traffic Split Ratio Percentage of traffic sent to each version (e.g., canary, baseline) Istio, Linkerd Shows how much traffic is being shifted during progressive delivery
Throughput Number of requests per second handled by the service Prometheus, Grafana Monitors system capacity under load
Service Availability Percentage of time a service is available and responsive Prometheus, Grafana, Kiali Used in service mesh dashboards to track uptime

Example: Grafana Dashboard Setup for Progressive Delivery

{
  "dashboard": {
    "title": "Progressive Delivery Metrics",
    "panels": [
      {
        "type": "graph",
        "title": "Request Success Rate Over Time",
        "targets": [
          {
            "expr": "rate(http_requests_total{status=~\"2.*\"}[5m]) / rate(http_requests_total[5m]) * 100"
          }
        ]
      },
      {
        "type": "graph",
        "title": "Latency (P95)",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))"
          }
        ]
      }
    ]
  }
}
Note: Metrics should be collected at both the service mesh level and application level to get a full view of progressive delivery behavior.

11. Implementing Progressive Delivery Webhooks

Example: Webhook Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: progressive-delivery-webhooks
data:
  webhook-config.yaml: |
    webhook:
      url: "https://progressive-delivery.example.com/hooks"
      secret: "webhook-secret-key"
      events:
        - type: "deployment.success"
          payload: |
            {
              "deployment": "{{.Deployment.Name}}",
              "status": "success",
              "timestamp": "{{.Timestamp}}"
            }
Field Description Values Notes
webhook.url URL to send webhook events https://example.com/webhook Must be a valid HTTPS endpoint
webhook.secret Secret key for webhook authentication string Used to sign payloads for verification
webhook.events List of events to trigger webhook array Supports deployment, rollback, health check events
Note: Webhook payloads are signed with HMAC-SHA256 using the secret key.

Example: Webhook Payload

{
  "event": "deployment.success",
  "deployment": "frontend-app-v2",
  "status": "success",
  "timestamp": "2026-01-15T10:30:00Z",
  "metadata": {
    "version": "v2.1.0",
    "environment": "staging"
  }
}

12. Managing Progressive Delivery Policies

Policy Description Configuration Example Notes
trafficShifting Gradually shifts traffic between different versions of a service. 
{
  "trafficShifting": {
    "percentages": [
      { "version": "v1", "percentage": 90 },
      { "version": "v2", "percentage": 10 }
    ]
  }
}
 Used for canary deployments
faultInjection Intentionally introduces failures to test resilience. 
{
  "faultInjection": {
    "delay": { "percentage": 50, "fixedDelay": "100ms" },
    "abort": { "percentage": 20, "httpStatus": 503 }
  }
}
 Can be used for chaos engineering
rateLimiting Limits the number of requests a service can handle. 
{
  "rateLimiting": {
    "requestsPerSecond": 100,
    "burst": 20
  }
}
 Prevents service overload
circuitBreaker Prevents cascading failures by tripping when errors exceed a threshold. 
{
  "circuitBreaker": {
    "maxConnections": 100,
    "maxRequests": 50,
    "errorThresholdPercentage": 50
  }
}
 Improves resilience under high load

Example: Traffic Shifting Policy

{
  "name": "canary-deployment",
  "policyType": "trafficShifting",
  "config": {
    "percentages": [
      { "version": "v1", "percentage": 95 },
      { "version": "v2", "percentage": 5 }
    ]
  }
}

Example: Fault Injection Policy

{
  "name": "chaos-test",
  "policyType": "faultInjection",
  "config": {
    "delay": { "percentage": 100, "fixedDelay": "200ms" },
    "abort": { "percentage": 50, "httpStatus": 500 }
  }
}

Was this article helpful?

Related Cheat Sheets