Implementing Blue-Green Deployments

1. Understanding Blue-Green Deployment Pattern

Aspect	Description	Notes
Definition	A blue-green deployment strategy involves running two identical production environments (Blue and Green).	Only one environment is active at a time, handling all traffic.
Active Environment	One environment (e.g., Blue) is live and serving traffic.	The other (Green) is idle but fully provisioned.
Deployment Process	Deploy new version to the inactive environment (Green).	Run tests and validate before switching traffic.
Traffic Switching	Switch all traffic from Blue to Green once validated.	Can be done instantly or gradually (canary).
Rollback Strategy	If issues occur, switch traffic back to Blue.	Revert without downtime or data loss.

Example: Traffic Switching in Service Mesh

# Example using Istio for traffic switching
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example-service
  http:
  - route:
    - destination:
        host: example-service-blue
        port:
          number: 80

Note: Service Mesh allows fine-grained control over traffic routing, making it ideal for implementing blue-green deployments with minimal downtime.

2. Configuring Blue-Green Subsets

Property	Description	Values	Notes
subset.name	Name of the subset	`blue`, `green`	Must be unique within the service
subset.labels	Labels to match traffic	`{version: "v1", env: "prod"}`	Used for routing rules
subset.trafficWeight	Percentage of traffic to this subset	`0-100`	Sum of all subsets must equal 100

Example: Service Configuration

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-service
spec:
  hosts:
  - my-service
  http:
  - route:
    - destination:
        host: my-service
        subset: blue
      weight: 90
    - destination:
        host: my-service
        subset: green
      weight: 10

Note: Traffic weights are applied per request, not globally. Ensure all subsets have a defined weight.

3. Implementing Instant Traffic Switch

Method	Description	Notes
service-mesh traffic switch	Instantly shift all traffic from one service version to another	Requires service mesh control plane support for dynamic routing
weight-based routing	Route traffic based on weights (e.g., 90% to v1, 10% to v2)	Allows gradual traffic shift; can be adjusted in real-time
canary release	Gradually shift traffic to new version using weighted rules	Can be combined with health checks for automated rollbacks

Example: Instant Traffic Switch Using Weighted Routes

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example-service
  http:
  - route:
    - destination:
        host: example-service-v1
        port:
          number: 80
      weight: 0
    - destination:
        host: example-service-v2
        port:
          number: 80
      weight: 100

Note: The weight: 0 for v1 and weight: 100 for v2 indicates that all traffic is routed to v2. To switch instantly, update the weights accordingly.

Warning: Ensure that all services involved in the traffic switch are compatible and can handle the sudden load. Use health checks to verify service readiness before switching.

4. Setting Up Version Labels

Label Key	Example Value	Description
app.kubernetes.io/version	`v1.2.3`	Version of the application
deployment.environment	`blue`	Environment identifier for deployment
release	`stable-v2`	Release version or tag

Example: Applying Labels to Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-blue
  labels:
    app.kubernetes.io/version: "v1.2.3"
    deployment.environment: "blue"
    release: "stable-v2"
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: my-app
  template:
    metadata:
      labels:
        app.kubernetes.io/version: "v1.2.3"
        deployment.environment: "blue"
    spec:
      containers:
      - name: app
        image: my-app:v1.2.3

Note: Ensure labels are consistently applied across all resources (Deployments, Services, ConfigMaps) to maintain version alignment in blue-green deployments.

5. Configuring Virtual Service Routes

Property	Description	Values	Notes
name	Unique identifier for the route	`string`	Required field
match	Conditions to match incoming requests	`object`	Supports headers, path, method, and query parameters
route	Destination service and weight distribution	`array`	Each entry includes `destination` and `weight`

Example: Basic Route Configuration

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-route
spec:
  hosts:
  - example-service
  http:
  - match:
    - uri:
        prefix: /api
    route:
    - destination:
        host: example-service-v1
        port:
          number: 80
      weight: 90
    - destination:
        host: example-service-v2
        port:
          number: 80
      weight: 10

Note: The weight values must sum to 100 for traffic distribution.

Warning: Ensure that all destination services are properly configured and accessible.

6. Implementing Pre-Production Testing

Example: Testing with Service Mesh

# Example service configuration with pre-production testing
apiVersion: v1
kind: Service
metadata:
  name: frontend-service
  labels:
    app: frontend
spec:
  selector:
    app: frontend
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: frontend-virtual-service
spec:
  hosts:
  - frontend-service
  http:
  - route:
    - destination:
        host: frontend-service
        subset: production
    - destination:
        host: frontend-service
        subset: testing
      weight: 10 # 10% traffic to testing

Testing Type	Description	Implementation	Status
Canary Testing	Gradual rollout of new version to a subset of users	Use traffic splitting in service mesh	NEW
A/B Testing	Comparing two versions of a service	Route traffic between versions based on criteria	BETA
Shadow Testing	Running new version in parallel without affecting traffic	Use mirroring features in service mesh	EXPERIMENTAL

Note: Pre-production testing in service mesh allows for safer rollouts by gradually increasing traffic to new versions.

Warning: Ensure proper monitoring and alerting before implementing pre-production tests to avoid unexpected traffic shifts.

Pre-Production Testing Strategy

Start with 5% traffic to new version
Gradually increase weight over time
Monitor metrics for performance degradation
Rollback if issues detected

7. Managing Database Migrations

Migration Type	Description	Tool/Method	Notes
Schema Changes	Modifications to database schema structure	`liquibase`, `flyway`	Supports version control and rollback capabilities
Data Migrations	Transformation or transfer of data between versions	`db-migrate`, `sequelize-cli`	Requires careful handling to avoid data loss
Rollback Strategy	Reverting to previous migration state	`rollback` commands	Must be planned and tested in staging

Example: Liquibase Migration Script

<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.8.xsd">
    <changeSet id="1" author="dev">
        <createTable tableName="users">
            <column name="id" type="int" autoIncrement="true">
                <constraints primaryKey="true"/>
            </column>
            <column name="name" type="varchar(255)"/>
        </createTable>
    </changeSet>
</databaseChangeLog>

Note: Ensure all database migrations are idempotent and safe to run in production environments.

Warning: Always test migrations on a copy of production data before applying to live systems.

Best Practices for Migration Management

Use version control for all migration scripts
Implement automated testing of migrations in staging
Plan rollback procedures for each migration
Validate data integrity after each migration step

8. Implementing Rollback Strategy

Example: Rollback Configuration in Service Mesh

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: app-virtual-service
spec:
  hosts:
  - app.example.com
  http:
  - route:
    - destination:
        host: app-v1
        port:
          number: 80
    retries:
      attempts: 3
      perTryTimeout: 2s
    timeout: 5s
    fault:
      abort:
        percent: 10
        httpStatus: 503

Field	Description	Values	Notes
retries.attempts	Number of retry attempts	`3`	Default retry count before rollback
retries.perTryTimeout	Timeout per retry attempt	`2s`	Time to wait for each retry
timeout	Overall request timeout	`5s`	Used to detect failures for rollback trigger
fault.abort.percent	Percentage of requests to abort	`10`	Simulates failure for rollback testing
fault.abort.httpStatus	HTTP status code to simulate	`503`	Service Unavailable for rollback trigger

Note: Rollback logic should be implemented in conjunction with circuit breaker patterns to prevent cascading failures during rollback.

Rollback Strategy Components

Health Checks - Monitor service health to detect failures
Traffic Shift - Gradually shift traffic to stable version
Timeouts & Retries - Define thresholds for failure detection
Circuit Breakers - Prevent further traffic to failing services

9. Configuring Health Checks

Check Type	Description	Configuration	Notes
Readiness Probe	Indicates whether the service is ready to handle requests	`{ "readinessProbe": { "httpGet": { "path": "/health/readiness", "port": 8080 }, "initialDelaySeconds": 10, "periodSeconds": 5 } }`	Used to prevent traffic to a service that isn't fully initialized
Liveness Probe	Indicates whether the service is alive and running correctly	`{ "livenessProbe": { "httpGet": { "path": "/health/liveness", "port": 8080 }, "initialDelaySeconds": 30, "periodSeconds": 10 } }`	Triggers restart if the service becomes unresponsive

Example: Health Check Configuration in Kubernetes

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
  - name: example-container
    image: example-image
    readinessProbe:
      httpGet:
        path: /health/readiness
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
    livenessProbe:
      httpGet:
        path: /health/liveness
        port: 8080
      initialDelaySeconds: 30
      periodSeconds: 10

Note: Health checks are critical for ensuring service stability during blue-green deployments. Misconfigured probes can lead to traffic being routed to unhealthy instances.

10. Monitoring Deployment Status

Aspect	Metric	Description	Tools
Health Checks	`readinessProbe`	Indicates if the service is ready to handle requests	Kubernetes, Istio
Liveness Checks	`livenessProbe`	Detects if the service is stuck or unresponsive	Kubernetes, Istio
Traffic Shift	`trafficWeight`	Percentage of traffic routed to new version	Istio, Linkerd
Error Rate	`errorRate`	Percentage of failed requests during deployment	Prometheus, Grafana
Latency	`responseTime`	Average response time for requests	Prometheus, Grafana

Example: Monitoring with Prometheus

groups:
- name: service-mesh
  rules:
  - alert: HighErrorRate
    expr: rate(http_requests_total{status="5xx"}[5m]) > 0.01
    for: 2m
    labels:
      severity: page
    annotations:
      summary: "High error rate detected"

Note: Use readinessProbe to ensure services are not receiving traffic until they're fully initialized.

Warning: Monitor errorRate and latency closely during traffic shift to detect issues early.

11. Managing Blue-Green with Service Mesh

Feature	Description	Implementation	Notes
Traffic Splitting	Gradually shift traffic between blue and green versions	`destinationRule` with `trafficPolicy`	Can be done via `VirtualService` or `DestinationRule`
Version Routing	Route traffic to specific versions of services	`VirtualService` with `route`	Supports weighted routing and header-based routing
Health Checks	Monitor service health for failover	`healthCheck` in `DestinationRule`	Can be configured for HTTP, TCP, or gRPC

Example: Traffic Splitting Configuration

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: app-virtual-service
spec:
  hosts:
  - app
  http:
  - route:
    - destination:
        host: app-blue
        subset: v1
      weight: 90
    - destination:
        host: app-green
        subset: v2
      weight: 10

Example: Health Check Configuration

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: app-destination-rule
spec:
  host: app
  trafficPolicy:
    healthCheck:
      interval: 5s
      timeout: 2s
      unhealthyThreshold: 3

Note: Blue-green deployments in service mesh require careful configuration of VirtualService and DestinationRule to ensure traffic is routed correctly during deployment.

Warning: Ensure that both versions of the service are compatible and can handle the same traffic load before switching.

12. Troubleshooting Blue-Green Issues

Common Blue-Green Deployment Problems

Service downtime during switchover
Stale traffic routing to old version
Configuration drift between environments
Resource contention during deployment
Rollback complications when switching back

Issue	Cause	Solution
Traffic not switching	Service mesh routing rules still point to old version	Verify `VirtualService` and `DestinationRule` configurations; ensure traffic policies are updated
Old version still receiving traffic	Service mesh not properly draining old version	Use `istioctl` to verify endpoint status; check for lingering connections
Application errors post-switch	Missing or incorrect service dependencies in new version	Validate service dependencies and configurations in new environment; run integration tests
Rollback fails	Deployment artifacts not properly stored or versioned	Ensure rollback scripts and configurations are backed up; maintain version tags

Example: Checking Service Mesh Traffic Rules

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: app-virtual-service
spec:
  hosts:
  - app.example.com
  http:
  - route:
    - destination:
        host: app-new.example.com
        port:
          number: 80

Note: Use istioctl proxy-config to inspect active routes and verify traffic routing in real-time.

Troubleshooting Workflow

Verify traffic routing rules in service mesh
Check for active connections to old version
Validate new version's service dependencies
Run integration tests on new version
Perform manual traffic switch if needed