Skip to main content

Implementing A/B Testing

1. Using Split Clients

The split_clients directive in Nginx allows you to split traffic between different versions of content or configurations based on a key, such as a user ID or IP address. This is useful for A/B testing.

Directive Description Example
split_clients Defines a variable that splits traffic based on a key. split_clients "${remote_addr}" $variant {
split_clients Specifies the percentage of traffic to assign to each variant. split_clients "${remote_addr}" $variant {

Example: Basic Split Client Configuration

http {
    split_clients "${remote_addr}" $variant {
        50%      "variant_a";
        50%      "variant_b";
    }

    server {
        listen 80;
        location / {
            if ($variant = "variant_a") {
                proxy_pass http://backend_a;
            }
            if ($variant = "variant_b") {
                proxy_pass http://backend_b;
            }
        }
    }
}
Note: The split_clients directive uses a consistent hashing algorithm to ensure that the same client always gets assigned to the same variant.

2. Distributing Traffic

Method Description Configuration Example Notes
ip_hash Hashes the client's IP address to determine which server to route to. upstream backend {
  ip_hash;
  server 192.168.1.10;
  server 192.168.1.11;
}
Ensures consistent routing for the same client IP
least_conn Selects the server with the least number of active connections. upstream backend {
  least_conn;
  server 192.168.1.10;
  server 192.168.1.11;
}
Good for balancing load across servers with varying capacities
random Randomly selects a server from the group. upstream backend {
  random;
  server 192.168.1.10;
  server 192.168.1.11;
}
Simple but not always optimal for consistent performance
weight Assigns weights to servers, affecting the probability of selection. upstream backend {
  server 192.168.1.10 weight=3;
  server 192.168.1.11 weight=1;
}
Useful for load balancing based on server capacity

Example: Using ip_hash for A/B Testing

upstream ab_test_backend {
    ip_hash;
    server 192.168.1.10:8080;
    server 192.168.1.11:8080;
}

location / {
    proxy_pass http://ab_test_backend;
}
Note: The ip_hash method ensures that users with the same IP address are consistently routed to the same server, which is essential for maintaining session consistency in A/B testing.
Directive Description Example
set Sets a variable with a value, often used for cookie handling. set $cookie_user_id $cookie_user_id;
map Maps a variable to another variable, useful for setting routing logic based on cookie values.
map $cookie_user_id $variant {
    "user_a" "variant_a";
    "user_b" "variant_b";
}
location / {
    # Set a cookie if not present
    if ($cookie_user_id = "") {
        set $user_id $request_id;
        add_header Set-Cookie "user_id=$user_id; Path=/";
    }

    # Route based on cookie value
    if ($cookie_user_id = "user_a") {
        proxy_pass http://variant_a;
    }

    if ($cookie_user_id = "user_b") {
        proxy_pass http://variant_b;
    }

    # Default to main site
    proxy_pass http://main_site;
}
Note: The request_id is a built-in Nginx variable that can be used to uniquely identify a user session.

4. Using Header-based Routing

Directive Description Example
location Defines a location block for routing based on URI or header match location / { ... }
if Conditional logic to check header values if ($http_user_agent ~* "Mobile") { ... }
proxy_pass Routes request to backend server based on condition proxy_pass http://backend_v1;

Example: Route by User-Agent Header

server {
    listen 80;
    location / {
        if ($http_user_agent ~* "Mobile") {
            proxy_pass http://mobile_backend;
        }
        proxy_pass http://desktop_backend;
    }
}
Note: The if directive can be inefficient and should be used carefully. Consider using map or variables for better performance.

Header-based Routing Configuration

  • Use if or map to inspect headers
  • Route traffic using proxy_pass to different backend servers
  • Supports regex matching for flexible header inspection

5. Implementing Feature Flags

Feature Description Implementation
flag.name Unique identifier for the feature flag my-feature-flag
flag.value Boolean or string value to enable/disable feature true, false, or variant-a
flag.target Target audience or user segment user_id, country, or percentage

Example: Feature Flag Configuration

map $http_user_agent $feature_flag {
    default 0;
    ~*^FeatureFlagEnabled$ 1;
}

server {
    listen 80;

    location / {
        if ($feature_flag = 1) {
            return 200 "Feature is enabled!";
        }
        return 404 "Feature not available";
    }
}
Note: Feature flags can be managed via external services or configuration files. Nginx does not natively support feature flags, so they must be implemented via map directives or external logic.

Feature Flag Use Cases

  • Gradual rollout of new features
  • A/B testing variants
  • Enabling/disabling features based on user segments
  • Safe deployment of experimental features

6. Routing to Multiple Upstreams

Directive Description Example
upstream Defines a group of servers to be used as upstreams
upstream backend {
    server 192.168.1.10:8080;
    server 192.168.1.11:8080;
}
server Defines a server in an upstream group server 192.168.1.10:8080 weight=3;
keepalive Enables keepalive connections to upstream servers keepalive 32;

Example: Load Balancing Across Multiple Upstreams

upstream api_v1 {
    server 10.0.1.10:8080 weight=2;
    server 10.0.1.11:8080 weight=1;
}

upstream api_v2 {
    server 10.0.2.10:8080 weight=3;
    server 10.0.2.11:8080 weight=1;
}

server {
    listen 80;

    location /api/v1/ {
        proxy_pass http://api_v1/;
    }

    location /api/v2/ {
        proxy_pass http://api_v2/;
    }
}
Note: The weight parameter controls how many requests each server receives. Servers with higher weights receive more traffic.

Example: Using Sticky Sessions with Upstreams

upstream backend {
    ip_hash;
    server 192.168.1.10:8080;
    server 192.168.1.11:8080;
}

server {
    listen 80;
    location / {
        proxy_pass http://backend;
    }
}

7. Tracking A/B Test Results

Field Description Example
event_type Type of event being tracked (e.g., conversion, click) conversion
variant Variant being tested (e.g., A, B) A
user_id Unique identifier for the user user_12345
timestamp Time when the event occurred 2026-04-05T10:00:00Z

Example: Logging A/B Test Events

const event = {
  event_type: "conversion",
  variant: "A",
  user_id: "user_12345",
  timestamp: "2026-04-05T10:00:00Z"
};

// Send to analytics backend or logging service
console.log(JSON.stringify(event));
Note: Ensure that tracking does not violate user privacy or comply with regulations like GDPR.

Key Metrics to Track

  • Conversion rate per variant
  • Statistical significance of results
  • User engagement levels
  • Server response times

8. Using Variables for Variants

Variable Description Usage
nginx_var Custom variable defined in Nginx configuration set $variant_a 'A';
http_x_variant HTTP header to identify variant proxy_set_header X-Variant $variant;

Example: Setting and Using Variables

location / {
    # Set variant based on user ID
    set $variant "";
    if ($http_cookie ~* "user_id=([0-9]+)") {
        set $user_id $1;
        if ($user_id < 500) {
            set $variant "A";
        } else {
            set $variant "B";
        }
    }

    # Use variant in proxy pass
    proxy_pass http://backend_$variant;
}
Note: Variables in Nginx are processed in the order they appear in the configuration file.
Warning: Be cautious with variable scope in complex configurations.

9. Implementing Canary Deployments

Directive Description Example
upstream Defines a group of servers to be load balanced
upstream backend {
    server 192.168.1.10:8080 weight=3;
    server 192.168.1.11:8080 weight=1;
}
location Matches URI and defines how to handle requests
location / {
    proxy_pass http://backend;
}

Example: Canary Deployment with Weighted Distribution

upstream api_backend {
    server 192.168.1.10:8080 weight=9;  # 90% of traffic
    server 192.168.1.11:8080 weight=1;  # 10% of traffic (canary)
}

server {
    listen 80;
    location /api/ {
        proxy_pass http://api_backend;
    }
}
Note: The weight parameter controls the distribution of traffic to each server in the upstream group.

Canary Deployment Strategy

  • Gradually shift traffic to the new version
  • Monitor for performance or error issues
  • Use weighted load balancing to control traffic distribution

10. Rolling Out Changes Gradually

Directive Description Example
location Defines how to handle requests based on URI location / { proxy_pass http://backend; }
proxy_pass Specifies the address of the proxied server proxy_pass http://backend_v1;
upstream Groups servers for load balancing upstream backend { server 127.0.0.1:8080; }

Example: Gradual Rollout with Weighted Distribution

upstream backend_v1 {
    server 127.0.0.1:8080 weight=1;
}

upstream backend_v2 {
    server 127.0.0.1:8081 weight=1;
}

server {
    listen 80;

    location / {
        proxy_pass http://backend_v1;
    }
}
Note: The weight parameter controls how many requests each server receives. Adjust weights to gradually shift traffic.

11. Persisting User Assignment

Method Description Notes
Cookie Store user assignment in a cookie for persistence across sessions. Simple and widely supported. Can be manipulated by users.
Database Save user assignment in a database with session or user ID. More secure and reliable for tracking. Requires backend integration.
Local Storage Use browser local storage to persist assignment. Client-side, no server interaction. Data lost if user clears storage.
// Set a cookie to persist user assignment
document.cookie = "ab_test_variant=A; max-age=86400; path=/";
// Read the cookie to get user assignment
function getCookie(name) {
  const value = `; ${document.cookie}`;
  const parts = value.split(`; ${name}=`);
  if (parts.length === 2) return parts.pop().split(';').shift();
}
const variant = getCookie('ab_test_variant');
Note: Ensure cookies are set with secure and HttpOnly flags when used in production environments.
Warning: Local storage and cookies can be cleared or modified by users, potentially affecting test integrity.

12. Measuring A/B Test Metrics

Metric Description Measurement Method Relevance
Conversion Rate Percentage of users who complete a desired action (e.g., purchase, sign-up) Count of conversions / Total visitors Core KPI for A/B testing success
Click-Through Rate (CTR) Percentage of users who click on a specific element Count of clicks / Total impressions Measures engagement with UI elements
Time on Page Average time users spend on a page Server logs or analytics tracking Indicates content engagement or confusion
Bounce Rate Percentage of users who leave after viewing only one page Analytics platform or server logs Indicates relevance or usability issues
Revenue Per User (ARPU) Average revenue generated per user Total revenue / Number of users Measures monetization impact

Example: Conversion Rate Calculation in Nginx

location /conversion {
    access_log /var/log/nginx/conversion.log main;
    return 200 "Conversion recorded";
}
Note: Conversion tracking requires integration with analytics tools or custom logging to capture user actions.

Key Metrics Summary

  • Conversion Rate is the most critical metric for A/B testing success.
  • Click-Through Rate helps understand user interaction with specific elements.
  • Time on Page can indicate content engagement or navigation issues.
  • Bounce Rate reflects how well the page meets user expectations.
  • Revenue Per User is essential for monetization-focused A/B tests.

Was this article helpful?