Implementing A/B Testing
1. Using Split Clients
The split_clients directive in Nginx allows you to split traffic between different versions of content or configurations based on a key, such as a user ID or IP address. This is useful for A/B testing.
| Directive | Description | Example |
|---|---|---|
| split_clients | Defines a variable that splits traffic based on a key. | split_clients "${remote_addr}" $variant { |
| split_clients | Specifies the percentage of traffic to assign to each variant. | split_clients "${remote_addr}" $variant { |
Example: Basic Split Client Configuration
http {
split_clients "${remote_addr}" $variant {
50% "variant_a";
50% "variant_b";
}
server {
listen 80;
location / {
if ($variant = "variant_a") {
proxy_pass http://backend_a;
}
if ($variant = "variant_b") {
proxy_pass http://backend_b;
}
}
}
}
Note: The
split_clients directive uses a consistent hashing algorithm to ensure that the same client always gets assigned to the same variant.
2. Distributing Traffic
| Method | Description | Configuration Example | Notes |
|---|---|---|---|
| ip_hash | Hashes the client's IP address to determine which server to route to. | upstream backend { ip_hash; server 192.168.1.10; server 192.168.1.11;} |
Ensures consistent routing for the same client IP |
| least_conn | Selects the server with the least number of active connections. | upstream backend { least_conn; server 192.168.1.10; server 192.168.1.11;} |
Good for balancing load across servers with varying capacities |
| random | Randomly selects a server from the group. | upstream backend { random; server 192.168.1.10; server 192.168.1.11;} |
Simple but not always optimal for consistent performance |
| weight | Assigns weights to servers, affecting the probability of selection. | upstream backend { server 192.168.1.10 weight=3; server 192.168.1.11 weight=1;} |
Useful for load balancing based on server capacity |
Example: Using ip_hash for A/B Testing
upstream ab_test_backend {
ip_hash;
server 192.168.1.10:8080;
server 192.168.1.11:8080;
}
location / {
proxy_pass http://ab_test_backend;
}
Note: The
ip_hash method ensures that users with the same IP address are consistently routed to the same server, which is essential for maintaining session consistency in A/B testing.
3. Setting Cookie-based Routing
| Directive | Description | Example |
|---|---|---|
| set | Sets a variable with a value, often used for cookie handling. | set $cookie_user_id $cookie_user_id; |
| map | Maps a variable to another variable, useful for setting routing logic based on cookie values. | |
Example: Setting a Cookie for A/B Testing
location / {
# Set a cookie if not present
if ($cookie_user_id = "") {
set $user_id $request_id;
add_header Set-Cookie "user_id=$user_id; Path=/";
}
# Route based on cookie value
if ($cookie_user_id = "user_a") {
proxy_pass http://variant_a;
}
if ($cookie_user_id = "user_b") {
proxy_pass http://variant_b;
}
# Default to main site
proxy_pass http://main_site;
}
Note: The
request_id is a built-in Nginx variable that can be used to uniquely identify a user session.
4. Using Header-based Routing
| Directive | Description | Example |
|---|---|---|
| location | Defines a location block for routing based on URI or header match | location / { ... } |
| if | Conditional logic to check header values | if ($http_user_agent ~* "Mobile") { ... } |
| proxy_pass | Routes request to backend server based on condition | proxy_pass http://backend_v1; |
Example: Route by User-Agent Header
server {
listen 80;
location / {
if ($http_user_agent ~* "Mobile") {
proxy_pass http://mobile_backend;
}
proxy_pass http://desktop_backend;
}
}
Note: The
if directive can be inefficient and should be used carefully. Consider using map or variables for better performance.
Header-based Routing Configuration
- Use
iformapto inspect headers - Route traffic using
proxy_passto different backend servers - Supports regex matching for flexible header inspection
5. Implementing Feature Flags
| Feature | Description | Implementation |
|---|---|---|
| flag.name | Unique identifier for the feature flag | my-feature-flag |
| flag.value | Boolean or string value to enable/disable feature | true, false, or variant-a |
| flag.target | Target audience or user segment | user_id, country, or percentage |
Example: Feature Flag Configuration
map $http_user_agent $feature_flag {
default 0;
~*^FeatureFlagEnabled$ 1;
}
server {
listen 80;
location / {
if ($feature_flag = 1) {
return 200 "Feature is enabled!";
}
return 404 "Feature not available";
}
}
Note: Feature flags can be managed via external services or configuration files. Nginx does not natively support feature flags, so they must be implemented via
map directives or external logic.
Feature Flag Use Cases
- Gradual rollout of new features
- A/B testing variants
- Enabling/disabling features based on user segments
- Safe deployment of experimental features
6. Routing to Multiple Upstreams
| Directive | Description | Example |
|---|---|---|
| upstream | Defines a group of servers to be used as upstreams | |
| server | Defines a server in an upstream group | server 192.168.1.10:8080 weight=3; |
| keepalive | Enables keepalive connections to upstream servers | keepalive 32; |
Example: Load Balancing Across Multiple Upstreams
upstream api_v1 {
server 10.0.1.10:8080 weight=2;
server 10.0.1.11:8080 weight=1;
}
upstream api_v2 {
server 10.0.2.10:8080 weight=3;
server 10.0.2.11:8080 weight=1;
}
server {
listen 80;
location /api/v1/ {
proxy_pass http://api_v1/;
}
location /api/v2/ {
proxy_pass http://api_v2/;
}
}
Note: The
weight parameter controls how many requests each server receives. Servers with higher weights receive more traffic.
Example: Using Sticky Sessions with Upstreams
upstream backend {
ip_hash;
server 192.168.1.10:8080;
server 192.168.1.11:8080;
}
server {
listen 80;
location / {
proxy_pass http://backend;
}
}
7. Tracking A/B Test Results
| Field | Description | Example |
|---|---|---|
| event_type | Type of event being tracked (e.g., conversion, click) | conversion |
| variant | Variant being tested (e.g., A, B) | A |
| user_id | Unique identifier for the user | user_12345 |
| timestamp | Time when the event occurred | 2026-04-05T10:00:00Z |
Example: Logging A/B Test Events
const event = {
event_type: "conversion",
variant: "A",
user_id: "user_12345",
timestamp: "2026-04-05T10:00:00Z"
};
// Send to analytics backend or logging service
console.log(JSON.stringify(event));
Note: Ensure that tracking does not violate user privacy or comply with regulations like GDPR.
Key Metrics to Track
- Conversion rate per variant
- Statistical significance of results
- User engagement levels
- Server response times
8. Using Variables for Variants
| Variable | Description | Usage |
|---|---|---|
| nginx_var | Custom variable defined in Nginx configuration | set $variant_a 'A'; |
| http_x_variant | HTTP header to identify variant | proxy_set_header X-Variant $variant; |
Example: Setting and Using Variables
location / {
# Set variant based on user ID
set $variant "";
if ($http_cookie ~* "user_id=([0-9]+)") {
set $user_id $1;
if ($user_id < 500) {
set $variant "A";
} else {
set $variant "B";
}
}
# Use variant in proxy pass
proxy_pass http://backend_$variant;
}
Note: Variables in Nginx are processed in the order they appear in the configuration file.
Warning: Be cautious with variable scope in complex configurations.
9. Implementing Canary Deployments
| Directive | Description | Example |
|---|---|---|
| upstream | Defines a group of servers to be load balanced | |
| location | Matches URI and defines how to handle requests | |
Example: Canary Deployment with Weighted Distribution
upstream api_backend {
server 192.168.1.10:8080 weight=9; # 90% of traffic
server 192.168.1.11:8080 weight=1; # 10% of traffic (canary)
}
server {
listen 80;
location /api/ {
proxy_pass http://api_backend;
}
}
Note: The
weight parameter controls the distribution of traffic to each server in the upstream group.
Canary Deployment Strategy
- Gradually shift traffic to the new version
- Monitor for performance or error issues
- Use weighted load balancing to control traffic distribution
10. Rolling Out Changes Gradually
| Directive | Description | Example |
|---|---|---|
| location | Defines how to handle requests based on URI | location / { proxy_pass http://backend; } |
| proxy_pass | Specifies the address of the proxied server | proxy_pass http://backend_v1; |
| upstream | Groups servers for load balancing | upstream backend { server 127.0.0.1:8080; } |
Example: Gradual Rollout with Weighted Distribution
upstream backend_v1 {
server 127.0.0.1:8080 weight=1;
}
upstream backend_v2 {
server 127.0.0.1:8081 weight=1;
}
server {
listen 80;
location / {
proxy_pass http://backend_v1;
}
}
Note: The
weight parameter controls how many requests each server receives. Adjust weights to gradually shift traffic.
11. Persisting User Assignment
| Method | Description | Notes |
|---|---|---|
| Cookie | Store user assignment in a cookie for persistence across sessions. | Simple and widely supported. Can be manipulated by users. |
| Database | Save user assignment in a database with session or user ID. | More secure and reliable for tracking. Requires backend integration. |
| Local Storage | Use browser local storage to persist assignment. | Client-side, no server interaction. Data lost if user clears storage. |
Example: Setting a Cookie for A/B Test Assignment
// Set a cookie to persist user assignment
document.cookie = "ab_test_variant=A; max-age=86400; path=/";
Example: Retrieving Assignment from Cookie
// Read the cookie to get user assignment
function getCookie(name) {
const value = `; ${document.cookie}`;
const parts = value.split(`; ${name}=`);
if (parts.length === 2) return parts.pop().split(';').shift();
}
const variant = getCookie('ab_test_variant');
Note: Ensure cookies are set with secure and HttpOnly flags when used in production environments.
Warning: Local storage and cookies can be cleared or modified by users, potentially affecting test integrity.
12. Measuring A/B Test Metrics
| Metric | Description | Measurement Method | Relevance |
|---|---|---|---|
| Conversion Rate | Percentage of users who complete a desired action (e.g., purchase, sign-up) | Count of conversions / Total visitors | Core KPI for A/B testing success |
| Click-Through Rate (CTR) | Percentage of users who click on a specific element | Count of clicks / Total impressions | Measures engagement with UI elements |
| Time on Page | Average time users spend on a page | Server logs or analytics tracking | Indicates content engagement or confusion |
| Bounce Rate | Percentage of users who leave after viewing only one page | Analytics platform or server logs | Indicates relevance or usability issues |
| Revenue Per User (ARPU) | Average revenue generated per user | Total revenue / Number of users | Measures monetization impact |
Example: Conversion Rate Calculation in Nginx
location /conversion {
access_log /var/log/nginx/conversion.log main;
return 200 "Conversion recorded";
}
Note: Conversion tracking requires integration with analytics tools or custom logging to capture user actions.
Key Metrics Summary
- Conversion Rate is the most critical metric for A/B testing success.
- Click-Through Rate helps understand user interaction with specific elements.
- Time on Page can indicate content engagement or navigation issues.
- Bounce Rate reflects how well the page meets user expectations.
- Revenue Per User is essential for monetization-focused A/B tests.