Skip to main content
Mapagam

Implementing Observability

1. Configuring OpenTelemetry Integration

Example: Basic OpenTelemetry Configuration

load_module modules/ngx_opentelemetry_module.so;

events {
    worker_connections 1024;
}

http {
    opentelemetry on;
    opentelemetry_exporter http;
    opentelemetry_endpoint http://otel-collector:4318;

    server {
        listen 80;
        location / {
            proxy_pass http://backend;
        }
    }
}
Directive Description Default Version
opentelemetry Enables or disables OpenTelemetry integration off 1.25.0+
opentelemetry_exporter Specifies the exporter to use (http, grpc) http 1.25.0+
opentelemetry_endpoint Endpoint URL for the OpenTelemetry collector http://localhost:4318 1.25.0+
opentelemetry_service_name Name of the service being monitored nginx 1.25.0+
Note: The OpenTelemetry module requires Nginx 1.25+ and must be compiled with the ngx_opentelemetry_module.
Warning: Ensure the OpenTelemetry collector is running and accessible from the Nginx server for telemetry to be sent.

2. Implementing Distributed Tracing

Directive Description Example
opentracing Enables OpenTracing support for tracing requests opentracing on;
opentracing_operation_name Sets the operation name for tracing opentracing_operation_name "nginx";
opentracing_tag Adds custom tags to trace spans opentracing_tag service_name $service;
opentracing_trace_context Specifies how trace context is propagated (e.g., via headers) opentracing_trace_context "traceparent";

Example: Basic OpenTracing Configuration

load_module modules/ngx_http_opentracing_module.so;

server {
    listen 80;
    opentracing on;
    opentracing_operation_name "nginx";
    
    location / {
        opentracing_tag service_name "web";
        proxy_pass http://backend;
    }
}
Note: Requires ngx_http_opentracing_module to be installed and loaded.

Tracing Integration Options

  • NEW OpenTelemetry support added in Nginx 1.25+
  • OpenTracing compatible with Jaeger, Zipkin, and other tracing systems
  • Can be used in conjunction with opentracing and opentelemetry modules

Distributed Tracing Options in Nginx

  • OpenTracing: Legacy but widely supported
  • OpenTelemetry: Modern, standardized, and actively developed
  • No Tracing: Default behavior, no trace propagation

3. Adding Trace Headers

Header Name Value Format Description
trace-id [a-f0-9]{32} Unique identifier for the trace
span-id [a-f0-9]{16} Identifier for the current span
trace-state key1=value1,key2=value2 Additional trace context (optional)

Example: Adding Trace Headers to Nginx

location /api/ {
    proxy_pass http://backend;
    proxy_set_header trace-id $request_id;
    proxy_set_header span-id $request_id;
    proxy_set_header trace-state $http_trace_state;
}
Note: The $request_id variable is a built-in Nginx variable that provides a unique identifier for each request.

4. Using Jaeger for Tracing

Directive Description Example
opentelemetry Enables OpenTelemetry tracing support in Nginx opentelemetry on;
opentelemetry_tracer Specifies the OpenTelemetry tracer to use (e.g., Jaeger) opentelemetry_tracer jaeger;
opentelemetry_sampler Configures sampling strategy (e.g., always, never, probabilistic) opentelemetry_sampler probabilistic 0.1;
opentelemetry_service_name Sets the service name reported to Jaeger opentelemetry_service_name "nginx-service";
opentelemetry_endpoint Specifies the Jaeger collector endpoint (e.g., http://localhost:14268/api/traces) opentelemetry_endpoint http://jaeger-collector:14268/api/traces;

Example: Basic Jaeger Tracing Configuration

events {
    worker_connections 1024;
}

http {
    opentelemetry on;
    opentelemetry_tracer jaeger;
    opentelemetry_sampler probabilistic 0.1;
    opentelemetry_service_name "nginx-service";
    opentelemetry_endpoint http://jaeger-collector:14268/api/traces;

    server {
        listen 80;
        location / {
            proxy_pass http://backend;
        }
    }
}
Note: Ensure that the OpenTelemetry module is installed and enabled in your Nginx build for tracing support.

5. Integrating with Zipkin

Property Value Description
zipkin.endpoint http://zipkin-server:9411/api/v2/spans Endpoint URL for Zipkin collector
zipkin.service.name nginx-observability Name of the service in Zipkin
zipkin.sample.rate 0.1 Sampling rate for trace collection (0.0 to 1.0)

Example: Nginx Configuration for Zipkin Integration

http {
    zipkin on;
    zipkin_endpoint http://zipkin-server:9411/api/v2/spans;
    zipkin_service_name nginx-observability;
    zipkin_sample_rate 0.1;

    server {
        listen 80;
        location / {
            proxy_pass http://backend;
        }
    }
}
Note: Ensure Zipkin server is running and accessible from the Nginx instance.
Warning: High sampling rates may impact performance; adjust zipkin_sample_rate accordingly.

6. Configuring Span Context Propagation

Property Description Default Notes
opentelemetry.span_context_propagation Enables or disables span context propagation for tracing. off Set to on to enable propagation.
opentelemetry.traceparent_header Header name used to propagate trace context. traceparent Can be customized for compatibility with other tracing systems.

Example: Enabling Span Context Propagation

opentelemetry span_context_propagation on;
opentelemetry traceparent_header traceparent;
Note: Ensure that your tracing backend supports the traceparent header format for proper context propagation.

7. Setting Up Log Correlation

Directive Description Example
log_format Defines custom log formats for Nginx access logs log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time';
access_log Specifies where to write access logs and which format to use access_log /var/log/nginx/access.log main;
error_log Specifies where to write error logs and log level error_log /var/log/nginx/error.log warn;

Example: Log Format with Trace ID

log_format json_trace '{"timestamp": "$time_iso8601", "trace_id": "$http_x_trace_id", "remote_addr": "$remote_addr", "request": "$request", "status": "$status", "body_bytes_sent": "$body_bytes_sent"}';

access_log /var/log/nginx/access.json json_trace;
Note: Ensure that your logging system supports structured logs (e.g., JSON) for effective correlation.

Workflow: Enabling Structured Logging in Nginx

  1. Create a custom log format using log_format directive with JSON structure
  2. Use access_log to specify the log file and format
  3. Include trace IDs or other unique identifiers in logs to enable correlation across services
  4. Configure log aggregation tools (e.g., ELK stack, Fluentd) to parse and store logs

8. Implementing Request Flow Tracking

Directive Description Usage
log_format Defines custom log formats for request tracking log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"';
access_log Enables or disables access logging with specified format and path access_log /var/log/nginx/access.log main;
error_log Configures error logging with level and destination error_log /var/log/nginx/error.log warn;

Example: Custom Log Format with Trace ID

log_format trace '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" X-Trace-ID: $request_id';

Example: Access Log with Request ID

server {
    listen 80;
    server_name example.com;

    access_log /var/log/nginx/access.log trace;
    error_log /var/log/nginx/error.log warn;

    location / {
        proxy_pass http://backend;
        proxy_set_header X-Request-ID $request_id;
    }
}
Note: The $request_id variable is available in Nginx 1.13.0+ and can be used to correlate logs across services in a distributed system.

Key Points for Request Flow Tracking

  • Use log_format to define custom logging formats that include trace identifiers.
  • Enable access_log and error_log to capture request flow details.
  • The $request_id variable helps correlate logs across microservices or backend systems.

9. Using OpenTracing Module

Directive Description Default Notes
opentracing_load_tracer Specifies the path to the OpenTracing tracer library. — Required for enabling tracing support.
opentracing_tracer_config Path to the OpenTracing tracer configuration file. — Can be used to configure the tracer's behavior.
opentracing_operation_name Operation name for the trace span. nginx Can be customized per location or server.
opentracing_tag Adds custom tags to the trace span. — Can be used for adding metadata to traces.

Example: Basic OpenTracing Configuration

load_module modules/ngx_http_opentracing_module.so;

server {
    listen 80;
    server_name example.com;

    opentracing_load_tracer /usr/lib/libjaegertracing.so;
    opentracing_tracer_config /etc/jaeger-config.json;

    location / {
        opentracing_operation_name "nginx-request";
        proxy_pass http://backend;
    }
}
Note: The OpenTracing module requires a compatible tracer library (e.g., Jaeger, Zipkin) to be installed and configured.

10. Monitoring Latency Distributions

Metric Description Configuration Notes
latency_ms Response time in milliseconds log_format with $request_time Use for tracking performance over time
latency_percentiles 90th, 95th, 99th percentiles of response times Use nginx-module-vts or custom logging Helps identify outliers and slow requests

Example: Logging Latency with Custom Format

log_format custom '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time';

access_log /var/log/nginx/access.log custom;
Note: The $request_time variable includes the time to read the request, process it, and send the response.

Latency Monitoring Best Practices

  • Log response times at regular intervals to detect performance degradation
  • Use percentiles to understand tail latencies that impact user experience
  • Integrate with monitoring tools like Prometheus or Grafana for visualization

11. Configuring Trace Sampling

Directive Description Default Notes
trace_sampling_ratio Sets the ratio of traces to sample for observability 0.1 Value between 0 and 1; 0.1 = 10% of traces sampled
trace_sampling_mode Mode for trace sampling (e.g., rate, dynamic) rate Controls how sampling is applied

Example: Setting Trace Sampling Ratio

http {
    trace_sampling_ratio 0.2;
}
Note: The trace_sampling_ratio directive is part of the nginx-module-otel or similar OpenTelemetry modules and requires appropriate instrumentation.

12. Exporting Metrics to Backends

Backend Export Method Configuration Example Notes
Prometheus nginx_exporter 
location /metrics {
    scrape;
}
 Exports metrics in Prometheus format
Datadog HTTP API log_format datadog '$status $bytes_sent $request_time'; Requires Datadog agent or API integration
Elastic Stack Logstash / Beats 
output {
  elasticsearch { hosts => [ "localhost:9200" ] }
}
 Can collect logs and metrics via Logstash
InfluxDB HTTP API 
influxdb {
  url ="http://localhost:8086"
  database ="nginx_metrics"
}
 Supports direct metric ingestion

Example: Prometheus Exporter Configuration

server {
    listen 80;
    location /metrics {
        scrape;
        auth_basic "Nginx Metrics";
        auth_basic_user_file /etc/nginx/htpasswd;
    }
}
Note: Ensure that the nginx_exporter is running and configured to scrape Nginx metrics.

Was this article helpful?

Related Cheat Sheets