Managing Access Logs

1. Enabling Access Logging

Configuration Option	Description	Default Value	Notes
accessLog.enabled	Enables or disables access logging	`false`	Set to `true` to enable logging
accessLog.format	Format of the access log entries	`json`	Options: `json`, `text`
accessLog.outputPath	File path where logs are written	`/dev/stdout`	Can be a file or standard output

Example: Enabling Access Logging in Istio

apiVersion: v1
kind: ConfigMap
metadata:
  name: istio
data:
  values: |
    pilot:
      configMap:
        accessLog:
          enabled: true
          format: json
          outputPath: /var/log/access.log

Note: Access logs can significantly impact performance in high-throughput environments. Consider enabling only for debugging or compliance purposes.

2. Configuring Log Format

Field	Description	Example
format	Log format to use for access logs	`json`, `text`
jsonFields	Fields to include in JSON logs	`["timestamp", "method", "url"]`
textFields	Fields to include in text logs	`["timestamp", "method", "url"]`

Example: JSON Log Format Configuration

accessLog:
  format: json
  jsonFields:
    - timestamp
    - method
    - url
    - responseCode

Example: Text Log Format Configuration

accessLog:
  format: text
  textFields:
    - timestamp
    - method
    - url
    - responseCode

3. Setting Log Output Destination

Property	Values	Description
accessLog.destination	`stdout`, `stderr`, `/path/to/file.log`	Specifies where access logs are written. Default is `stdout`.
accessLog.format	`json`, `text`	Format of the log output. JSON provides structured logs; text is human-readable.

Example: Configure Log Output to File

accessLog:
  destination: /var/log/service-mesh/access.log
  format: json

Note: Ensure the destination path has appropriate permissions for log writing.

4. Configuring Log Fields

Field	Description	Default Value	Notes
request.url	Full URL of the request	`/`	Includes query parameters
request.method	HTTP method used	`GET`	Supports all standard methods
response.status	HTTP status code returned	`200`	Can be used for monitoring
response.size	Size of response in bytes	`0`	Useful for bandwidth tracking
request.duration	Time taken to process the request	`0ms`	Measured in milliseconds
source.ip	IP address of the client	`0.0.0.0`	Can be used for access control
destination.ip	IP address of the service	`0.0.0.0`	Used for service-to-service logging
request.headers	Request headers sent by client	`{}`	Can be filtered for sensitive data
response.headers	Response headers sent to client	`{}`	Useful for debugging
request.body	Request body content (if enabled)	`disabled`	Disabled by default for security

Example: Configuring Log Fields in Istio

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: access-log-config
spec:
  workloadSelector:
    labels:
      app: frontend
  config:
    listeners:
    - filterChains:
      - filters:
        - name: envoy.filters.listener.http_connection_manager
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            accessLog:
            - name: envoy.access_loggers.file
              typedConfig:
                '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                path: /dev/stdout
                format: |
                  [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%"

5. Implementing Conditional Logging

Condition	Description	Example
logLevel	Minimum severity level to log	`INFO`, `DEBUG`, `WARN`
logFormat	Output format of logs	`json`, `text`
logFields	Fields to include in log entries	`requestId`, `sourceIp`, `destinationService`

Example: Conditional Logging Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: istio-logging-config
data:
  logLevel: INFO
  logFormat: json
  logFields:
    - requestId
    - sourceIp
    - destinationService

Note: Conditional logging can be implemented using sidecars or dedicated logging agents like Fluentd or Logstash.

6. Managing Log Sampling

Property	Description	Default	Notes
logSamplingRate	Percentage of logs to sample for processing	`1.0`	Values between 0.0 and 1.0
logLevel	Minimum severity level to include in logs	`INFO`	Can be `DEBUG`, `INFO`, `WARN`, or `ERROR`

Example: Configuring Log Sampling

apiVersion: v1
kind: ServiceMesh
metadata:
  name: example-mesh
spec:
  logSamplingRate: 0.5
  logLevel: INFO

Note: Setting logSamplingRate to 0.0 disables all logs.

Warning: High sampling rates may impact performance in high-volume environments.

7. Setting Log Filtering

Filter Type	Description	Configuration	Notes
accessLog.filter	Defines the filter to apply to access logs	`"accessLog.filter": "level:INFO, type:REQUEST"`	Can be used to reduce log verbosity
accessLog.exclude	Excludes specific log entries from being recorded	`"accessLog.exclude": ["status:200", "path:/health"]`	Useful for filtering out health checks or successful responses
accessLog.include	Includes only specific log entries in the output	`"accessLog.include": ["method:GET", "path:/api"]`	Allows fine-grained control over what logs to capture

Example: Configuring Log Filters in Istio

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: access-log-filter
spec:
  workloadSelector:
    labels:
      app: frontend
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: "envoy.filters.listener.http_connection_manager"
    patch:
      operation: INSERT_BEFORE
      value:
        name: "envoy.filters.http.access_log"
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.access_log.v3.AccessLog"
          path: "/dev/stdout"
          filter:
            and_filter:
              filters:
              - status_code_filter:
                  comparison:
                    op: EQ
                    value:
                      value: 200
              - duration_filter:
                  comparison:
                    op: GT
                    value:
                      value: 1000

Note: Log filtering can significantly reduce log volume and improve performance in high-traffic environments.

8. Implementing Structured Logging

Field	Description	Example
log.format	Format of the log output (json, text)	`json`
log.level	Minimum level to log (trace, debug, info, warn, error)	`info`
log.output	Destination of logs (stdout, file)	`stdout`

Example: Configuring Structured Logging

log:
  format: json
  level: info
  output: stdout

Note: Structured logs are easier to parse and analyze with tools like ELK stack or Grafana.

9. Configuring Log Rotation

Property	Description	Default	Notes
log.rotation.maxSize	Maximum size of a log file before rotation occurs	`100MB`	Can be specified in MB, GB, or KB
log.rotation.maxBackups	Number of backup log files to keep	`5`	Older backups are deleted when new ones are created
log.rotation.maxAge	Maximum age of a log file before rotation occurs	`24h`	Time format supports hours (h), minutes (m), seconds (s)

Example: Log Rotation Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: service-mesh-logging
data:
  log-config.yaml: |
    log:
      rotation:
        maxSize: 50MB
        maxBackups: 3
        maxAge: 12h

Note: Log rotation settings can be adjusted dynamically without restarting the service mesh components.

10. Analyzing Access Logs

Field	Description	Example
timestamp	Time when the log entry was generated	`2025-04-05T14:30:00Z`
source	Originating service or client	`frontend-app`
destination	Target service or endpoint	`backend-api`
method	HTTP method used	`GET`
path	Requested resource path	`/api/users/123`
status	HTTP response status code	`200`
responseTime	Duration of the request in milliseconds	`150`
requestSize	Size of the request payload in bytes	`256`
responseSize	Size of the response payload in bytes	`1024`

Example: Log Entry Sample

{
  "timestamp": "2025-04-05T14:30:00Z",
  "source": "frontend-app",
  "destination": "backend-api",
  "method": "GET",
  "path": "/api/users/123",
  "status": 200,
  "responseTime": 150,
  "requestSize": 256,
  "responseSize": 1024
}

Key Metrics for Analysis

Response Time - Critical for performance monitoring
Status Codes - Identify errors and success rates
Request/Response Size - Useful for bandwidth and caching analysis
Source/Destination - Helps in tracing traffic flow

Note: Access logs can be parsed and analyzed using tools like ELK Stack (Elasticsearch, Logstash, Kibana) or Prometheus with Grafana for real-time dashboards.

11. Integrating with Log Aggregation Systems

System	Integration Method	Configuration	Notes
Fluentd	Forwarding via TCP/UDP	`{ "forward": { "host": "fluentd-host", "port": 24224 } }`	Supports structured JSON logs
Logstash	Input plugin (tcp/udp)	`{ "input": { "tcp": { "port": 5959 } } }`	Can parse and enrich logs
Prometheus	Remote write endpoint	`{ "remote_write": [ { "url": "http://prometheus:9090/api/v1/write" } ] }`	For metrics and log correlation
Elastic Stack (ELK)	Logstash or Filebeat	`{ "filebeat": { "prospector": { "paths": [ "/var/log/service-mesh/*.log" ] } } }`	Full-text search capabilities
Datadog	Agent-based or API ingestion	`{ "logs": { "enabled": true, "source": "service-mesh" } }`	Real-time monitoring and alerting

Example: Fluentd Integration

# fluentd config
<source>
  @type forward
  port 24224
</source>

<match service.mesh.*>
  @type stdout
</match>

Example: Logstash Pipeline

# logstash config
input {
  tcp {
    port => 5959
    codec => json_lines
  }
}

output {
  stdout { codec => rubydebug }
}

Note: Ensure log aggregation systems support structured logging formats like JSON for optimal parsing and filtering.

Warning: High-volume logs can overwhelm log aggregation systems; consider implementing rate limiting or sampling.

12. Managing Log Retention Policies

Policy	Description	Configuration	Notes
log.retention.days	Number of days to retain logs	`30`	Default is 30 days
log.retention.size	Maximum size of log storage (e.g., 10GB)	`10GB`	Logs are rotated when size limit is reached
log.retention.strategy	Strategy for log rotation	`time-based` or `size-based`	Can be combined for flexible retention

Example: Configuring Log Retention in Istio

apiVersion: v1
kind: ConfigMap
metadata:
  name: istio
data:
  meshConfig: |
    defaultConfig:
      accessLogFile: /dev/stdout
      logLevel: info
    policy:
      retention:
        days: 30
        size: "10GB"
        strategy: "time-based"

Note: Log retention policies apply to access logs generated by the service mesh, including request and response metadata.