Working with APM Server

1. Understanding APM Architecture

Component	Description	Role	Version
APM Server	Collects and processes APM data from agents	Central processing unit for APM data	8.12.0
Elasticsearch	Stores APM data (traces, metrics, logs)	Data storage and indexing	8.12.0
Kibana	Provides UI for APM data visualization	Data visualization and dashboarding	8.12.0
Elastic Agents	Collects telemetry data from services	Data collection and agent management	8.12.0

Example: APM Server Configuration

apm-server:
  host: "localhost:8200"
  rum:
    enabled: true
    allow_origins: ["*"]
  kibana:
    enabled: true
    host: "http://localhost:5601"

APM Data Flow

Elastic Agents collect telemetry data from services
APM Server receives and processes the data
Data is indexed into Elasticsearch
Kibana visualizes the APM data

2. Installing APM Server

Operating System	Installation Method	Notes
Linux (Debian/Ubuntu)	`apt install elastic-apm-server`	Requires Elasticsearch and Kibana to be installed first
Linux (RHEL/CentOS)	`yum install elastic-apm-server`	Requires Elasticsearch and Kibana to be installed first
macOS	`brew install elastic/apm-server/apm-server`	Requires Elasticsearch and Kibana to be installed first
Windows	`choco install elastic-apm-server`	Requires Elasticsearch and Kibana to be installed first
Docker	`docker run docker.elastic.co/apm/apm-server:8.15.0`	Use `-v /path/to/apm-server.yml:/usr/share/apm-server/apm-server.yml` to mount config

Example: Docker Installation

docker run -d \
  --name apm-server \
  --network elasticsearch-network \
  -p 8200:8200 \
  docker.elastic.co/apm/apm-server:8.15.0 \
  -e "output.elasticsearch.hosts=[\"http://elasticsearch:9200\"]" \
  -e "setup.kibana.host=http://kibana:5601"

Note: Ensure that the APM Server version matches your Elasticsearch and Kibana versions for compatibility.

3. Configuring APM Server

Property	Values	Description
apm-server.host	`"localhost:8200"`	Host and port where APM Server listens for agent data
apm-server.secret_token	`"secret-token"`	Secret token for securing APM Server endpoints
apm-server.ssl.enabled	`false`	Enable or disable SSL for APM Server communication
apm-server.ssl.certificate	`"/path/to/cert.pem"`	Path to SSL certificate file
apm-server.ssl.key	`"/path/to/key.pem"`	Path to SSL private key file
apm-server.rum.enabled	`false`	Enable or disable Real User Monitoring (RUM)
apm-server.rum.allow_origins	`["*"]`	Allowed origins for RUM requests
apm-server.kibana.host	`"localhost:5601"`	Host and port of Kibana instance
apm-server.kibana.username	`"elastic"`	Username for Kibana authentication
apm-server.kibana.password	`"password"`	Password for Kibana authentication

Example: Basic APM Server Configuration

apm-server:
  host: "localhost:8200"
  secret_token: "secret-token"
  ssl:
    enabled: false
  rum:
    enabled: true
    allow_origins: ["*"]
  kibana:
    host: "localhost:5601"
    username: "elastic"
    password: "password"

Note: Ensure that the secret_token is securely stored and not exposed in configuration files or logs.

Warning: Enabling SSL in APM Server requires proper certificate setup to avoid connection errors.

4. Installing APM Agents

Agent	Language/Platform	Installation Method	Notes
APM Agent for Node.js	Node.js	`npm install @elastic/apm-agent-nodejs`	Supports Node.js 12+
APM Agent for Java	Java	`mvn install org.elasticsearch.apm:apm-agent`	Works with Java 8+
APM Agent for Python	Python	`pip install elastic-apm`	Supports Python 3.6+
APM Agent for .NET	.NET	`nuget install Elastic.Apm`	Supports .NET Framework 4.6.1+ and .NET Core 2.1+
APM Agent for Go	Go	`go get github.com/elastic/apm-agent-go`	Requires Go 1.12+
APM Agent for PHP	PHP	`composer require elastic/apm-agent`	Requires PHP 7.2+
APM Agent for Ruby	Ruby	`gem install elastic-apm`	Supports Ruby 2.5+
APM Agent for RUM	JavaScript (Browser)	`npm install @elastic/apm-rum`	For real user monitoring in browsers

Example: Installing APM Agent for Node.js

npm install @elastic/apm-agent-nodejs

Example: Installing APM Agent for Java

&lt;dependency&gt;
  &lt;groupId&gt;org.elasticsearch.apm&lt;/groupId&gt;
  &lt;artifactId&gt;apm-agent&lt;/artifactId&gt;
  &lt;version&gt;1.32.0&lt;/version&gt;
&lt;/dependency&gt;

5. Configuring APM Agents

Property	Values	Description
apm.server_url	`http://localhost:8200`	URL of the APM Server to send data to
apm.service_name	`my-service`	Name of the service being monitored
apm.service_version	`1.0.0`	Version of the service being monitored
apm.environment	`production`	Environment where the service is running (e.g., production, staging)
apm.log_level	`info`	Log level for APM agent logs (debug, info, warn, error)
apm.transaction_sample_rate	`1.0`	Sample rate for transaction collection (0.0 to 1.0)

Example: APM Agent Configuration in Node.js

const apm = require('@elastic/apm-node').start({
  serverUrl: 'http://localhost:8200',
  serviceName: 'my-service',
  serviceVersion: '1.0.0',
  environment: 'production',
  logLevel: 'info',
  transactionSampleRate: 1.0
});

Example: APM Agent Configuration in Python

import elasticapm

elasticapm.Client(
    server_url='http://localhost:8200',
    service_name='my-service',
    service_version='1.0.0',
    environment='production',
    log_level='info',
    transaction_sample_rate=1.0
)

Note: Ensure that the APM Server is running and accessible at the specified URL before starting the agent.

6. Viewing Traces

Field	Description	Example
trace.id	Unique identifier for a trace	`1234567890abcdef`
transaction.id	Unique identifier for a transaction within a trace	`abcdef1234567890`
parent.id	ID of the parent transaction, if any	`0000000000000000`
timestamp	ISO 8601 formatted timestamp of the trace start	`2024-05-20T10:00:00.000Z`
duration	Duration of the trace in milliseconds	`1250.5`

Example: Trace Overview in APM UI

{
  "trace": {
    "id": "1234567890abcdef",
    "timestamp": "2024-05-20T10:00:00.000Z",
    "duration": 1250.5
  },
  "transactions": [
    {
      "id": "abcdef1234567890",
      "name": "GET /api/users",
      "duration": 1000.0,
      "parent": null
    }
  ]
}

Note: Traces are visualized in the APM UI as a timeline showing transaction dependencies and performance metrics.

7. Analyzing Transaction Performance

Property	Description	Example
transaction.name	Name of the transaction	`"GET /api/users"`
transaction.duration	Duration of the transaction in milliseconds	`125.3`
transaction.type	Type of transaction (e.g., web, queue)	`"request"`
transaction.spans	Number of spans within the transaction	`5`

Example: Transaction Performance Query

GET /elasticsearch/_search
{
  "query": {
    "match": {
      "transaction.name": "GET /api/users"
    }
  },
  "aggs": {
    "avg_duration": {
      "avg": {
        "field": "transaction.duration"
      }
    }
  }
}

Note: Use APM UI dashboards to visualize transaction performance over time.

Key Metrics for Transaction Analysis

Duration - How long a transaction takes to complete
Throughput - Number of transactions per time unit
Error Rate - Percentage of failed transactions
Spans - Sub-operations within a transaction

8. Monitoring Errors

Error Type	Description	Impact	Resolution
Transaction Errors	Errors occurring during transaction processing in APM	Can cause incomplete or failed transactions, affecting performance metrics	Review transaction logs and adjust error handling logic in application code
Span Errors	Errors within spans that are part of a transaction	May lead to inaccurate tracing data and missing performance insights	Identify problematic spans in the trace and fix root cause in service code
Agent Errors	Issues with APM agent configuration or runtime errors	Can prevent data collection from the application	Check agent logs for configuration issues and update to latest version if needed
Index Errors	Errors in indexing or querying data in Elasticsearch	May result in missing or delayed APM data in Kibana	Verify Elasticsearch health and adjust index settings or mappings as needed

Example: Transaction Error Handling

try {
  const result = await someAsyncOperation();
} catch (error) {
  // Log error to APM
  apm.captureError(error);
  // Handle or rethrow error appropriately
}

Note: APM server logs errors to its own log file, which can be used for troubleshooting issues with error reporting.

9. Using Service Maps

Example: Viewing Service Map in Kibana

curl -X GET "http://localhost:5601/api/apm/service-map" \
  -H "Authorization: Bearer <access-token>" \
  -H "Content-Type: application/json"

Field	Description	Values
service.name	Name of the service	`"frontend"`
service.environment	Environment where service is running	`"production"`
service.version	Version of the service	`"1.2.3"`
service.agent.name	APM agent used	`"rum-js"`

Note: Service maps visualize dependencies between services in your APM environment.

Service Map vs Trace View

Service Map: Shows relationships and dependencies between services
Trace View: Displays individual trace details for a specific transaction

Steps to Configure Service Map

Navigate to APM in Kibana
Go to the Service Maps section
Select services to visualize
Adjust time range if needed

10. Configuring Sampling Rates

Property	Default Value	Description	Notes
apm-server.sampling.rate	`100`	Global sampling rate for APM events	Value between 0 and 100. 0 means no sampling, 100 means all events sampled
apm-server.sampling.rate.limit	`1000`	Maximum number of events per second to sample	Applies when sampling rate is set to 100% or higher

Example: Setting Sampling Rate in apm-server.yml

apm-server:
  sampling:
    rate: 50
    rate.limit: 2000

Note: Setting rate to 0 disables sampling and collects all events. This can significantly impact performance and should be used with caution in production environments.

11. Setting Up RUM

Real User Monitoring (RUM) in the ELK Stack allows you to collect performance data from real users' browsers. This section outlines how to configure and set up RUM in your APM Server.

Configuration	Description	Value
apm-server.rum.enabled	Enables Real User Monitoring	`true`
apm-server.rum.allow_origins	Whitelisted origins for RUM requests	`["https://yourdomain.com"]`
apm-server.rum.library_pattern	Patterns to exclude library files from RUM	`["node_modules", "vendor"]`
apm-server.rum.exclude_from_grouping	Patterns to exclude from grouping	`["health"]`

Example: Enabling RUM in apm-server.yml

apm-server:
  rum:
    enabled: true
    allow_origins: ["https://yourdomain.com"]
    library_pattern: ["node_modules", "vendor"]
    exclude_from_grouping: ["health"]

Note: Ensure that the allow_origins setting includes the domain where your frontend application is hosted to avoid CORS issues.

Warning: Enabling RUM can increase the load on your APM Server. Monitor performance after enabling it.

12. Integrating with OpenTelemetry

Component	Description	Configuration
opentelemetry.enabled	Enables OpenTelemetry integration	`true`
opentelemetry.host	Host address for OpenTelemetry collector	`localhost:4317`
opentelemetry.exporter	Exporter type for APM data	`otlp`

Example: OpenTelemetry Collector Configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  elasticsearch:
    hosts: [http://localhost:9200]
    index: apm-7.15.0-transaction
processors:
  batch:
extensions:
  health_check:
  pprof:
  zpages:
service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [elasticsearch]

Note: Ensure the OpenTelemetry collector is configured to export data to APM Server using the elasticsearch exporter.

Warning: The otlp exporter must be enabled in APM Server to receive data from OpenTelemetry.