Configuring Global Settings
1. Setting Scrape Interval
| Property | Value | Description |
|---|---|---|
| scrape_interval | 15s |
How frequently to scrape targets by default |
| scrape_timeout | 10s |
Timeout for scraping a single target |
Example: Configuring scrape intervals
global:
scrape_interval: 15s
scrape_timeout: 10s
evaluation_interval: 15s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
2. Setting Evaluation Interval
| Property | Values | Description |
|---|---|---|
| evaluation_interval | 1m |
How often Prometheus evaluates rules and samples. |
| evaluation_interval | 30s |
Default value for rule evaluation frequency. |
Note: The
evaluation_interval setting affects how frequently Prometheus evaluates recording and alerting rules. It does not affect the scrape interval or the time series data sampling rate.
3. Configuring Scrape Timeout
| Configuration Option | Default Value | Description |
|---|---|---|
| scrape_timeout | 10s |
Timeout for scraping targets. If a scrape takes longer than this, it is considered failed. |
Example: Setting Scrape Timeout
scrape_configs:
- job_name: "prometheus"
scrape_timeout: 15s
static_configs:
- targets: [localhost:9090]
4. Setting Up External Labels
External labels are key-value pairs added to all metrics scraped by Prometheus. They help identify the source of metrics and can be used for filtering and grouping.
| Label | Description | Example |
|---|---|---|
| job | Identifies the job that is being scraped | web-server |
| instance | Identifies the instance being scraped (host:port) | localhost:9090 |
| environment | Environment label to distinguish between dev, staging, prod | production |
Example: Configuring External Labels in Prometheus
global:
external_labels:
job: web-server
environment: production
region: us-west-1
Note: External labels are applied to all metrics and can be overridden by metric-specific labels with the same name.
5. Configuring Query Log File
| Option | Values | Notes |
|---|---|---|
| query.log.file | string |
Path to the file where query logs are written. If not set, query logs are disabled. |
| query.log.level | string |
Log level for query logs. Valid values: info, warn, error. Default: info. |
6. Setting Query Timeout
| Configuration Option | Default Value | Description |
|---|---|---|
| query.timeout | 2m |
Maximum time a query is allowed to run before being cancelled. |
Note: The timeout value can be specified in seconds (e.g.,
30s), minutes (e.g., 2m), or hours (e.g., 1h). Setting a timeout prevents long-running queries from consuming excessive resources.
7. Configuring Max Samples
| Option | Values | Notes |
|---|---|---|
| storage.tsdb.max-samples-per-timeseries | 100000 |
Maximum number of samples per time series stored in the database |
| storage.tsdb.max-block-size | 512MB |
Maximum size of a block in the database |
Example: Setting Max Samples in Prometheus Configuration
storage:
tsdb:
max-samples-per-timeseries: 100000
max-block-size: 512MB
Note: Increasing
max-samples-per-timeseries can lead to higher memory usage and potential performance degradation if not managed properly.
8. Setting Query Concurrency
| Option | Values | Description |
|---|---|---|
| query.max-concurrency | 10 |
Maximum number of concurrent queries allowed |
| query.timeout | 2m |
Maximum time a query can take before being cancelled |
Note: Increasing
query.max-concurrency can improve performance for complex queries but may lead to resource exhaustion under high load.
9. Configuring Lookback Delta
| Property | Default Value | Description |
|---|---|---|
| lookback_delta | 5m |
Duration for which the Prometheus server retains samples in memory for lookback queries. |
Example: Setting lookback_delta in configuration
global:
evaluation_interval: 1m
scrape_interval: 15s
external_labels:
monitor: 'codelab-monitor'
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 15s
metrics_path: /metrics
static_configs:
- targets: ['localhost:9090']
rule_files:
- 'alerts.yml'
remote_write:
- url: 'http://localhost:9090/api/v1/write'
queue_config:
max_samples_per_send: 1000
batch_send_deadline: 5s
remote_read:
- url: 'http://localhost:9090/api/v1/read'
lookback_delta: 10m
Note: The
lookback_delta setting affects how far back in time Prometheus can query for data. Increasing this value allows longer lookback queries but increases memory usage.
10. Configuring Rule Files Path
| Property | Value | Description |
|---|---|---|
| rule_files | ["rules/*.yml"] |
Path glob for rule files to load. Supports wildcards. |
| rule_files.path | rules/*.yml |
Directory or file path to load rules from. |
Note: Rule files must be valid YAML or JSON format and follow Prometheus recording and alerting rules syntax.
11. Setting Alertmanager Configuration
| Property | Description | Default Value | Notes |
|---|---|---|---|
| alertmanager.config | Path to the Alertmanager configuration file | "" |
Must be a valid YAML file path |
| alertmanager.timeout | Timeout for requests to Alertmanager | 10s |
Must be a valid duration string |
Example: Alertmanager Configuration File
global:
resolve_timeout: 5m
route:
group_by: [alertname]
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
receiver: team-a-notifications
receivers:
- name: team-a-notifications
email_configs:
- to: team-a@example.com
from: prometheus@example.com
smarthost: smtp.example.com:587
auth_username: prometheus@example.com
auth_password: "secret"
12. Understanding Global Defaults
| Setting | Default Value | Description |
|---|---|---|
| global.scrape_interval | 15s |
How frequently to scrape targets by default. |
| global.scrape_timeout | 10s |
Timeout for scraping targets. |
| global.evaluation_interval | 15s |
How frequently to evaluate rules. |
| global.external_labels | {} |
Labels to add to all timeseries. |
| global.query.log_file | nil |
File to write query logs to. |