Implementing Service Discovery

1. Using Static Discovery

Static service discovery in Prometheus involves manually defining targets and their configurations within the Prometheus configuration file. This method is useful for small, stable environments where services do not frequently change.

Field	Description	Example
targets	List of target endpoints to scrape	`["localhost:9090", "localhost:9100"]`
labels	Additional labels to attach to metrics	`{"job": "prometheus", "env": "prod"}`

Example: Static Configuration

scrape_configs:
  - job_name: "static-service"
    static_configs:
      - targets: [&#34;localhost:9090&#34;, &#34;localhost:9100&#34;]
        labels:
          job: &#34;prometheus&#34;
          env: &#34;prod&#34;

Note: Static discovery is not suitable for environments with dynamic scaling or frequent service changes. It requires manual updates to the configuration file when targets change.

2. Configuring File-Based Discovery

File-based service discovery in Prometheus allows you to define targets for scraping via static configuration files. This method is useful when you have a fixed set of services or when you want to manage targets without relying on external service discovery mechanisms.

Configuration File	Description	Example
file_sd_config	Defines the file-based service discovery configuration.	`{ "files": ["targets.json"] }`

Example: File-Based Discovery Configuration

scrape_configs:
  - job_name: "file_sd_job"
    file_sd_configs:
      - files:
          - "targets.json"

Note: The file specified in file_sd_configs must be in JSON format and contain a list of targets.

2. Target File Format

The target file must be a JSON array containing objects with the following fields:

Field	Description	Example
targets	List of target addresses in the format `host:port`.	`["localhost:9090", "prometheus.example.com:9090"]`
labels	Additional labels to apply to the targets.	`{"job": "prometheus", "env": "prod"}`

Example: Target File Content

[
  {
    "targets": ["localhost:9090", "prometheus.example.com:9090"],
    "labels": {
      "job": "prometheus",
      "env": "prod"
    }
  },
  {
    "targets": ["localhost:9100"],
    "labels": {
      "job": "node_exporter",
      "env": "dev"
    }
  }
]

2. Reloading Configuration

Prometheus automatically reloads the target files when they change. You can also trigger a manual reload using the /-/reload endpoint or by sending a SIGHUP signal to the Prometheus process.

Workflow: Reloading Configuration

Modify the target file (e.g., targets.json)
Send a SIGHUP signal to Prometheus: kill -HUP <prometheus_pid>
Prometheus reloads the configuration and starts scraping new targets

3. Setting Up DNS Discovery

Property	Value	Description
dns_sd_configs	`[]`	Configuration for DNS-based service discovery.

Example: DNS Service Discovery Configuration

scrape_configs:
  - job_name: "dns-sd-example"
    dns_sd_configs:
      - names:
          - "prometheus.example.com"
        type: "SRV"
        port: 9090

Note: DNS service discovery uses the SRV record type by default to resolve targets. Ensure DNS records are correctly configured for service discovery.

4. Implementing Consul Discovery

Example: Service Registration in Consul

{
  "ID": "prometheus-01",
  "Name": "prometheus",
  "Tags": [
    "monitoring",
    "production"
  ],
  "Address": "10.0.1.10",
  "Port": 9090,
  "Check": {
    "HTTP": "http://10.0.1.10:9090/metrics",
    "Interval": "10s"
  }
}

Field	Description	Required
ID	Unique identifier for the service instance	Yes
Name	Name of the service	Yes
Tags	List of tags for service categorization	No
Address	IP address of the service instance	Yes
Port	Port number where service is listening	Yes
Check	Health check configuration for service	No

Note: The Consul service registration must be done via HTTP API or using a configuration file.

Example: Prometheus Configuration for Consul SD

scrape_configs:
  - job_name: 'prometheus'
    consul_sd_configs:
      - server: 'http://consul.example.com:8500'
        services:
          - 'prometheus'

Consul Service Discovery Configuration

Register services in Consul with health checks
Configure Prometheus to use Consul SD for service discovery
Use tags to filter services in scrape jobs

5. Using Kubernetes Discovery

Prometheus can automatically discover and monitor services running in a Kubernetes cluster using the built-in kubernetes_sd_config. This feature allows Prometheus to dynamically discover targets based on Kubernetes labels, namespaces, and other metadata.

Field	Description	Example
role	The role of the Kubernetes objects to discover. Supported roles: `endpoints`, `service`, `pod`, `node`, `ingress`.	`endpoints`
namespace	Limit discovery to a specific namespace. If omitted, all namespaces are used.	`default`
label_selector	Label selector to filter discovered objects.	`app=frontend`
field_selector	Field selector to filter discovered objects.	`metadata.name=web-app`

Example: Service Discovery for Kubernetes Endpoints

scrape_configs:
  - job_name: "kubernetes-endpoints"
    kubernetes_sd_configs:
      - role: endpoints
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+?)(?::\d+)?;(\d+)
        replacement: $1:$2

Note: Ensure that Prometheus has the necessary RBAC permissions to access Kubernetes API resources. For example, you may need to configure a ClusterRole and ClusterRoleBinding to allow Prometheus to read endpoints, services, and pods.

Warning: When using kubernetes_sd_config, make sure to configure proper relabeling rules to avoid scraping metrics from unwanted targets or misconfiguring the target address.

6. Configuring EC2 Discovery

Field	Description	Example
role	The AWS role to use for EC2 discovery. Must be either `ec2` or `ecs`.	`ec2`
region	The AWS region to discover EC2 instances in.	`us-east-1`
access_key	The AWS access key ID to use for authentication.	`AKIAIOSFODNN7EXAMPLE`
secret_key	The AWS secret access key to use for authentication.	`wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`
refresh_interval	How often to refresh the list of EC2 instances (default is 30s).	`60s`
port	The port to scrape metrics from (default is 9100).	`9100`

Example: EC2 Discovery Configuration

scrape_configs:
  - job_name: "ec2"
    ec2_sd_configs:
      - region: "us-east-1"
        access_key: "AKIAIOSFODNN7EXAMPLE"
        secret_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
        port: 9100
        refresh_interval: 60s

7. Setting Up Azure Discovery

Example: Azure SD Configuration

scrape_configs:
  - job_name: "azure"
    azure_sd_configs:
      - subscription_id: "your-subscription-id"
        tenant_id: "your-tenant-id"
        client_id: "your-client-id"
        client_secret: "your-client-secret"
        port: 9100

Field	Description	Required	Default
subscription_id	Azure subscription ID	Yes	None
tenant_id	Azure tenant ID (directory ID)	Yes	None
client_id	Service principal client ID	Yes	None
client_secret	Service principal client secret	Yes	None
port	Port to scrape targets on	No	9100

Note: Ensure the service principal has appropriate permissions to read Azure resources.

Warning: Client secret should be stored securely, not in plain text.

Configuration Summary

NEW Requires Azure AD authentication
Supports scraping targets on port 9100 by default
Uses Azure Resource Manager API to discover resources

8. Using GCE Discovery

Prometheus can use Google Compute Engine (GCE) service discovery to automatically discover and monitor services running on Google Cloud Platform.

Configuration Parameter	Description	Default Value
gce_sd_configs	Configures GCE service discovery	Empty list

Example: GCE Service Discovery Configuration

scrape_configs:
  - job_name: "gce-prometheus"
    gce_sd_configs:
      - project: "my-gcp-project"
        zone: "us-central1-a"
        filter: "labels.app = 'prometheus'"
        port: 9090

Note: Ensure that the Prometheus instance has the necessary permissions to access GCE resources. This typically involves setting up a service account with appropriate roles such as roles/compute.viewer.

Warning: The filter parameter uses the GCE filter syntax, which supports complex queries. Be cautious with performance implications when using broad filters.

Key Points

GCE SD allows Prometheus to dynamically discover targets based on GCP labels and metadata.
Requires proper IAM permissions for accessing GCE resources.
Supports filtering using GCP's label-based query syntax.

9. Implementing Docker Discovery

Option	Values	Notes
docker_sd_configs	`[]`	Configuration for Docker service discovery
host	`string`	Docker daemon host address (default: unix:///var/run/docker.sock)
refresh_interval	`duration`	How often to refresh targets (default: 60s)

Example: Docker SD Configuration

scrape_configs:
  - job_name: "docker-app"
    docker_sd_configs:
      - host: "unix:///var/run/docker.sock"
        refresh_interval: 60s

Note: Requires Docker daemon to be accessible and Prometheus to have permissions to read container metadata.

Warning: Ensure Docker socket is properly secured to prevent unauthorized access.

Field	Description	Default
host	Docker daemon host address	`unix:///var/run/docker.sock`
refresh_interval	How often to query Docker for container updates	`60s`
filters	Optional filters to limit containers	`[]`

10. Configuring OpenStack Discovery

Option	Values	Description
sd_openstack	`enabled`	Enables OpenStack service discovery
sd_openstack.role	`instance`, `loadbalancer`	Specifies the OpenStack role to discover (e.g., instance or loadbalancer)
sd_openstack.region	`RegionOne`	OpenStack region to query for services
sd_openstack.tenant_id	`tenant-id`	OpenStack tenant ID to use for authentication
sd_openstack.endpoint_type	`publicURL`, `internalURL`, `adminURL`	Endpoint type to use for OpenStack service URLs
sd_openstack.refresh_interval	`30s`	Interval to refresh service discovery (default 30s)

Example: OpenStack SD Configuration

scrape_configs:
  - job_name: "openstack"
    openstack_sd_configs:
      - role: instance
        region: RegionOne
        tenant_id: "tenant-id"
        endpoint_type: publicURL
        refresh_interval: 30s

Note: Ensure that the OpenStack credentials and permissions are correctly configured for Prometheus to access the service catalog.

11. Using HTTP Service Discovery

Field	Description	Example
targets	List of target endpoints to scrape	`["localhost:9090", "localhost:9091"]`
labels	Additional labels to attach to metrics	`{"job": "prometheus"}`

Example: HTTP SD Configuration

scrape_configs:
  - job_name: "http_sd"
    http_sd_configs:
      - url: "http://localhost:8080/sd"

Example: HTTP SD Response

[{
  "targets": ["localhost:9090", "localhost:9091"],
  "labels": {
    "job": "prometheus",
    "env": "prod"
  }
}]

Note: The HTTP SD endpoint must return a JSON array of service discovery objects.

12. Understanding Refresh Intervals

Property	Value	Description
scrape_interval	`15s`	How often Prometheus scrapes targets
scrape_timeout	`10s`	Timeout for individual scrapes
refresh_interval	`30s`	How often service discovery re-reads targets

Example: Configuring refresh interval

scrape_configs:
  - job_name: "prometheus"
    scrape_interval: 15s
    scrape_timeout: 10s
    refresh_interval: 30s
    static_configs:
      - targets: ['localhost:9090']

Note: The refresh_interval setting determines how frequently Prometheus re-evaluates service discovery configurations to find new targets or remove stale ones. It does not affect how often metrics are scraped from discovered targets.