Skip to main content
Mapagam

54 SECTIONS · 1573 TOPICS

Prometheus Roadmap

  1. Installing and Configuring Prometheus

    1. 1. Downloading Prometheus
    2. Example: Extracting Prometheus
    3. 2. Setting Up Configuration File
    4. Example: Basic Prometheus Configuration
    5. 3. Running Prometheus Server
    6. Example: Starting Prometheus with default configuration
    7. Example: Running Prometheus with custom retention and storage path
    8. 4. Configuring Storage Path
    9. Example: Configuring Storage Path
    10. 5. Setting Web Listen Address
    11. Example: Listen on specific IP and port
    12. Example: Listen on all interfaces with custom port
    13. 6. Configuring External URL
    14. Example: Setting External URL
    15. 7. Setting Log Level and Format
    16. Example: Configuring Log Level and Format
    17. 8. Validating Configuration
    18. Example: Validate Prometheus configuration file
    19. 9. Running as Service
    20. Example: Systemd Service Configuration
    21. Service Setup Steps
    22. 10. Using Command-Line Help
    23. Example: Viewing Prometheus Help
    24. 11. Enabling Feature Flags
    25. Example: Enabling Feature Flags
    26. 12. Upgrading Prometheus Versions
    27. Example: Upgrading Prometheus
    28. Upgrade Considerations

  2. Configuring Global Settings

    1. 1. Setting Scrape Interval
    2. Example: Configuring scrape intervals
    3. 2. Setting Evaluation Interval
    4. Example: Configuring Evaluation Interval
    5. 3. Configuring Scrape Timeout
    6. Example: Setting Scrape Timeout
    7. 4. Setting Up External Labels
    8. Example: Configuring External Labels in Prometheus
    9. 5. Configuring Query Log File
    10. Example: Query Log Configuration
    11. 6. Setting Query Timeout
    12. Example: Setting Query Timeout in Prometheus Configuration
    13. 7. Configuring Max Samples
    14. Example: Setting Max Samples in Prometheus Configuration
    15. 8. Setting Query Concurrency
    16. Example: Configuring Query Concurrency
    17. 9. Configuring Lookback Delta
    18. Example: Setting lookback_delta in configuration
    19. 10. Configuring Rule Files Path
    20. Example: rule_files configuration
    21. 11. Setting Alertmanager Configuration
    22. Example: Alertmanager Configuration File
    23. 12. Understanding Global Defaults
    24. Example: Default Configuration

  3. Using Web UI and Expression Browser

    1. 1. Accessing Web Interface
    2. Example: Starting Prometheus with default UI
    3. 2. Using Expression Browser
    4. Example: Querying a metric
    5. Example: Using functions
    6. Example: Filtering with labels
    7. 3. Viewing Targets Status
    8. Example: Querying Target Status
    9. 4. Checking Service Discovery
    10. Example: Service Discovery Configuration
    11. 5. Viewing Configuration
    12. Example: Basic Configuration File
    13. 6. Checking Rule Status
    14. Example: Querying rule status
    15. 7. Viewing Active Alerts
    16. Example: Alert Rule Expression
    17. 8. Using TSDB Status Page
    18. Example: Accessing TSDB Status Page
    19. TSDB Status Page Overview
    20. 9. Checking Runtime Information
    21. Example: Fetching Build Information
    22. Example: Fetching Runtime Info
    23. 10. Using Graph Visualization
    24. Example: Basic Graph Query
    25. Example: Query with Labels
    26. Steps to Visualize Metrics
    27. Graph vs. Table View
    28. 11. Viewing Command-Line Flags
    29. Example: Starting Prometheus with custom flags
    30. 12. Accessing Health Endpoints
    31. Example: Checking Health Endpoints

  4. Understanding Prometheus Metric Types

    1. 1. Working with Counter Metrics
    2. Example: Incrementing a Counter
    3. Example: Counter with Labels
    4. 2. Working with Gauge Metrics
    5. Example: Setting a Gauge Value
    6. Example: Incrementing a Gauge
    7. Example: Decrementing a Gauge
    8. 3. Working with Histogram Metrics
    9. Example: Histogram Configuration
    10. Example: Querying Histogram Quantiles
    11. Key Points About Histograms
    12. 4. Working with Summary Metrics
    13. Example: Creating a Summary Metric
    14. Summary Metric Properties
    15. 5. Understanding Metric Naming Conventions
    16. Example: Naming Convention Best Practices
    17. 6. Using Metric Labels
    18. Example: Label Usage in Prometheus Query
    19. Example: Adding Labels to a Counter Metric
    20. 7. Understanding Reserved Labels
    21. 8. Understanding Metric Exposition Format
    22. Example: Counter exposition
    23. Example: Histogram exposition
    24. 9. Working with Timestamps
    25. Example: Setting Custom Timestamps
    26. 10. Using Metric Help Text
    27. Example: Defining Help Text in a Metric
    28. 11. Choosing Histogram vs Summary
    29. Example: Histogram Usage
    30. Example: Summary Usage
    31. Decision Guide
    32. 12. Understanding Counter Resets
    33. Example: Counter Reset Simulation

  5. Working with Labels and Metadata

    1. 1. Creating Metric Labels
    2. Example: Adding Labels to a Metric
    3. 2. Understanding Label Naming Conventions
    4. Example: Valid Label Names
    5. 3. Using Target Labels
    6. Example: Adding Target Labels in Prometheus Configuration
    7. 4. Understanding Instrumentation Labels
    8. Example: Instrumenting a Counter with Labels
    9. 5. Using Meta Labels
    10. Example: Querying with Meta Labels
    11. Example: Filtering by Instance
    12. 6. Handling High-Cardinality Labels
    13. Example: Label Cardinality Issue
    14. Strategies to Reduce Label Cardinality
    15. 7. Implementing Label Inheritance
    16. Example: Adding Inherited Labels to Metrics
    17. Label Inheritance Best Practices
    18. 8. Using Label Matchers
    19. Example: Using Label Matchers in Queries
    20. Example: Excluding Specific Jobs
    21. 9. Understanding Label Precedence
    22. Example: Label Precedence in Action
    23. Example: Metric with Label Precedence
    24. 10. Dropping Unnecessary Labels
    25. Example: Dropping labels via relabeling
    26. 11. Adding Static Labels
    27. Example: Adding static labels via configuration
    28. 12. Using External Labels
    29. Example: Configuring External Labels in Prometheus
    30. 11. Adding Static Labels
    31. Example: Adding static labels via configuration
    32. 12. Using External Labels
    33. Example: Configuring External Labels in Prometheus

  6. Working with OpenMetrics Format

    1. 1. Understanding OpenMetrics Specification
    2. Example: Basic Metric Format
    3. 2. Scraping OpenMetrics Endpoints
    4. Example: Basic Scrape Configuration
    5. Example: Scrape with Relabeling
    6. 3. Using Typed Metrics
    7. Example: Counter Usage in a Web Server
    8. Example: Histogram with Buckets
    9. 4. Understanding Metric Units
    10. Example: Using Units in Prometheus Metrics
    11. 5. Working with Exemplars
    12. Example: Adding Exemplars to Histograms
    13. 6. Using Info Metrics
    14. Example: Info Metric Definition
    15. 7. Understanding StateSet Metrics
    16. Example: StateSet Metrics in Prometheus
    17. Key Points About StateSet Metrics
    18. 8. Using Histogram Buckets
    19. Example: Histogram with Custom Buckets
    20. Example: Histogram with Default Buckets
    21. 9. Understanding EOF Marker
    22. Example: EOF Marker in Metrics Stream
    23. 10. Configuring Accept Header
    24. Example: Setting Accept Header in cURL
    25. 11. Implementing Created Timestamps
    26. Example: Adding Created Timestamp to a Metric
    27. 12. Understanding OpenMetrics vs Prometheus Format

  7. Configuring Scrape Jobs

    1. 1. Defining Static Targets
    2. Example: Basic Static Target Configuration
    3. 2. Setting Job-Specific Scrape Interval
    4. Example: Configuring scrape interval for a job
    5. 3. Configuring Metrics Path
    6. Example: Configuring Metrics Path
    7. 4. Setting Scrape Scheme
    8. Example: Configuring HTTP Scrape Scheme
    9. Example: Configuring HTTPS Scrape Scheme
    10. 5. Adding Job Labels
    11. Example: Adding Job Labels
    12. 6. Setting Scrape Timeout
    13. Example: Configuring Scrape Timeout
    14. 7. Enabling Sample Limit
    15. Example: Configuring Sample Limit in Prometheus Configuration
    16. 8. Configuring Label Limits
    17. Example: Setting Label Limits in Configuration
    18. 9. Using Params for Scraping
    19. Example: Configuring params in scrape job
    20. 10. Configuring Honor Labels
    21. Example: Configuring honor_labels
    22. 11. Configuring Honor Timestamps
    23. Example: Setting honor_timestamps to false
    24. 12. Setting Body Size Limit
    25. Example: Configuring Body Size Limit

  8. Implementing Service Discovery

    1. 1. Using Static Discovery
    2. Example: Static Configuration
    3. 2. Configuring File-Based Discovery
    4. Example: File-Based Discovery Configuration
    5. 2. Target File Format
    6. Example: Target File Content
    7. 2. Reloading Configuration
    8. Workflow: Reloading Configuration
    9. 3. Setting Up DNS Discovery
    10. Example: DNS Service Discovery Configuration
    11. 4. Implementing Consul Discovery
    12. Example: Service Registration in Consul
    13. Example: Prometheus Configuration for Consul SD
    14. Consul Service Discovery Configuration
    15. 5. Using Kubernetes Discovery
    16. Example: Service Discovery for Kubernetes Endpoints
    17. 6. Configuring EC2 Discovery
    18. Example: EC2 Discovery Configuration
    19. 7. Setting Up Azure Discovery
    20. Example: Azure SD Configuration
    21. Configuration Summary
    22. 8. Using GCE Discovery
    23. Example: GCE Service Discovery Configuration
    24. Key Points
    25. 9. Implementing Docker Discovery
    26. Example: Docker SD Configuration
    27. 10. Configuring OpenStack Discovery
    28. Example: OpenStack SD Configuration
    29. 11. Using HTTP Service Discovery
    30. Example: HTTP SD Configuration
    31. Example: HTTP SD Response
    32. 12. Understanding Refresh Intervals
    33. Example: Configuring refresh interval

  9. Working with Relabeling

    1. 1. Understanding Relabel Actions
    2. Example: Replace action usage
    3. Example: Keep action usage
    4. Example: Drop action usage
    5. 2. Using source_labels
    6. Example: Relabeling with source_labels
    7. 3. Configuring target_label
    8. Example: Basic target_label configuration
    9. 4. Setting Regex Patterns
    10. Example: Relabeling with Regex
    11. 5. Using Replacement Strings
    12. Example: Using Capture Groups in Replacement
    13. 6. Implementing Keep Action
    14. Example: Keep action configuration
    15. 7. Implementing Drop Action
    16. Example: Drop Metrics Based on Label Value
    17. 8. Using labelmap Action
    18. Example: Using labelmap to rename metrics
    19. 9. Implementing labeldrop Action
    20. Example: Dropping internal labels
    21. 10. Using labelkeep Action
    22. Example: Using labelkeep to filter metrics by job name
    23. 11. Understanding hashmod Action
    24. Example: Using hashmod to distribute metrics
    25. 12. Configuring Metric Relabeling
    26. Example: Replace label value using regex
    27. Example: Drop metrics based on label value
    28. Example: Hashmod for load balancing

  10. Implementing Authentication

    1. 1. Configuring Basic Authentication
    2. Example: Basic Auth Configuration
    3. 2. Using Bearer Token Authentication
    4. Example: Configuring Bearer Token Authentication
    5. 3. Setting Up TLS Encryption
    6. Example: Prometheus TLS Configuration
    7. 4. Configuring Client Certificates
    8. Example: Prometheus Configuration with Client Certificates
    9. 5. Setting CA Certificate
    10. Example: Configuring CA Certificate for TLS
    11. 6. Implementing Server Name Verification
    12. Example: Configuring Server Name Verification
    13. 7. Using OAuth2 Authentication
    14. Example: OAuth2 Configuration
    15. 8. Configuring Web Authentication
    16. Example: Basic Web Authentication Setup
    17. Example: Using Password File
    18. 9. Setting Up User Credentials
    19. Example: Configuring User Credentials
    20. 10. Using Service Account Tokens
    21. Example: Configuring bearer token in Prometheus config
    22. Example: Using bearer token file
    23. 11. Securing Remote Write
    24. Example: Basic Auth for Remote Write
    25. Example: Bearer Token for Remote Write
    26. 12. Implementing API Security
    27. Example: Setting up JWT Authentication

  11. Hardening Prometheus Security

    1. 1. Enabling TLS for Web Interface
    2. Example: TLS Configuration File
    3. 2. Restricting Network Access
    4. Example: Secure Prometheus Configuration
    5. Network Access Best Practices
    6. 3. Disabling Admin API Endpoints
    7. Example: Disabling Admin API Endpoints
    8. 4. Securing TSDB Admin Endpoints
    9. Example: Protecting TSDB Admin Endpoints with Basic Auth
    10. 5. Using Reverse Proxy Authentication
    11. Example: Nginx Configuration with Basic Auth
    12. 6. Configuring HTTPS Scraping
    13. Example: HTTPS scraping with TLS config
    14. 7. Implementing mTLS
    15. Example: mTLS Configuration in Prometheus
    16. 8. Securing Alertmanager Communication
    17. Example: Alertmanager TLS Configuration
    18. 9. Using Secrets Management
    19. Example: Using Environment Variables for Secrets
    20. Best Practices for Secret Management
    21. 10. Auditing Prometheus Access
    22. Example: Audit Log Entry Format
    23. 11. Implementing Role-Based Access
    24. Example: RBAC Configuration in Prometheus
    25. 12. Securing Configuration Files
    26. Example: Secure prometheus.yml Permissions

  12. Querying with PromQL Selectors

    1. 1. Selecting Metrics by Name
    2. Example: Selecting a metric by name
    3. Example: Selecting a metric with label filters
    4. 2. Using Label Matchers
    5. Example: Matching by Label Value
    6. Example: Using Regex Matchers
    7. 3. Selecting Instant Vectors
    8. Example: Selecting a metric with labels
    9. Example: Using regex to match labels
    10. 4. Selecting Range Vectors
    11. Example: Selecting a Range Vector with Time Range
    12. Example: Selecting a Range Vector with Offset
    13. Example: Selecting a Range Vector with Step
    14. 5. Using Offset Modifier
    15. Example: Comparing Current and Historical Data
    16. 6. Implementing @ Modifier
    17. Example: Using @ Modifier with a Timestamp
    18. Example: Using @ Modifier with Time Range
    19. 7. Understanding Time Durations
    20. Example: Using time durations in queries
    21. Example: Time duration in range vectors
    22. 8. Combining Multiple Matchers
    23. Example: Combining Matchers
    24. 9. Using Regex Matchers
    25. Example: Selecting Metrics with Regex
    26. Example: Excluding Metrics with Regex
    27. 10. Selecting All Series
    28. Example: Select all series with the metric name
    29. Example: Select all time series in the system
    30. Example: Select all time series with the metric name
    31. 11. Understanding Vector Types
    32. Example: Instant Vector
    33. Example: Range Vector
    34. 12. Using Negative Offset
    35. Example: Comparing Current and Historical Data

  13. Using PromQL Aggregation Operators

    1. 1. Calculating Sum
    2. Example: Sum of HTTP Requests
    3. Example: Sum by Method
    4. 2. Computing Average
    5. Example: Average HTTP Request Duration
    6. Example: Average Over Time Window
    7. 3. Finding Minimum and Maximum
    8. Example: Using min() and max() with grouping
    9. 4. Counting Series
    10. Example: Using count
    11. Example: Using count_values
    12. 5. Computing Standard Deviation
    13. Example: Using stddev_over_time
    14. 6. Using Group Operator
    15. Example: Grouping by Job
    16. 7. Implementing topk and bottomk
    17. Example: Using topk to identify the highest HTTP request counts
    18. Example: Using bottomk to identify the lowest HTTP request counts
    19. 8. Using count_values
    20. Example: Counting HTTP Request Values
    21. 9. Aggregating by Labels
    22. Example: Aggregating by Labels
    23. 10. Aggregating without Labels
    24. Example: Using sum without labels
    25. Example: Using avg without labels
    26. 11. Combining Aggregation Operators
    27. Example: Combining sum and avg
    28. Example: Using count and max together
    29. 12. Understanding Aggregation Behavior
    30. Example: Aggregation with grouping
    31. Example: Aggregation with quantile

  14. Working with PromQL Rate Functions

    1. 1. Calculating Rate
    2. Example: Calculating rate over 5 minutes
    3. 2. Using irate for Instant Rate
    4. Example: Using irate for instant rate
    5. 3. Computing Increase
    6. Example: Calculating increase over 1 hour
    7. 4. Computing Delta
    8. Example: Calculating delta over 10 minutes
    9. 5. Using idelta for Instant Delta
    10. Example: Using idelta for instant delta
    11. 6. Predicting Linear Values
    12. Example: Predicting linear growth over 1 hour
    13. 7. Understanding Rate vs irate
    14. Rate vs irate
    15. 8. Handling Counter Resets
    16. 9. Choosing Range Duration
    17. 10. Using Rate with Aggregations
    18. Example: Aggregated rate calculation
    19. 11. Calculating Per-Second Rates
    20. 12. Avoiding Rate Pitfalls
    21. Common Pitfalls

  15. Using PromQL Functions

    1. 1. Calculating Histogram Quantile
    2. 2. Using Rounding Functions
    3. Example: Using ceil() to round up
    4. Example: Using floor() to round down
    5. Example: Using round() for nearest integer
    6. 3. Implementing Clamping
    7. 4. Working with Time Functions
    8. Example: Using time() to calculate duration
    9. Example: Filtering by hour of day
    10. 5. Using Sorting Functions
    11. Example: Sorting metrics by value
    12. Example: Sorting metrics by label value
    13. 6. Implementing Vector Matching
    14. Example: Vector Matching with Labels
    15. 7. Computing Absolute Values
    16. Example: Using abs() function
    17. 8. Using Label Functions
    18. Example: Using label_replace to Normalize Hostnames
    19. Example: Using label_join to Combine Environment and Service
    20. 9. Using Mathematical Functions
    21. Example: Using abs() to handle negative values
    22. Example: Using ceil() to round up response times
    23. 10. Implementing Exponential Functions
    24. 11. Using Trigonometric Functions
    25. 12. Understanding sgn Function
    26. Example: Using sgn in a Query

  16. Using PromQL Over-Time Functions

    1. 1. Calculating Average Over Time
    2. Example: Using avg_over_time to calculate average CPU usage over time
    3. 2. Finding Min and Max Over Time
    4. 3. Computing Sum Over Time
    5. 4. Counting Samples Over Time
    6. Example: Using count_over_time()
    7. Example: Using count_values()
    8. 5. Using Quantile Over Time
    9. Example: Calculating 95th Percentile Over Time
    10. 6. Computing Standard Deviation Over Time
    11. 7. Finding Last Value Over Time
    12. Example: Finding Last Value Over Time
    13. 8. Finding First Value Over Time
    14. Example: Using first_over_time()
    15. 9. Using Absent Over Time
    16. Example: Detecting Missing Metrics
    17. 10. Calculating Changes Over Time
    18. Example: Calculating Request Rate
    19. Example: Tracking Counter Resets
    20. 11. Detecting Resets Over Time
    21. Example: Using increase() to handle counter resets
    22. Example: Detecting resets with resets()
    23. 12. Computing Derivative Over Time

  17. Using PromQL Special Functions

    1. 1. Calculating Time Deltas
    2. Example: Using delta() to calculate total requests over time
    3. Example: Using increase() to handle counter resets
    4. 2. Using Vector Constructors
    5. 3. Implementing Scalar Conversion
    6. Example: Using scalar() to calculate uptime
    7. 4. Using Changes Function
    8. Example: Using changes() to count value changes in a time window
    9. 5. Implementing Resets Detection
    10. Example: Detecting Counter Resets with
    11. Example: Using
    12. 6. Using Deriv Function
    13. Example: Calculating Derivative of Counter
    14. 7. Using Absent Functions
    15. 8. Implementing Day of Week
    16. Example: Extracting Day of Week from a Metric
    17. 9. Using Hour of Day
    18. Example: Hour of Day in a Query
    19. 10. Calculating Year and Month
    20. 11. Implementing Days in Month
    21. Example: Calculating Days in a Month
    22. 12. Using Day of Month

  18. Implementing Binary Operators

    1. 1. Using Arithmetic Operators
    2. 2. Implementing Comparison Operators
    3. 3. Using Logical Operators
    4. 4. Understanding Vector Matching
    5. 5. Implementing Many-to-One Matching
    6. Example: Many-to-One Matching
    7. 6. Implementing One-to-Many Matching
    8. Example: One-to-Many Matching
    9. 7. Using on Clause
    10. Example: Using
    11. 8. Using ignoring Clause
    12. Example: Using
    13. 9. Understanding Operator Precedence
    14. 10. Combining Multiple Operators
    15. Example: Combining Multiple Operators
    16. 11. Using Bool Modifier
    17. Example: Using
    18. 12. Implementing Comparison Filtering
    19. Example: Comparison Filtering

  19. Working with Set Operators

    1. 1. Using Set Operators
    2. Example: Using group_left
    3. 2. Implementing Intersection
    4. Example: Intersection using on()
    5. 3. Using Union
    6. Example: Union of Metrics
    7. 4. Implementing Difference
    8. Example: Difference using unless
    9. 5. Combining with Vector Matching
    10. Example: Vector Matching with group_left
    11. 6. Using Set Operators for Filtering
    12. Example: Filtering with unless
    13. 7. Implementing Alert Correlation
    14. Example: Alert Correlation using group_left
    15. 8. Using Unless for Exclusion
    16. Example: Exclusion using unless
    17. 9. Combining Multiple Conditions
    18. Example: Combining Conditions
    19. 10. Optimizing Set Operations
    20. Example: Optimized Query
    21. 11. Implementing Series Presence Checks
    22. Example: Presence Check using on()
    23. 12. Using Vector Matching with Sets
    24. Example: Vector Matching with Sets

  20. Working with PromQL Subqueries

    1. 1. Understanding Subquery Syntax
    2. Example: Basic Subquery
    3. Example: Subquery with Step
    4. 2. Implementing Nested Aggregations
    5. Example: Nested Aggregation with Rate
    6. Example: Aggregating Over Time and Labels
    7. 3. Using Subqueries for Smoothing
    8. Example: Smoothing with rate() and window
    9. Example: Using subqueries for trend analysis
    10. 4. Calculating Moving Averages
    11. Example: 5-minute moving average of HTTP requests
    12. Example: 1-hour average of HTTP requests
    13. 5. Implementing Derivative Calculations
    14. Example: Derivative of Counter Metric
    15. Example: Instantaneous Delta
    16. 6. Using Subqueries with Functions
    17. Example: Using rate() with subqueries
    18. Example: Using increase() with subqueries
    19. Example: Using delta() with subqueries
    20. Example: Using deriv() with subqueries
    21. Example: Using clamp_min() with subqueries
    22. Example: Using clamp_max() with subqueries
    23. 7. Configuring Step Duration
    24. Example: Setting Step Duration in Subquery
    25. 8. Optimizing Subquery Performance
    26. Example: Optimized Subquery
    27. 9. Combining Subqueries with Offset
    28. Example: Using offset with rate
    29. 10. Debugging Subquery Results
    30. Example: Using
    31. Debugging Steps
    32. 11. Using Default Step Interval
    33. Example: Default Step Interval in Subquery
    34. 12. Implementing Multi-Level Subqueries
    35. Example: Multi-Level Aggregation
    36. Example: Nested Subqueries with Histograms

  21. Implementing Advanced PromQL Patterns

    1. 1. Calculating Percentage Change
    2. Example: Percentage Change in HTTP Requests
    3. 2. Implementing Anomaly Detection
    4. Example: Detecting CPU Usage Anomalies
    5. Example: Detecting Network Traffic Spikes
    6. 3. Using Moving Averages
    7. Example: Calculating a 5-Minute Moving Average
    8. Example: Using Rate with a 10-Minute Window
    9. 4. Calculating Ratio Metrics
    10. Example: Calculating Error Rate
    11. Example: Error Rate with Label Filtering
    12. 5. Implementing Threshold Detection
    13. Example: Detecting High CPU Usage
    14. Example: Detecting Absent Metrics
    15. Threshold Detection Best Practices
    16. 6. Using Absent for Alerting
    17. Example: Alerting on Missing Metrics
    18. Example: Alerting on Missing Metrics with Time Range
    19. 7. Calculating Request Rates
    20. Example: Calculating Request Rate Per Second
    21. Example: Using irate for Alerting
    22. 8. Implementing SLO Queries
    23. Example: Basic SLO Query Using Counters
    24. Example: SLO Query with Latency Threshold
    25. SLO Query Components
    26. 9. Using Quantile Functions
    27. Example: Calculating Percentiles
    28. 10. Calculating Saturation Metrics
    29. Example: Calculating CPU Saturation
    30. 11. Implementing Cardinality Queries
    31. Example: Identifying High Cardinality Metrics
    32. Example: Monitoring Label Values
    33. 12. Using Label Functions for Grouping
    34. Example: Using label_replace()
    35. Example: Using label_join()

  22. Using HTTP API Endpoints

    1. 1. Querying Instant Vectors
    2. Example: Querying a metric at a specific time
    3. Example: Querying a metric with default timestamp
    4. Example: Response format (JSON)
    5. 2. Querying Range Vectors
    6. Example: Querying Range Vectors
    7. Example: Matrix Response Format
    8. 3. Getting Series Metadata
    9. Example: Querying Series Metadata
    10. 4. Finding Label Values
    11. Example: Retrieving Label Values
    12. Example: Retrieving Label Values with Time Range
    13. 5. Getting All Label Names
    14. Example: Fetching All Label Names
    15. Example: Response Format
    16. 6. Querying Targets
    17. 7. Querying Rules
    18. Example: Querying Rules
    19. Example: Querying with Time
    20. Example: Querying with Timeout
    21. 8. Querying Alerts
    22. Example: Fetching Active Alerts
    23. Example: Fetching Alerts Grouped by Labels
    24. 9. Checking Health
    25. Example: Checking Health Status
    26. 10. Getting Configuration
    27. Example: Fetching Configuration
    28. 11. Getting Runtime Info
    29. Example: Fetching Runtime Info
    30. Example: Response Format
    31. 12. Managing TSDB
    32. Example: Delete Series
    33. Example: Snapshot TSDB

  23. Configuring Recording Rules

    1. 1. Creating Rule Groups
    2. Example: Basic Rule Group Configuration
    3. 2. Defining Recording Rules
    4. Example: Basic Recording Rule
    5. 3. Setting Evaluation Interval
    6. Example: Configuring Evaluation Interval
    7. 4. Adding Labels to Rules
    8. Example: Adding Labels to Recording Rule
    9. 5. Organizing Rule Files
    10. Example: Rule file structure
    11. Example: Including rule files in main config
    12. 6. Naming Recording Rules
    13. Example: Naming Convention
    14. 7. Using Recording Rules for Performance
    15. Example: Recording Rule for CPU Usage
    16. Benefits of Recording Rules
    17. 8. Implementing Multi-Level Recording
    18. Example: Aggregating CPU Usage by Instance
    19. Multi-Level Recording Benefits
    20. 9. Validating Rule Syntax
    21. Example: Basic Rule Group Validation
    22. Example: Invalid Rule Expression
    23. 10. Monitoring Rule Evaluation
    24. Example: Rule Evaluation Configuration
    25. Rule Evaluation Process
    26. 11. Testing Recording Rules
    27. Example: Test recording rules with Prometheus console
    28. 12. Implementing Rule Dependencies
    29. Example: Rule Group with Dependency

  24. Configuring Alerting Rules

    1. 1. Defining Alert Rules
    2. Example: Basic Alert Rule Definition
    3. 2. Setting Alert Duration
    4. Example: Setting Alert Duration
    5. 3. Adding Alert Labels
    6. Example: Alert Rule with Labels
    7. 4. Adding Alert Annotations
    8. Example: Alert Rule with Annotations
    9. 5. Using Templates in Annotations
    10. Example: Template in Annotation
    11. 6. Implementing Alert Severity
    12. Example: Alert Rule with Severity
    13. 7. Creating Alert Dependencies
    14. Example: Alert Dependency Using Recording Rules
    15. 8. Configuring Alert Thresholds
    16. Example: Static Threshold Alert Rule
    17. Example: Dynamic Threshold Using Rate
    18. 9. Using Absent Function
    19. Example: Detecting Missing Metrics
    20. 10. Validating Alert Rules
    21. Example: Basic Alert Rule Validation
    22. Example: Using promtool to Validate Rules
    23. Validation Steps
    24. 11. Testing Alert Expressions
    25. Example: Testing Alert Expression
    26. Testing Steps
    27. 12. Implementing Alerting Best Practices
    28. Key Principles
    29. Example: Alert Rule with Severity
    30. Example: Inhibition Rule

  25. Working with Alertmanager Integration

    1. 1. Configuring Alertmanager URL
    2. Example: Alertmanager URL Configuration
    3. 2. Setting Up Static Alertmanagers
    4. Example: Static Alertmanager Configuration
    5. 3. Using Service Discovery for Alertmanagers
    6. Example: File-based Service Discovery
    7. Example: Kubernetes Service Discovery
    8. 4. Configuring Alert Relabeling
    9. Example: Relabeling job name
    10. Example: Dropping labels
    11. 5. Setting Alert Timeout
    12. Example: Configuring Alert Timeout
    13. 6. Configuring Alert Queue Capacity
    14. Example: Setting Alert Queue Capacity
    15. 7. Using External Labels for Routing
    16. Example: Alertmanager Configuration with External Labels
    17. 8. Implementing Alert Batching
    18. Example: Alertmanager Configuration for Batching
    19. 9. Configuring Basic Auth
    20. Example: Basic Auth Configuration
    21. 10. Setting Up TLS
    22. Example: Alertmanager TLS Configuration
    23. 11. Monitoring Alert Delivery
    24. Example: Configuring Alertmanager URL
    25. Alert Delivery Monitoring
    26. 12. Implementing Alert Deduplication
    27. Example: Alertmanager Configuration with Deduplication
    28. 5. Setting Alert Timeout
    29. Example: Configuring Alert Timeout
    30. 6. Configuring Alert Queue Capacity
    31. Example: Setting Alert Queue Capacity
    32. 7. Using External Labels for Routing
    33. Example: Alertmanager Configuration with External Labels
    34. 8. Implementing Alert Batching
    35. Example: Alertmanager Configuration for Batching
    36. 9. Configuring Basic Auth
    37. Example: Basic Auth Configuration
    38. 10. Setting Up TLS
    39. Example: Alertmanager TLS Configuration
    40. 11. Monitoring Alert Delivery
    41. Example: Configuring Alertmanager URL
    42. Alert Delivery Monitoring
    43. 12. Implementing Alert Deduplication
    44. Example: Alertmanager Configuration with Deduplication

  26. Configuring Notification Templates

    1. 1. Using Template Variables
    2. Example: Alert Template with Variables
    3. 2. Implementing Conditional Logic
    4. Example: Using Go Templates for Conditional Logic
    5. Example: Nested Conditional Logic in Alert Templates
    6. 3. Using Range Loops
    7. Example: Iterating Over Labels
    8. Example: Processing Alert Values
    9. 4. Formatting Values
    10. Example: Using Value Formatting in Templates
    11. 5. Accessing External Labels
    12. Example: Using external labels in a template
    13. 6. Using Template Functions
    14. Example: Using Template Functions in Alertmanager Configuration
    15. 7. Implementing Multi-Line Templates
    16. Example: Multi-line template with line breaks
    17. Example: Template with multi-line message
    18. 8. Adding URLs to Annotations
    19. Example: Adding URL to Alert Annotations
    20. 9. Using Custom Template Functions
    21. Example: Defining a Custom Template Function
    22. 10. Testing Templates
    23. Example: Template Validation
    24. Example: Alert Template
    25. Template Testing Workflow
    26. 11. Implementing Runbook URLs
    27. Example: Alertmanager Configuration with Runbook URL
    28. 12. Using Dashboard Links
    29. Example: Dashboard URL Template
    30. Example: Dashboard Link in Alert

  27. Working with Exporters

    1. 1. Understanding Exporter Pattern
    2. Example: Exporter Configuration
    3. 2. Using Node Exporter
    4. Example: Starting Node Exporter with Custom Configuration
    5. Key Features of Node Exporter
    6. 3. Using Blackbox Exporter
    7. Example: Configuring a Blackbox Exporter job
    8. Example: Probe configuration for DNS queries
    9. 4. Integrating Pushgateway
    10. Example: Pushing Metrics to Pushgateway
    11. 5. Configuring Custom Exporters
    12. Example: Running a Custom Exporter
    13. 6. Setting Exporter-Specific Parameters
    14. Example: Configuring Exporter Parameters
    15. 7. Using Exporter Labels
    16. Example: Adding Labels to Metrics
    17. 8. Implementing Textfile Collector
    18. Example: Textfile Collector Configuration
    19. Example: Textfile Format
    20. 9. Monitoring Exporter Health
    21. Example: Health Check Endpoint
    22. Example: Health Status Response
    23. 10. Handling Exporter Timeouts
    24. Example: Configuring Scrape Timeout
    25. Timeout Handling Best Practices
    26. 11. Using Community Exporters
    27. Example: Configuring Prometheus to Scrape Node Exporter
    28. 12. Implementing Multi-Target Exporters
    29. Example: Configuring Multi-Target Exporter
    30. Key Points

  28. Using Pushgateway

    1. 1. Installing Pushgateway
    2. Example: Download and Run Pushgateway
    3. 2. Pushing Metrics to Gateway
    4. Example: Pushing Metrics to Pushgateway
    5. 3. Using Client Libraries
    6. Example: Pushing Metrics from Go
    7. Example: Pushing Metrics from Python
    8. 4. Configuring Job and Instance Labels
    9. Example: Setting Job and Instance Labels via CLI
    10. Example: Setting Labels via HTTP PUT Request
    11. 5. Deleting Metrics from Gateway
    12. Example: Delete a metric by name
    13. Example: Delete a metric by job and name
    14. 6. Configuring Persistence
    15. Example: Prometheus Configuration with Persistence Settings
    16. 7. Scraping Pushgateway from Prometheus
    17. Example: Basic Pushgateway Scrape Configuration
    18. Example: Pushgateway Scrape with Filtering
    19. 8. Understanding Push vs Pull Trade-offs
    20. Pushgateway vs Direct Pull
    21. 9. Implementing Batch Job Metrics
    22. Example: Pushing Metrics from a Batch Job
    23. 10. Using Pushgateway with Cron Jobs
    24. Example: Pushing Metrics from a Cron Job
    25. 11. Avoiding Pushgateway Pitfalls
    26. Example: Proper Labeling with Pushgateway
    27. 12. Monitoring Pushgateway Health
    28. Example: Checking Pushgateway Health

  29. Implementing Custom Metrics

    1. 1. Using Client Libraries
    2. Example: Go Client Library
    3. Example: Python Client Library
    4. 2. Creating Counter Metrics
    5. Example: Registering a Counter in Go
    6. 3. Creating Gauge Metrics
    7. Example: Creating a Gauge Metric
    8. 4. Creating Histogram Metrics
    9. Example: Histogram Metric Definition
    10. Example: Recording Histogram Values
    11. 5. Creating Summary Metrics
    12. Example: Creating a Summary Metric in Go
    13. 6. Exposing Metrics Endpoint
    14. Example: Basic Metrics Endpoint Setup
    15. 7. Using Metric Registry
    16. Example: Registering a Counter Metric
    17. Example: Registering a Histogram Metric
    18. 8. Implementing Metric Labels
    19. Example: Adding Labels to a Counter Metric
    20. 9. Following Naming Conventions
    21. 10. Using Pushgateway for Batch Jobs
    22. Example: Pushing Metrics to Pushgateway
    23. 11. Implementing Metric Instrumentation
    24. Example: Registering a Counter Metric
    25. Example: Registering a Histogram Metric
    26. 12. Testing Custom Metrics
    27. Example: Unit Test for Custom Metric
    28. Testing Best Practices

  30. Working with Kubernetes Integration

    1. 1. Using Kubernetes Service Discovery
    2. Example: Service Discovery Configuration
    3. 2. Discovering Pods
    4. Example: Querying Pod Metrics
    5. 3. Discovering Services
    6. Example: ServiceMonitor Configuration
    7. 4. Discovering Endpoints
    8. Example: Querying for label values
    9. Example: Discovering series matching a label
    10. 5. Discovering Nodes
    11. Example: Service Discovery Configuration
    12. 6. Using Ingress Discovery
    13. Example: Ingress Discovery Configuration
    14. Key Features of Ingress Discovery
    15. 7. Implementing Pod Annotations
    16. Example: Pod with Prometheus Annotations
    17. 8. Configuring Namespace Filtering
    18. Example: Filtering Metrics by Namespace
    19. 9. Using Relabeling for K8s Labels
    20. Example: Relabeling configuration for Kubernetes labels
    21. Key Concepts
    22. 10. Implementing Bearer Token Auth
    23. Example: Configuring Bearer Token in Prometheus
    24. 11. Scraping Kubelet Metrics
    25. Example: Prometheus configuration to scrape Kubelet metrics
    26. 12. Using Service Monitors
    27. Example: Basic ServiceMonitor

  31. Configuring Prometheus Operator

    1. 1. Installing Prometheus Operator
    2. Example: Install via Helm
    3. 2. Creating Prometheus Custom Resource
    4. Example: Basic Prometheus CR
    5. 3. Configuring ServiceMonitor Resources
    6. Example: ServiceMonitor CR
    7. 4. Using PodMonitor Resources
    8. Example: PodMonitor CR
    9. 5. Creating PrometheusRule Resources
    10. Example: PrometheusRule CR
    11. 6. Configuring AlertmanagerConfig
    12. Example: AlertmanagerConfig CR
    13. 7. Using ThanosRuler Resources
    14. Example: ThanosRuler CR
    15. 8. Setting Resource Limits
    16. Example: Resource Limits in CR
    17. 9. Configuring Persistent Storage
    18. Example: Persistent Storage in CR
    19. 10. Using Additional Scrape Configs
    20. Example: Additional Scrape Configs
    21. 11. Implementing Remote Write with Operator
    22. Example: Remote Write Configuration
    23. 12. Monitoring Operator Resources
    24. Example: Monitoring Operator

  32. Managing TSDB Storage

    1. 1. Understanding TSDB Directory Structure
    2. Example: TSDB Directory Layout
    3. 2. Configuring Data Retention
    4. Example: Configuring Data Retention
    5. 3. Using Write-Ahead Log
    6. Example: WAL Compression Configuration
    7. 4. Implementing Block Compaction
    8. Example: Compaction Configuration
    9. Compaction Process Overview
    10. 5. Configuring Chunk Encoding
    11. Example: Setting Chunk Compression
    12. 6. Setting Up Memory Mapping
    13. Example: Memory Mapping Configuration
    14. 7. Managing Disk Space
    15. Example: Configuring retention and storage limits
    16. Example: Monitoring disk usage via Prometheus metrics
    17. Workflow: Managing TSDB disk space
    18. Key Points
    19. 8. Implementing Snapshot API
    20. Example: Creating a Snapshot via HTTP API
    21. Example: Listing Snapshots
    22. Example: Deleting a Snapshot
    23. 9. Using Clean Tombstones
    24. Example: Configuring Tombstone Retention
    25. 10. Monitoring Storage Metrics
    26. Example: Querying Storage Metrics
    27. 11. Configuring Out-of-Order Samples
    28. Example: Enabling Out-of-Order Sample Storage
    29. 12. Using WAL Compression
    30. Example: Enabling WAL Compression

  33. Configuring Storage Retention

    1. 1. Setting Retention Time
    2. Example: Configuring Retention Time
    3. 2. Configuring Retention Size
    4. Example: Setting retention size in Prometheus configuration
    5. 3. Setting Max Block Duration
    6. Example: Configuring max_block_duration
    7. 4. Understanding Retention Precedence
    8. Example: Setting retention via CLI flag
    9. Example: Setting retention in config file
    10. Retention Precedence Summary
    11. 5. Calculating Storage Requirements
    12. Example: Estimating Storage for 10M Samples
    13. Storage Calculation Factors
    14. 6. Monitoring Disk Usage
    15. Example: Disk Usage Alert Configuration
    16. 7. Implementing Retention Policies
    17. Example: Setting retention policy via config file
    18. 8. Using Remote Storage for Long-Term
    19. Example: Configuring remote_write
    20. Example: Configuring remote_read
    21. 9. Configuring Block Compaction
    22. Example: Compaction Configuration
    23. 10. Managing Head Block Size
    24. Example: Adjusting Head Block Size
    25. 11. Setting Minimum Block Duration
    26. Example: Configuring min-block-duration
    27. 12. Understanding Tombstone Cleanup
    28. Example: Configuring Tombstone Cleanup

  34. Implementing Remote Write

    1. 1. Configuring Remote Write
    2. Example: Basic remote write configuration
    3. 2. Setting Remote Write URL
    4. Example: Configure remote write URL
    5. 3. Configuring Write Relabeling
    6. Example: Relabeling for Job Name
    7. 4. Setting Queue Configuration
    8. Example: Remote Write Queue Configuration
    9. 5. Implementing Basic Auth
    10. Example: Configuring Basic Auth in Prometheus
    11. 6. Configuring TLS
    12. Example: TLS Configuration in Prometheus Config
    13. 7. Using External Labels
    14. Example: Configuring External Labels in Prometheus
    15. 8. Setting Capacity and Batching
    16. Example: Configuring remote write batching
    17. 9. Configuring Send Exemplars
    18. Example: Enabling Send Exemplars
    19. 10. Implementing Retry Logic
    20. Example: Configuring Retry Logic
    21. 11. Monitoring Remote Write
    22. Example: Configuring Remote Write
    23. Remote Write Monitoring Metrics
    24. 12. Tuning Remote Write Performance
    25. Example: Tuning remote_write settings in Prometheus config
    26. Performance Tuning Tips

  35. Implementing Remote Read

    1. 1. Configuring Remote Read
    2. Example: Basic Remote Read Configuration
    3. Example: Remote Read with Authentication
    4. 2. Setting Remote Read URL
    5. Example: Configuring remote read URL in Prometheus config
    6. 3. Configuring Read Recent
    7. Example: Configuring Read Recent
    8. 4. Using Remote Timeout
    9. Example: Configuring remote read timeout
    10. 5. Implementing Basic Auth
    11. Example: Basic Auth Configuration
    12. 6. Configuring TLS
    13. Example: TLS Configuration in Prometheus Config
    14. 7. Using Required Matchers
    15. Example: Using Required Matchers in Remote Read
    16. 8. Implementing Filter External Labels
    17. Example: Filtering External Labels
    18. 9. Configuring Name Filtering
    19. Example: Name Filtering Configuration
    20. 10. Monitoring Remote Read
    21. Example: Remote Read Configuration
    22. 11. Optimizing Remote Read Performance
    23. Example: Remote Read Configuration
    24. Optimization Tips
    25. 12. Combining Local and Remote Data
    26. Example: Combining Local and Remote Data
    27. Key Considerations

  36. Setting Up Federation

    1. 1. Configuring Federate Endpoint
    2. Example: Federation Configuration
    3. 2. Selecting Metrics for Federation
    4. Example: Selecting Metrics for Federation
    5. Best Practices for Metric Selection
    6. 3. Implementing Hierarchical Federation
    7. Example: Federation Configuration
    8. Key Concepts
    9. 4. Using Cross-Service Federation
    10. Example: Federation Configuration
    11. Example: Remote Write Configuration
    12. Key Points
    13. 5. Configuring Honor Labels
    14. Example: Configuring honor_labels in Prometheus scrape configuration
    15. 6. Setting Scrape Federation Targets
    16. Example: Federation Configuration
    17. 7. Implementing Global View
    18. Example: Federation Configuration
    19. Global View Implementation Steps
    20. 8. Using External Labels
    21. Example: Configuring External Labels in Prometheus
    22. 9. Optimizing Federation Queries
    23. Example: Using PromQL to Reduce Federation Load
    24. Example: Federation Rule with Query Optimization
    25. Query Optimization Tips
    26. 10. Monitoring Federation Performance
    27. Example: Alerting on Remote Storage Failures
    28. 11. Implementing Partial Federation
    29. Example: Federation Configuration
    30. 12. Handling Federation Failures
    31. Example: Configuring Federation with Retry Logic
    32. Failure Handling Best Practices

  37. Using Prometheus Agent Mode

    1. 1. Understanding Agent Mode Purpose
    2. Example: Agent Mode Configuration
    3. 2. Enabling Agent Mode
    4. Example: Enabling Agent Mode in Prometheus Configuration
    5. 3. Configuring Remote Write Endpoint
    6. Example: Configuring remote_write in Prometheus
    7. 4. Understanding Storage Differences
    8. Example: Remote Write Configuration
    9. 5. Using Agent for Edge Deployments
    10. Example: Agent Configuration for Edge Deployment
    11. Key Considerations for Edge Deployments
    12. 6. Configuring WAL in Agent Mode
    13. Example: WAL Configuration in Agent Mode
    14. 7. Implementing Agent Discovery
    15. Example: Kubernetes Service Discovery
    16. 8. Setting Resource Limits
    17. Example: Configuring Resource Limits
    18. 9. Monitoring Agent Health
    19. Example: Agent Health Check Endpoint
    20. Health Check Configuration
    21. 10. Using Agent with Thanos Receive
    22. Example: Agent Configuration with Thanos Receive
    23. 11. Implementing Agent High Availability
    24. Example: Configuring Prometheus Agent for High Availability
    25. High Availability Setup Steps
    26. 12. Debugging Agent Issues
    27. Example: Agent logs for debugging

  38. Integrating with Thanos

    1. 1. Understanding Thanos Architecture
    2. Thanos Components Overview
    3. Thanos vs Prometheus
    4. 2. Using Thanos Sidecar
    5. Example: Sidecar Configuration
    6. 3. Configuring Thanos Query
    7. Example: Basic Thanos Query Configuration
    8. 4. Implementing Thanos Store Gateway
    9. Example: Store Gateway Configuration
    10. Key Features
    11. 5. Setting Up Thanos Compactor
    12. Example: Basic Thanos Compactor Configuration
    13. 6. Using Thanos Ruler
    14. Example: Basic Thanos Ruler Configuration
    15. Key Features of Thanos Ruler
    16. 7. Configuring Thanos Receive
    17. Example: Basic Thanos Receive Configuration
    18. 8. Implementing Object Storage
    19. Example: S3 Configuration
    20. Key Considerations
    21. 9. Using Thanos Query Frontend
    22. Example: Configuring Thanos Query Frontend
    23. Key Features
    24. 10. Configuring Deduplication
    25. Example: Enabling Deduplication
    26. 11. Implementing Multi-Cluster Monitoring
    27. Example: Configuring Thanos for Multi-Cluster Setup
    28. Key Steps for Multi-Cluster Setup
    29. 12. Monitoring Thanos Components
    30. Example: Querying Thanos Component Metrics

  39. Integrating with Grafana

    1. 1. Adding Prometheus Data Source
    2. Example: Add Prometheus as a data source in Grafana
    3. 2. Configuring Authentication
    4. Example: Basic Authentication Configuration
    5. Example: JWT Authentication Configuration
    6. Example: OAuth2 Authentication Configuration
    7. 3. Using PromQL in Grafana
    8. Example: Basic Query in Grafana
    9. Example: Aggregation with Grouping
    10. Example: Alerting with PromQL
    11. Key PromQL Functions in Grafana
    12. 4. Creating Dashboards
    13. Example: Basic Dashboard Configuration
    14. Dashboard Creation Steps
    15. 5. Implementing Template Variables
    16. Example: Using Template Variables in Grafana Dashboard
    17. Template Variable Configuration Options
    18. 6. Using Exemplar Integration
    19. Example: Adding Exemplars to Metrics
    20. Key Points
    21. 7. Configuring Alert Rules
    22. Example: Basic Alert Rule Configuration
    23. Example: Alert with Multiple Conditions
    24. Example: Alert with Recording Rule
    25. 8. Implementing Annotations
    26. Example: Annotation in Prometheus Rule
    27. 9. Using Explore Mode
    28. Example: Querying metrics in Explore mode
    29. Explore vs. Dashboard Panels
    30. 10. Importing Community Dashboards
    31. Example: Importing a Dashboard via UI
    32. 11. Configuring Recording Rules in Grafana
    33. Example: Recording Rule Configuration
    34. 12. Implementing Dashboard Provisioning
    35. Example: Grafana Dashboard Configuration
    36. Example: Dashboard JSON Definition

  40. Implementing Cardinality Management

    1. 1. Monitoring Series Count
    2. Example: Querying Series Count
    3. 2. Identifying High-Cardinality Labels
    4. Example: Monitoring label cardinality in Prometheus
    5. Best Practices for Label Cardinality
    6. 3. Using Relabeling to Drop Labels
    7. Example: Dropping Unwanted Labels
    8. 4. Implementing Label Limits
    9. Example: Configuring label limits in Prometheus
    10. 5. Configuring Sample Limits
    11. Example: Configuring sample limits in Prometheus configuration
    12. 6. Using Metric Relabeling
    13. Example: Rename a Label
    14. Example: Drop Metrics Based on Label Value
    15. 7. Dropping Unwanted Metrics
    16. Example: Using Relabeling to Drop Metrics
    17. 8. Implementing Target Relabeling
    18. Example: Replace Address Label
    19. Example: Drop Metrics Based on Label
    20. 9. Monitoring Cardinality Growth
    21. Example: Alerting on High Cardinality
    22. 10. Using TSDB Status Page
    23. Example: Accessing TSDB Status Page
    24. 11. Optimizing Label Design
    25. Example: Label Design for HTTP Metrics
    26. 12. Avoiding Cardinality Explosions
    27. Example: Label values with high cardinality
    28. Best Practices

  41. Optimizing Query Performance

    1. 1. Using Recording Rules
    2. Example: Recording Rule Configuration
    3. 2. Limiting Query Range
    4. Example: Limiting Query Range
    5. 3. Avoiding High-Cardinality Queries
    6. Example: High-cardinality query
    7. Example: Optimized query
    8. 4. Using Efficient Selectors
    9. Example: Efficient Selector Usage
    10. 5. Implementing Query Timeouts
    11. Example: Setting Query Timeout in Prometheus Configuration
    12. 6. Configuring Max Samples
    13. Example: Setting max_samples in Prometheus configuration
    14. 7. Using Subqueries Efficiently
    15. Example: Efficient Subquery with Aggregation
    16. Example: Range Vector Subquery
    17. 8. Avoiding Regex Overuse
    18. Example: Inefficient Regex Usage
    19. Example: Efficient Alternative
    20. 9. Implementing Result Caching
    21. Example: Using Prometheus Caching with Grafana
    22. Best Practices for Caching
    23. 10. Monitoring Query Performance
    24. Example: Using Prometheus UI to Monitor Query Performance
    25. Example: Querying Query Duration Histogram
    26. Query Performance Monitoring Best Practices
    27. 11. Using Lookback Delta
    28. Example: Configuring lookback_delta
    29. 12. Optimizing Aggregations

  42. Implementing Performance Tuning

    1. 1. Optimizing Scrape Interval
    2. Example: Configuring scrape interval
    3. 2. Configuring Evaluation Interval
    4. Example: Setting Evaluation Interval
    5. 3. Tuning Storage Retention
    6. Example: Setting retention policy
    7. 4. Implementing Recording Rules
    8. Example: Recording Rule for HTTP Request Rate
    9. 5. Optimizing Label Cardinality
    10. Example: High Cardinality Label (Not Recommended)
    11. Example: Optimized Label Usage
    12. Label Cardinality Optimization Tips
    13. 6. Configuring Memory Limits
    14. Example: Setting Memory Limits
    15. 7. Using WAL Compression
    16. Example: Enabling WAL Compression
    17. Example: Using Zstd Compression (v2.35+)
    18. 8. Implementing Query Concurrency
    19. Example: Configuring Query Concurrency
    20. 9. Tuning Compaction Settings
    21. Example: Compaction Configuration
    22. 10. Optimizing Remote Write
    23. Example: Basic Remote Write Configuration
    24. Example: Optimized Queue Configuration
    25. 11. Using Chunk Encoding
    26. Example: Configuring chunked encoding in Prometheus
    27. 12. Monitoring Resource Usage
    28. Example: Querying CPU Usage
    29. Example: Querying Memory Usage

  43. Monitoring Prometheus Itself

    1. 1. Tracking Scrape Duration
    2. Example: Querying Scrape Duration
    3. 2. Monitoring Sample Ingestion
    4. Example: Configuring Sample Ingestion
    5. 3. Checking WAL Corruption
    6. Example: Running WAL Integrity Check
    7. 4. Monitoring Rule Evaluation
    8. Example: Rule Evaluation Configuration
    9. Example: Recording Rule
    10. Example: Alerting Rule
    11. 5. Tracking Query Performance
    12. Example: Querying for slow queries
    13. Example: Monitoring query sample count
    14. Query Performance Best Practices
    15. 6. Monitoring Storage Size
    16. Example: Setting Storage Retention Size
    17. Storage Size Monitoring Tips
    18. 7. Checking Scrape Failures
    19. Example: Checking scrape failures via Prometheus UI
    20. Example: Querying scrape failures in Prometheus
    21. Example: Alerting on scrape failures
    22. 8. Tracking Remote Write
    23. Example: Remote Write Configuration
    24. Remote Write Key Features
    25. 9. Monitoring Memory Usage
    26. Example: Memory Usage Configuration
    27. 10. Checking Compaction Status
    28. Example: Querying Compaction Metrics
    29. 11. Tracking Time Series Count
    30. Example: Querying Series Count
    31. Example: Monitoring Series Growth
    32. 12. Monitoring Alert Queue
    33. Example: Alert Queue Metrics

  44. Debugging and Troubleshooting

    1. 1. Checking Prometheus Logs
    2. Example: Log levels and configuration
    3. Example: Log file configuration
    4. Common Log Messages
    5. 2. Using Promtool for Validation
    6. Example: Validate a Prometheus Rule File
    7. Example: Validate a Prometheus Configuration File
    8. Example: Validate a Rule File with Custom Rules
    9. 3. Debugging Scrape Failures
    10. Example: Configuring scrape timeout and interval
    11. Common Scrape Failure Causes
    12. Scrape Failure Troubleshooting Steps
    13. 4. Investigating Target Status
    14. Example: Querying target status via Prometheus API
    15. Example: Filtering targets by job name
    16. Common Target Status Issues
    17. 5. Debugging Relabeling
    18. Example: Relabeling Configuration
    19. Debugging Steps
    20. Common Relabeling Actions
    21. 6. Using Query Inspector
    22. Example: Inspecting a Query in Prometheus
    23. Steps to Use Query Inspector
    24. Query Inspector vs Direct Query
    25. 7. Debugging Missing Metrics
    26. Example: Checking Metric Exposure
    27. Common Causes of Missing Metrics
    28. 8. Investigating High Memory Usage
    29. Example: Memory Metrics Query
    30. Example: Query Duration Analysis
    31. 9. Debugging Alert Rules
    32. Example: Alert Rule Definition
    33. Example: Querying Alert State
    34. Workflow: Debugging Alert Rule Issues
    35. 10. Using TSDB Status
    36. Example: Querying TSDB Status
    37. 11. Checking Recording Rule Evaluation
    38. Example: Querying Recording Rule Metrics
    39. Example: Checking Rule Evaluation Duration
    40. 12. Debugging Federation Issues
    41. Example: Federation Configuration

  45. Troubleshooting Common Issues

    1. 1. Resolving Out of Memory Errors
    2. Example: Adjusting scrape settings to reduce memory usage
    3. Example: Using metric relabeling to drop high-cardinality metrics
    4. 2. Fixing Failed Scrapes
    5. Example: Configuring Basic Auth for Scraping
    6. 3. Debugging Missing Metrics
    7. Example: Checking Metric Exposure
    8. Common Causes of Missing Metrics
    9. 4. Resolving High Cardinality
    10. Example: Relabeling to Reduce Cardinality
    11. 5. Fixing Slow Queries
    12. Example: Optimizing a Slow Query
    13. 6. Debugging WAL Corruption
    14. Example: Using promtool to check WAL
    15. 7. Resolving Disk Space Issues
    16. Example: Configuring Retention Policy
    17. 8. Fixing Rule Evaluation Failures
    18. Example: Alert Rule with Syntax Error
    19. Example: Valid Alert Rule
    20. Common Causes of Rule Evaluation Failures
    21. Steps to Debug Rule Evaluation Failures
    22. 9. Debugging Remote Write Failures
    23. Example: Configuring remote write with debug logging
    24. Steps to Debug Remote Write Failures
    25. Common Remote Write Issues
    26. 10. Resolving Target Discovery Issues
    27. Example: Kubernetes SD Config
    28. 11. Fixing Alert Routing Problems
    29. Example: Alertmanager Route Configuration
    30. 12. Debugging TLS Connection Issues
    31. Example: TLS Configuration in Prometheus

  46. Using Command-Line Flags

    1. 1. Setting Configuration File
    2. Example: Setting Configuration File
    3. 2. Configuring Storage Path
    4. Example: Setting Storage Path and Retention
    5. 3. Setting Retention Time
    6. Example: Setting 30-day retention
    7. Example: Setting 50GB size limit
    8. 4. Configuring Retention Size
    9. Example: Setting retention size to 50GB
    10. 5. Setting Web Listen Address
    11. Example: Change Web Listen Address
    12. 6. Configuring External URL
    13. Example: Setting External URL
    14. 7. Enabling Feature Flags
    15. Example: Enabling Feature Flags
    16. 8. Setting Query Timeout
    17. Example: Setting Query Timeout
    18. 9. Configuring Max Samples
    19. Example: Setting Max Samples Per Chunk
    20. 10. Setting Log Level
    21. Example: Setting Log Level to Debug
    22. 11. Configuring WAL Compression
    23. Example: Enabling WAL Compression
    24. 12. Setting Max Block Duration
    25. Example: Setting Max Block Duration

  47. Using Promtool CLI

    1. 1. Validating Configuration
    2. 2. Checking Rule Files
    3. Example: Validating a rule file
    4. 3. Testing Rules Against Data
    5. Example: Testing a Rule
    6. 4. Querying Prometheus
    7. Example: Querying rate of HTTP requests
    8. Example: Using Promtool to query a range
    9. 5. Debugging Metrics
    10. 6. Checking Metric Format
    11. Example: Valid Metric Format
    12. Example: Invalid Metric Format
    13. Workflow: Validate Metrics Using Promtool
    14. 7. Importing TSDB Data
    15. Example: Importing TSDB data using promtool
    16. 8. Analyzing TSDB Blocks
    17. Example: Analyze TSDB block
    18. 9. Creating TSDB Snapshots
    19. Example: Creating a TSDB Snapshot
    20. 10. Dumping TSDB Data
    21. Example: Dumping TSDB data to JSON
    22. 11. Running Unit Tests
    23. Example: Test a Prometheus rule file
    24. 12. Benchmarking Queries
    25. Example: Benchmarking a Query

  48. Implementing Backup and Recovery

    1. 1. Creating TSDB Snapshots
    2. Example: Create a Snapshot
    3. 2. Backing Up Data Directory
    4. 3. Using Remote Write for Backup
    5. Example: Remote Write Configuration
    6. 4. Implementing Point-in-Time Recovery
    7. Recovery Steps
    8. 5. Backing Up Configuration Files
    9. Example: Backup Script
    10. 6. Testing Backup Restoration
    11. 7. Automating Backup Schedule
    12. Example: Cron Job
    13. 8. Monitoring Backup Success
    14. Example: Alert Rule
    15. 9. Implementing Disaster Recovery Plan
    16. Disaster Recovery Steps
    17. 10. Using Cloud Storage for Backups
    18. Example: Upload to S3
    19. 11. Implementing Incremental Backups
    20. 12. Validating Restored Data
    21. Example: Query Validation

  49. Configuring High Availability

    1. 1. Running Multiple Prometheus Instances
    2. Example: Configuring HA with two Prometheus instances
    3. 2. Using Thanos for HA
    4. Example: Basic Thanos Setup
    5. Thanos vs Native Prometheus HA
    6. Thanos HA Setup Workflow
    7. 3. Implementing Deduplication
    8. Example: Enabling Deduplication
    9. 4. Configuring Load Balancing
    10. Example: Load Balancer Configuration
    11. 5. Using Remote Storage for HA
    12. Example: Configuring Remote Write
    13. Example: Configuring Remote Read
    14. 6. Implementing Alert Deduplication
    15. Example: Alertmanager Configuration with Deduplication
    16. 7. Setting Up Cross-Region Replication
    17. Example: Remote Write Configuration
    18. Example: Remote Read Configuration
    19. 8. Using Consistent Hashing
    20. Example: Configuring Consistent Hashing
    21. 9. Implementing Health Checks
    22. Example: HTTP Probe Configuration
    23. Example: TCP Probe Configuration
    24. 10. Monitoring HA Setup
    25. Example: HA Setup Overview
    26. HA Monitoring Workflow
    27. HA Modes Comparison
    28. 11. Handling Split-Brain Scenarios
    29. Example: Configuring Quorum in Prometheus
    30. Best Practices for Split-Brain Prevention
    31. 12. Implementing Failover Strategies
    32. Example: Active-Passive Setup

  50. Implementing Sharding and Scaling

    1. 1. Sharding by Hash
    2. Example: Configuring Hash Sharding
    3. 2. Sharding by Target
    4. Example: Configuring sharding by target label
    5. 3. Using Modulus Relabeling
    6. Example: Configuring Modulus Relabeling
    7. 4. Implementing Horizontal Scaling
    8. Example: Configuring Prometheus for Horizontal Scaling
    9. Horizontal Scaling Strategies
    10. Steps to Implement Horizontal Scaling
    11. 5. Configuring Shard Count
    12. Example: Setting Shard Count in Configuration
    13. 6. Using Service Discovery Filtering
    14. Example: Filtering by Label in Prometheus Configuration
    15. 7. Implementing Dynamic Sharding
    16. Example: Dynamic sharding configuration
    17. 8. Balancing Shard Load
    18. Example: Load Balancing Configuration
    19. Load Balancing Best Practices
    20. 9. Monitoring Shard Performance
    21. Example: Querying Head Series Count
    22. Example: Monitoring Compaction Duration
    23. 10. Using Thanos for Scaling
    24. Example: Deploying Thanos with Prometheus
    25. Thanos Scaling Benefits
    26. 11. Implementing Cross-Shard Queries
    27. Example: Cross-shard metric aggregation
    28. Key Considerations for Cross-Shard Queries
    29. 12. Planning Capacity for Shards
    30. Example: Shard Capacity Estimation
    31. Key Considerations

  51. Configuring Exemplars

    1. 1. Understanding Exemplar Concept
    2. Example: Enabling Exemplars in Configuration
    3. 2. Enabling Exemplar Storage
    4. Example: Enabling Exemplar Storage
    5. 3. Configuring Exemplar Retention
    6. Example: Prometheus Configuration with Exemplar Retention
    7. 4. Scraping Exemplars from Targets
    8. Example: Enabling exemplar scraping in Prometheus config
    9. 5. Querying Exemplars via API
    10. Example: Querying exemplars via API
    11. Response Format
    12. 6. Using Exemplars in Histograms
    13. Example: Configuring Histogram with Exemplars
    14. Example: Histogram Metric with Exemplar
    15. 7. Integrating with Tracing Systems
    16. Example: Enabling exemplar storage in Prometheus config
    17. Tracing System Integration Options
    18. Integration Workflow
    19. 8. Configuring Exemplar Limits
    20. Example: Setting exemplar limits in Prometheus configuration
    21. 9. Visualizing Exemplars
    22. Example: Grafana Dashboard with Exemplars
    23. Key Points for Visualization
    24. 10. Monitoring Exemplar Storage
    25. Example: Configuring exemplar storage in Prometheus
    26. 11. Implementing Exemplar Best Practices
    27. Example: Configuring Exemplar Storage
    28. Best Practices for Exemplar Configuration
    29. 12. Using Exemplars for Debugging
    30. Example: Enabling Exemplar Storage
    31. Key Points

  52. Implementing Native Histograms

    1. 1. Understanding Native Histograms
    2. Example: Histogram Configuration
    3. 2. Enabling Native Histograms
    4. Example: Enabling Native Histograms via CLI
    5. 3. Scraping Native Histograms
    6. Example: Configuring Native Histograms in Prometheus
    7. 4. Querying Native Histograms
    8. Example: Calculating Quantiles from Native Histograms
    9. Example: Comparing Histograms with Different Buckets
    10. 5. Using Histogram Operators
    11. Example: Calculating 95th Percentile
    12. Example: Histogram with Custom Buckets
    13. 6. Configuring Bucket Schema
    14. Example: Setting bucket schema in Prometheus configuration
    15. 7. Converting Classic to Native
    16. Example: Converting Classic Histogram to Native
    17. 8. Calculating Aggregations
    18. Example: Calculating Sum and Count
    19. Example: Calculating Quantiles
    20. 9. Implementing Histogram Merging
    21. Example: Merging Histograms in Go
    22. 10. Monitoring Native Histogram Storage
    23. Example: Configuring Native Histogram Storage
    24. 11. Understanding Bucket Resolution
    25. Example: Setting Bucket Resolution in Prometheus
    26. 12. Implementing Custom Bucket Boundaries
    27. Example: Defining Custom Buckets in Prometheus

  53. Configuring Console Templates

    1. 1. Creating Console Template Files
    2. Example: Basic Console Template Structure
    3. 2. Using Template Syntax
    4. Example: Basic Template Syntax
    5. Example: Conditional Rendering
    6. Example: Looping Through Data
    7. 3. Implementing Navigation Links
    8. Example: Navigation Link Configuration
    9. 4. Querying Metrics in Templates
    10. Example: Using query function
    11. Example: Using query_range function
    12. 5. Creating Dynamic Dashboards
    13. Example: Basic Dashboard Configuration
    14. Example: Dashboard Template with Metrics
    15. Example: Using Template Functions
    16. 6. Using Template Libraries
    17. Example: Using Grafana Console Templates
    18. 7. Implementing Graphs
    19. Example: Basic Graph Configuration
    20. Example: Querying Metrics for Graphs
    21. Example: Using Template Variables in Graphs
    22. 8. Formatting Metric Values
    23. Example: Using bytes formatting in a console template
    24. Example: Using duration formatting in a console template
    25. 9. Creating Reusable Components
    26. Example: Define a reusable template component for metrics
    27. 10. Configuring Console Library Path
    28. Example: Setting Console Library Path
    29. 11. Implementing Conditional Rendering
    30. Example: Basic Conditional in Template
    31. 12. Using Table Formatting
    32. Example: Table Formatting in Console Templates

  54. Implementing Best Practices

    1. 1. Following Naming Conventions
    2. Example: Naming Convention in Practice
    3. 2. Designing Low-Cardinality Labels
    4. Example: High Cardinality Label (Avoid)
    5. Example: Low Cardinality Label (Recommended)
    6. Key Principles
    7. 3. Using Instrumentation Libraries
    8. Example: Instrumenting a Go Application
    9. Example: Instrumenting a Python Application
    10. 4. Implementing Four Golden Signals
    11. Example: Alerting on Error Rate
    12. 5. Avoiding Missing Metrics
    13. Example: Labeling best practices to prevent missing data
    14. Best Practices Summary
    15. 6. Using Direct Instrumentation
    16. Example: Instrumenting a Web Server
    17. 7. Implementing Metric Aggregation
    18. Example: Histogram Aggregation
    19. Example: Counter Aggregation
    20. 8. Using Recording Rules Wisely
    21. Example: Recording Rule for CPU Usage
    22. Best Practices for Recording Rules
    23. 9. Following Alert Design Patterns
    24. Example: Alertmanager Silence Configuration
    25. Key Principles for Alert Design
    26. 10. Implementing Runbook Links
    27. Example: Alert with Runbook Link
    28. 11. Using External Labels
    29. Example: Setting External Labels in Prometheus Configuration
    30. 12. Testing Configuration Changes
    31. Example: Testing Prometheus Configuration
    32. Testing Configuration Changes