Using Pipeline Aggregations

1. Understanding Pipeline Aggregations

Pipeline aggregations in the ELK Stack are a powerful feature that allows you to perform calculations on the results of other aggregations. Unlike regular aggregations, which operate on document fields, pipeline aggregations work on the output of other aggregations.

Aggregation Type	Description	Use Case
Avg Bucket	Calculates the average of values from a specified metric aggregation.	Compute average performance metrics across time buckets.
Max Bucket	Finds the maximum value from a specified metric aggregation.	Determine peak values in time-series data.
Min Bucket	Finds the minimum value from a specified metric aggregation.	Identify lowest values in time-series data.
Sum Bucket	Calculates the sum of values from a specified metric aggregation.	Aggregate totals across multiple time buckets or groups.
Stats Bucket	Computes statistics (min, max, avg, sum) from a specified metric aggregation.	Get overall statistics for grouped data.
Extended Stats Bucket	Computes extended statistics (std deviation, variance) from a specified metric aggregation.	Analyze variability in grouped data.
Percentiles Bucket	Calculates percentiles from a specified metric aggregation.	Determine performance thresholds (e.g., 95th percentile response times).
Derivative	Calculates the derivative of a metric aggregation.	Analyze rate of change in time-series data.
Moving Average	Computes a moving average over a specified window.	Smooth out fluctuations in time-series data.
Cumulative Sum	Calculates a cumulative sum of values.	Track running totals over time.

Example: Using Avg Bucket to calculate average response time

{
  "aggs": {
    "monthly_response_times": {
      "date_histogram": {
        "field": "timestamp",
        "calendar_interval": "month"
      },
      "aggs": {
        "avg_response_time": {
          "avg": {
            "field": "response_time"
          }
        }
      }
    },
    "avg_monthly_response": {
      "avg_bucket": {
        "buckets_path": "monthly_response_times>avg_response_time"
      }
    }
  }
}

Note: Pipeline aggregations can only be applied to the results of other aggregations, not directly on fields.

Warning: When using pipeline aggregations, ensure that the referenced aggregation exists and has valid data to avoid errors.

2. Using Bucket Script

The bucket_script aggregation in the ELK Stack is used to perform calculations on the results of other aggregations. It allows you to reference metrics from previous aggregations and compute new values based on them.

Property	Description	Values
buckets_path	Defines the path to the metrics or aggregations to use in the calculation.	`metrics`
script	Defines the script to execute for computing the result.	`script`
format	Optional. Specifies the format for the output value.	`string`

Example: Using Bucket Script to Calculate Average

{
  "aggs": {
    "avg_rating": {
      "avg": {
        "field": "rating"
      }
    },
    "bucket_avg": {
      "bucket_script": {
        "buckets_path": {
          "avgRating": "avg_rating"
        },
        "script": "params.avgRating * 2"
      }
    }
  }
}

3. Using Derivative Aggregation

Property	Description	Values
derivative	Calculates the derivative of a metric aggregation	`derivative`
unit	Time unit for the derivative calculation	`second`, `minute`, `hour`, `day`
missing	Value to use when a document does not have a value for the field	`0`, `null`, or a numeric value

Example: Derivative Aggregation

{
  "aggs": {
    "derivative_agg": {
      "derivative": {
        "buckets_path": "metric_agg"
      }
    }
  }
}

Note: The derivative aggregation requires a metric aggregation to compute the derivative from.

Warning: Derivative aggregation is not supported for percentiles and cardinality aggregations.

4. Using Cumulative Sum

Property	Description	Values	Notes
buckets_path	Path to the metrics field	`field_name`	Required
format	Formatting for the output value	`string`	Optional
missing	Value to use if the field is missing	`number`	Optional

Example: Cumulative Sum Aggregation

{
  "aggs": {
    "sales_sum": {
      "cumulative_sum": {
        "buckets_path": "sales"
      }
    }
  }
}

Note: The cumulative_sum aggregation is useful for tracking running totals over time series data.

5. Using Moving Average

Property	Description	Values	Notes
moving_avg	Aggregation to calculate the moving average over a set of values	`moving_avg`	Requires a `field` or `script` to compute the average
window	Number of documents to include in the moving average calculation	`10`	Defaults to 10, can be adjusted for different window sizes
gap_policy	How to handle gaps in data when calculating the moving average	`skip`, `insert_zero`	Defaults to `skip`, can be set to `insert_zero`
buckets_path	Path to the metric or aggregation used in the moving average calculation	`sales`	Must point to a numeric aggregation or field

Example: Moving Average over Sales Data

{
  "aggs": {
    "sales_ma": {
      "moving_avg": {
        "buckets_path": "sales",
        "window": 5
      }
    }
  }
}

Note: The moving_avg aggregation is useful for smoothing out fluctuations in time-series data to identify trends.

6. Using Serial Diff

Property	Description	Values	Notes
serial_diff	Computes the difference between the current value and the previous value in a series	`field`, `lag`	Requires a sorted time series data set

Example: Calculating Serial Difference

{
  "aggs": {
    "serial_diff": {
      "serial_diff": {
        "field": "cpu_usage"
      }
    }
  }
}

Note: The serial_diff aggregation is useful for detecting anomalies in time-series data by comparing consecutive values.

7. Using Bucket Selector

The bucket_selector aggregation is used to filter buckets based on a condition applied to the metrics of those buckets. It's particularly useful for filtering out buckets that don't meet certain criteria, such as a minimum threshold.

Property	Description	Values
buckets_path	Path to the metrics used in the condition	`metrics_path`
script	Script to evaluate the condition	`script`
filter_path	Optional path to filter buckets	`filter_path`

Example: Filtering buckets with bucket_selector

{
  "aggs": {
    "sales_over_time": {
      "date_histogram": {
        "field": "timestamp",
        "calendar_interval": "month"
      },
      "aggs": {
        "total_sales": {
          "sum": {
            "field": "amount"
          }
        },
        "high_sales_buckets": {
          "bucket_selector": {
            "buckets_path": {
              "totalSales": "total_sales"
            },
            "script": "params.totalSales > 10000"
          }
        }
      }
    }
  }
}

Note: The bucket_selector aggregation is a pipeline aggregation and must be used in conjunction with other aggregations to define the buckets that are being filtered.

8. Using Bucket Sort

Property	Description	Default	Notes
bucket_sort	Sorts buckets by a specified metric	None	Must be used with a bucket aggregation

Example: Sorting Buckets by Count

{
  "aggs": {
    "categories": {
      "terms": {
        "field": "category"
      },
      "aggs": {
        "bucket_sort": {
          "sort": [
            {
              "_count": {
                "order": "desc"
              }
            }
          ],
          "size": 5
        }
      }
    }
  }
}

Example: Sorting Buckets by Custom Metric

{
  "aggs": {
    "sales_by_region": {
      "terms": {
        "field": "region"
      },
      "aggs": {
        "avg_sales": {
          "avg": {
            "field": "sales"
          }
        },
        "bucket_sort": {
          "sort": [
            {
              "avg_sales": {
                "order": "desc"
              }
            }
          ],
          "size": 10
        }
      }
    }
  }
}

Note: The bucket_sort aggregation is useful for limiting the number of buckets returned and sorting them by a metric or field. It can be applied to any bucket aggregation such as terms, range, or date_histogram.

9. Using Sum Bucket

Property	Description	Default	Notes
sum_bucket	Aggregation that sums the values of a specified metric field across all buckets in a pipeline	None	Used for computing totals across grouped data

Example: Sum Bucket Aggregation

{
  "aggs": {
    "sales_over_time": {
      "date_histogram": {
        "field": "timestamp",
        "calendar_interval": "month"
      },
      "aggs": {
        "total_sales": {
          "sum": {
            "field": "amount"
          }
        }
      }
    },
    "total_sales_sum": {
      "sum_bucket": {
        "buckets_path": "sales_over_time>total_sales"
      }
    }
  }
}

Note: The buckets_path parameter specifies the path to the metric field to be summed across buckets.

10. Using Avg Bucket

Property	Description	Default	Notes
buckets_path	Path to the metrics aggregation to compute the average from	Required	Must point to a bucket aggregation or a metrics aggregation
format	Format to apply to the result	null	Can be used to format numeric output (e.g., `0.00`)
missing	Value to use when a bucket is missing	0	Used when a bucket does not exist in the aggregation

Example: Avg Bucket Aggregation

{
  "aggs": {
    "avg_bucket_agg": {
      "avg_bucket": {
        "buckets_path": "sales_per_month>sales"
      }
    }
  }
}

Note: The buckets_path uses a dot notation to reference the aggregation and its metric (e.g., sales_per_month>sales).

11. Using Stats Bucket

Property	Description	Values	Notes
stats_bucket	Aggregation to compute statistics over a set of named metric values from previous aggregations	`object`	Used in pipeline aggregations to calculate stats over bucket values

Example: Using stats_bucket aggregation

{
  "aggs": {
    "sales_per_quarter": {
      "terms": {
        "field": "quarter"
      },
      "aggs": {
        "total_sales": {
          "sum": {
            "field": "sales"
          }
        }
      }
    },
    "stats_bucket_agg": {
      "stats_bucket": {
        "buckets_path": "sales_per_quarter>total_sales"
      }
    }
  }
}

Note: The buckets_path uses a path expression to reference the metric value from a previous aggregation.

Stats Bucket Aggregation Summary

Calculates statistics (min, max, avg, sum) over a set of bucket values
Requires a buckets_path to point to the metric values
Can be used in combination with other pipeline aggregations