Skip to main content
Mapagam

Working with Aggregations

1. Understanding Aggregation Types

Type Description Use Case Example
Metrics Aggregations Calculate metrics over a set of documents, such as min, max, sum, average Statistical analysis of document fields 
{
  "avg_price": {
    "avg": {
      "field": "price"
    }
  }
}
Bucket Aggregations Group documents into buckets based on field values or ranges Segmenting data by categories or time periods 
{
  "sales_by_category": {
    "terms": {
      "field": "category"
    }
  }
}
Pipeline Aggregations Operate on the output of other aggregations, not documents directly Calculating ratios or running stats across previous aggregations 
{
  "avg_monthly_sales": {
    "avg_bucket": {
      "buckets_path": "sales_per_month>sales",
      "gap_policy": "skip"
    }
  }
}
Matrix Aggregations Perform calculations on multiple fields simultaneously Correlation or covariance between fields 
{
  "matrix_stats": {
    "matrix_stats": {
      "fields": ["price", "quantity"]
    }
  }
}

2. Using Metric Aggregations

Aggregation Type Description Example Notes
avg Calculates the average of numeric values 
{
  "aggs": {
    "avg_rating": {
      "avg": {
        "field": "rating"
      }
    }
  }
}
 Supports missing parameter to handle missing values
max Calculates the maximum value in a set of numeric values 
{
  "aggs": {
    "max_price": {
      "max": {
        "field": "price"
      }
    }
  }
}
 Can be used with script for computed values
min Calculates the minimum value in a set of numeric values 
{
  "aggs": {
    "min_price": {
      "min": {
        "field": "price"
      }
    }
  }
}
 Supports missing parameter to handle missing values
sum Calculates the sum of numeric values 
{
  "aggs": {
    "total_sales": {
      "sum": {
        "field": "sales"
      }
    }
  }
}
 Can be used with script for computed values
count Counts the number of documents or non-null values 
{
  "aggs": {
    "doc_count": {
      "value_count": {
        "field": "category"
      }
    }
  }
}
 Can be used with script for computed counts
cardinality Estimates the number of unique values in a field 
{
  "aggs": {
    "unique_users": {
      "cardinality": {
        "field": "user_id"
      }
    }
  }
}
 Uses HyperLogLog algorithm; precision_threshold for accuracy tuning

Example: Aggregating Average Rating by Category

{
  "aggs": {
    "avg_rating_by_category": {
      "terms": {
        "field": "category"
      },
      "aggs": {
        "avg_rating": {
          "avg": {
            "field": "rating"
          }
        }
      }
    }
  }
}
Note: Metric aggregations can be combined with bucket aggregations to group results by categories or other fields.

3. Using Bucket Aggregations

Aggregation Type Description Use Case Example
terms Groups documents by unique values in a field Count occurrences of each value 
{
  "terms": {
    "field": "status"
  }
}
range Groups documents into ranges based on field values Analyze data in buckets by value ranges 
{
  "range": {
    "field": "age",
    "ranges": [
      { "to": 18 },
      { "from": 18, "to": 65 },
      { "from": 65 }
    ]
  }
}
date_histogram Groups documents into time-based buckets Time-series analysis of logs 
{
  "date_histogram": {
    "field": "@timestamp",
    "calendar_interval": "1d"
  }
}
histogram Groups documents into numeric buckets Statistical distribution of numeric fields 
{
  "histogram": {
    "field": "response_time",
    "interval": 100
  }
}

Example: Terms Aggregation

{
  "aggs": {
    "status_counts": {
      "terms": {
        "field": "status"
      }
    }
  }
}

Example: Date Histogram Aggregation

{
  "aggs": {
    "logs_over_time": {
      "date_histogram": {
        "field": "@timestamp",
        "calendar_interval": "1h"
      }
    }
  }
}

4. Using Date Histogram

Property Description Values
field Field to aggregate on timestamp
calendar_interval Interval for grouping dates 1d, 1w, 1M
format Date format for display yyyy-MM-dd

Example: Date Histogram Aggregation

{
  "aggs": {
    "date_histogram": {
      "field": "timestamp",
      "calendar_interval": "1d"
    }
  }
}

5. Using Range Aggregations

Range aggregations in the ELK Stack allow you to group documents into ranges based on numeric or date values. They are particularly useful for analyzing distributions and creating histograms.

Property Description Example
range Defines the ranges to group documents into. Each range has a to and/or from value. 
{
  "range": {
    "field": "age",
    "ranges": [
      { "to": 18 },
      { "from": 18, "to": 35 },
      { "from": 35 }
    ]
  }
}
field The numeric or date field to use for grouping. "field": "price"
to Upper bound (exclusive) for the range. "to": 100
from Lower bound (inclusive) for the range. "from": 50

Example: Aggregating Age Groups

{
  "aggs": {
    "age_ranges": {
      "range": {
        "field": "age",
        "ranges": [
          { "to": 18 },
          { "from": 18, "to": 35 },
          { "from": 35 }
        ]
      }
    }
  }
}

Example: Date Range Aggregation

{
  "aggs": {
    "date_ranges": {
      "range": {
        "field": "timestamp",
        "ranges": [
          { "to": "2023-01-01T00:00:00" },
          { "from": "2023-01-01T00:00:00", "to": "2024-01-01T00:00:00" },
          { "from": "2024-01-01T00:00:00" }
        ]
      }
    }
  }
}

6. Nesting Aggregations

Nesting aggregations in the ELK Stack allows you to build complex analytics by combining multiple aggregation levels. This is particularly useful for multi-dimensional analysis and hierarchical data exploration.

Aggregation Type Description Use Case
Nested Groups documents by a nested field and applies sub-aggregations to that field. When analyzing nested objects within documents (e.g., comments in a blog post).
Children Applies aggregations to child documents in a parent-child relationship. When analyzing relationships between parent and child documents (e.g., orders and order items).
Reverse Nested Reverses the nesting context to access parent documents from nested aggregations. When you need to aggregate on the parent document level after performing nested aggregations.

Example: Nested Aggregation

{
  "aggs": {
    "comments": {
      "nested": {
        "path": "comments"
      },
      "aggs": {
        "comment_count": {
          "value_count": {
            "field": "comments.id"
          }
        },
        "avg_rating": {
          "avg": {
            "field": "comments.rating"
          }
        }
      }
    }
  }
}

Example: Children Aggregation

{
  "aggs": {
    "orders": {
      "children": {
        "type": "order_item"
      },
      "aggs": {
        "total_items": {
          "value_count": {
            "field": "order_item.id"
          }
        },
        "avg_price": {
          "avg": {
            "field": "order_item.price"
          }
        }
      }
    }
  }
}
Note: When nesting aggregations, ensure that the nested or child field is properly mapped in Elasticsearch to avoid performance issues.

7. Using Cardinality Aggregation

The cardinality aggregation is used to calculate the approximate count of distinct values for a field. It's particularly useful for analyzing unique values in large datasets efficiently.

Property Description Default Notes
field The field to calculate cardinality for Required Must be a numeric or string field
missing Value to use if the field is missing null Applicable to numeric fields
precision_threshold Threshold for precision adjustment 100 Higher values increase accuracy but memory usage

Example: Basic Cardinality Aggregation

{
  "aggs": {
    "unique_users": {
      "cardinality": {
        "field": "user_id"
      }
    }
  }
}

Example: Cardinality with Precision Threshold

{
  "aggs": {
    "unique_ips": {
      "cardinality": {
        "field": "ip_address",
        "precision_threshold": 50
      }
    }
  }
}
Note: The cardinality aggregation uses the HyperLogLog++ algorithm, which provides a good balance between accuracy and memory usage. For very large datasets, consider increasing the precision_threshold to reduce estimation errors.

8. Using Percentiles Aggregation

Property Description Values
field Field to calculate percentiles for string
percents Array of percentiles to calculate array
missing Value to use when field is missing number

Example: Basic Percentiles Aggregation

{
  "aggs": {
    "price_percentiles": {
      "percentiles": {
        "field": "price",
        "percents": [25, 50, 75, 95]
      }
    }
  }
}

Example: Percentiles with Custom Field

{
  "aggs": {
    "duration_percentiles": {
      "percentiles": {
        "field": "response_time",
        "percents": [50, 90, 99]
      }
    }
  }
}
Note: Percentiles are useful for understanding data distribution and identifying outliers in performance metrics.

9. Using Top Hits Aggregation

Property Description Values
top_hits Aggregation to retrieve the top documents matching a query {"size": 5, "sort": [{"timestamp": {"order": "desc"}}]}

Example: Retrieving top hits by timestamp

{
  "aggs": {
    "top_by_timestamp": {
      "top_hits": {
        "size": 3,
        "sort": [
          {
            "timestamp": {
              "order": "desc"
            }
          }
        ]
      }
    }
  }
}
Note: The top_hits aggregation returns the actual documents, not just the top values. It is useful for retrieving full context of top matching results.

10. Using Filters Aggregation

Property Description Values
filters Defines the filter conditions to apply {"status": "active"}
aggs Sub-aggregations to run on filtered results {"avg_price": {"avg": {"field": "price"}}}

Example: Applying Filters Aggregation

{
  "aggs": {
    "active_users": {
      "filters": {
        "filters": {
          "status_active": {"term": {"status": "active"}},
          "status_inactive": {"term": {"status": "inactive"}}
        }
      },
      "aggs": {
        "avg_age": {"avg": {"field": "age"}}
      }
    }
  }
}

11. Using Significant Terms

Property Description Values
significant_terms Aggregation to find terms that are "significantly more frequent" in a subset of data compared to the overall dataset field, min_doc_count, shard_min_doc_count, execution_hint

Example: Basic Significant Terms Aggregation

{
  "aggs": {
    "significant_terms": {
      "field": "category.keyword",
      "min_doc_count": 5
    }
  }
}
Note: The significant_terms aggregation uses a statistical algorithm to identify terms that occur more frequently in the filtered data than in the overall dataset.

12. Using Composite Aggregation

Property Description Values Notes
sources List of aggregation sources to composite [{"terms": {"field": "status"}, "aggs": {"avg_duration": {"avg": {"field": "duration"}}}}] Required field
size Number of composite buckets to return 10 Default is 10, max is 10000
after Token to continue from a previous composite aggregation {"status": "200"} Used for pagination

Example: Basic Composite Aggregation

{
  "aggs": {
    "composite_agg": {
      "composite": {
        "sources": [
          {
            "status": {
              "terms": {
                "field": "status"
              }
            }
          }
        ],
        "size": 10
      },
      "aggs": {
        "avg_duration": {
          "avg": {
            "field": "duration"
          }
        }
      }
    }
  }
}

Example: Composite Aggregation with After Token

{
  "aggs": {
    "composite_agg": {
      "composite": {
        "sources": [
          {
            "status": {
              "terms": {
                "field": "status"
              }
            }
          }
        ],
        "size": 10,
        "after": {
          "status": "200"
        }
      },
      "aggs": {
        "avg_duration": {
          "avg": {
            "field": "duration"
          }
        }
      }
    }
  }
}

Was this article helpful?

Related Cheat Sheets