Managing Data Streams

1. Understanding Data Streams

Property	Description	Values
data_stream.type	Defines the type of data stream	`logs`, `metrics`, `traces`
data_stream.dataset	Identifies the dataset within the stream	`myapp.logs`, `nginx.metrics`
data_stream.namespace	Namespace for organizing streams	`production`, `staging`

Example: Defining a Data Stream

data_stream:
  type: logs
  dataset: myapp.logs
  namespace: production

Note: Data streams are managed by the elasticsearch and logstash components of the ELK stack.

2. Creating Data Streams

Example: Create a Data Stream

PUT /_data_stream/my-data-stream
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "message": {
        "type": "text"
      }
    }
  }
}

Property	Description	Default
name	Name of the data stream	None
settings	Stream-specific settings (e.g., shard count, replica count)	Defaults based on index template
mappings	Field mappings for the data stream	Empty

Note: Data streams must be created with a matching index template or they will not function properly.

3. Indexing to Data Stream

Field	Description	Example
index.name	Name of the index to which data is indexed	`my-index-2024`
data_stream.name	Name of the data stream to which data is indexed	`my-data-stream`
data_stream.timestamp_field	Field used for timestamp in the data stream	`@timestamp`

Example: Indexing to a Data Stream

{
  "index": {
    "name": "my-data-stream-2024"
  },
  "data_stream": {
    "name": "my-data-stream",
    "timestamp_field": "@timestamp"
  }
}

Note: Data streams are designed for time-series data and provide automatic index management.

4. Rolling Over Data Stream

Property	Description	Values	Notes
rollover_alias	Alias used to rollover the data stream	`logs-nginx`	Must match the alias defined in the data stream
max_size	Maximum size of the data stream before rollover	`50GB`	Triggers rollover when size is exceeded
max_age	Maximum age of the data stream before rollover	`30d`	Triggers rollover when age is exceeded
max_docs	Maximum number of documents before rollover	`1000000`	Triggers rollover when document count is exceeded

Example: Create a Data Stream with Rollover Configuration

{
  "index_patterns": [
    "logs-nginx-*"
  ],
  "data_stream": {
    "retention": "30d"
  },
  "rollover_alias": "logs-nginx",
  "max_size": "50gb",
  "max_age": "30d",
  "max_docs": 1000000
}

Note: The rollover configuration is defined in the index template and applies to all data streams created from that template.

5. Viewing Data Stream Stats

Stat	Description	Command
data_stream.name	Name of the data stream	`GET /_data_stream`
data_stream.status	Current status of the data stream (green/yellow/red)	`GET /_data_stream?pretty`
data_stream.store_size	Total storage size of the data stream	`GET /_data_stream/_stats`
data_stream.doc_count	Number of documents in the data stream	`GET /_data_stream/_stats?filter_path=.indices..docs.count`
data_stream.index_count	Number of backing indices in the data stream	`GET /_data_stream/_stats?filter_path=.indices..index_count`

Example: Retrieve Data Stream Stats

GET /_data_stream/my-data-stream/_stats

Note: The _stats endpoint provides detailed statistics for each backing index within the data stream.

6. Searching Data Streams

Field	Description	Example
query	Search query to execute against data streams	`{ "match": { "message": "error" } }`
size	Number of documents to return	`100`
from	Starting document offset for pagination	`0`
sort	Sort order for results	`[{ "timestamp": { "order": "desc" } }]`

Example: Basic Search Query

{
  "query": {
    "match": {
      "level": "ERROR"
    }
  },
  "size": 50,
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ]
}

Example: Search with Aggregations

{
  "query": {
    "range": {
      "timestamp": {
        "gte": "2024-01-01T00:00:00Z"
      }
    }
  },
  "aggs": {
    "error_types": {
      "terms": {
        "field": "level"
      }
    }
  }
}

Note: Use from and size for pagination to avoid performance issues with large result sets.

Warning: Avoid using wildcards in queries like *error* as they can be slow and resource-intensive.

7. Using Backing Indices

Property	Description	Values
index.name	Name of the index	`logs-2024.01.01`
index.uuid	Unique identifier for the index	`a1b2c3d4e5f6`
index.creation_date	Date the index was created	`2024-01-01T00:00:00Z`
index.status	Current status of the index	`open`, `close`

Example: Creating a new backing index

PUT /logs-2024.01.02
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  }
}

Example: Updating index settings

PUT /logs-2024.01.01/_settings
{
  "index": {
    "number_of_replicas": 2
  }
}

Note: Backing indices are managed automatically by Elasticsearch when using data streams. Manual creation is generally not required unless specific customizations are needed.

8. Deleting Data Streams

Option	Values	Notes
data_stream.name	`logs-nginx-access`	Name of the data stream to delete
elasticsearch.hosts	`["http://localhost:9200"]`	Hosts of the Elasticsearch cluster

Example: Delete a Data Stream

curl -X DELETE "http://localhost:9200/_data_stream/logs-nginx-access"

Note: Ensure the data stream is not in use before deletion to avoid data loss.

Warning: Deletion is irreversible. Back up data streams before deletion if needed.

9. Promoting Write Index

Property	Description	Value
index.name	Name of the index to promote	`logs-2023.04.01`
index.alias	Alias to promote the index to	`logs-write`
index.lifecycle.name	Index lifecycle policy name	`logs-policy`

Example: Promoting Write Index

{
  "index": "logs-2023.04.01",
  "alias": "logs-write"
}

Note: Ensure that the index is not already in use by other processes before promoting it.

10. Using Data Stream Aliases

Alias Name	Target Data Stream	Description
logs-nginx-access	`nginx-access-logs`	Alias for nginx access logs data stream
metrics-system	`system-metrics`	Alias for system metrics data stream

Example: Creating a Data Stream Alias

PUT /_data_stream/my-logs-alias
{
  "indices": [
    {
      "index": "my-logs-2024.05.01"
    }
  ],
  "alias": "my-logs"
}

Example: Updating Data Stream Alias

POST /_data_stream/my-logs-alias/_alias
{
  "actions": [
    {
      "add": {
        "index": "my-logs-2024.05.02",
        "alias": "my-logs"
      }
    }
  ]
}

Note: Data stream aliases provide a way to manage multiple data streams under a single logical name, useful for querying and managing large volumes of time-series data.

11. Migrating to Data Streams

Note: This section assumes familiarity with Elasticsearch's Index Lifecycle Management (ILM) and basic data stream concepts.

Migration from traditional indices to data streams involves several steps to ensure data continuity and operational consistency. Data streams provide a more robust and scalable approach for managing time-series data.

Step	Description	Action Required
1. Identify Legacy Indices	Locate indices that are part of the legacy index structure and not yet converted to data streams.	Use `GET /_cat/indices?v` to list indices and filter by pattern or age.
2. Evaluate ILM Policies	Review existing ILM policies to ensure they are compatible with data streams or need updates.	Check for policies that manage index lifecycle and update them if necessary.
3. Create New Data Stream	Set up a new data stream with appropriate settings and mappings.	Use `PUT /_data_stream/<data-stream-name>` with desired configuration.
4. Reindex Data	Reindex data from legacy indices into the new data stream.	Use `POST /_reindex` to move data, ensuring no loss or duplication.
5. Update Index Patterns	Adjust index patterns in Kibana and other tools to point to the new data stream.	Modify index patterns in Kibana and update any dashboards or visualizations.
6. Validate Data	Ensure that all data has been successfully migrated and is accessible.	Run queries to compare data counts and verify integrity between old and new structures.

Example: Reindexing Legacy Index to Data Stream

{
  "source": {
    "index": "legacy-logs-2024.01.01"
  },
  "dest": {
    "index": "logs-data-stream"
  }
}

Migration Considerations

Backward Compatibility: Ensure that any applications or tools using the legacy indices can be updated to use data streams.
Performance Impact: Reindexing can be resource-intensive; schedule during low-traffic periods.
Data Integrity: Validate that all data is correctly migrated and no records are lost.
ILM Policy Alignment: Align ILM policies with data stream lifecycle management for automatic rollover and deletion.