Implementing Index Lifecycle Management

1. Understanding ILM Phases

Phase	Description	Actions	Notes
Hot	Active index with frequent writes and reads	Indexing, searching, and updating	Default phase; optimized for performance
Warm	Index is still accessed but less frequently	Reads, updates, and migration to cold	Reduced shard allocation; optimized for search
Cold	Index is rarely accessed but still searchable	Reads and migration to frozen	Reduced memory usage; optimized for search
Frozen	Index is no longer updated and rarely accessed	Reads only	Optimized for long-term storage; minimal resources
Delete	Index is no longer needed and will be deleted	Deletion of index	Final phase; no further actions

Example: ILM Policy Configuration

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "readonly": {},
          "set_priority": {
            "priority": 50
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

2. Creating ILM Policies

Policy Name	Settings	Description
hot	`{"min_age": "0ms", "rollover": {"max_size": "50gb", "max_age": "30d"}, "set_priority": {"priority": 100}}`	Initial phase where data is actively written to
warm	`{"min_age": "30d", "set_priority": {"priority": 50}, "allocate": {"include": {"box_type": "warm"}}}`	Data is moved here after hot phase
cold	`{"min_age": "90d", "set_priority": {"priority": 0}, "allocate": {"include": {"box_type": "cold"}}}`	Archived data with low access frequency
delete	`{"min_age": "180d", "delete": {}}`	Final phase to delete data

Example: ILM Policy JSON

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "rollover": {
          "max_size": "50gb",
          "max_age": "30d"
        },
        "set_priority": {
          "priority": 100
        }
      },
      "warm": {
        "min_age": "30d",
        "set_priority": {
          "priority": 50
        },
        "allocate": {
          "include": {
            "box_type": "warm"
          }
        }
      },
      "delete": {
        "min_age": "180d",
        "delete": {}
      }
    }
  }
}

Note: ILM policies are managed via the Elasticsearch PUT /_ilm/policy/<policy_name> API endpoint.

3. Configuring Hot Phase

Property	Description	Values
min_age	Time after which the index enters the hot phase	`"0m"` (default)
roaring	Enables roaring bitmap for efficient filtering	`false` (default)
max_size	Maximum size of the index before transitioning to warm phase	`"50gb"` (default)
max_docs	Maximum number of documents before transitioning to warm phase	`-1` (no limit)
max_age	Maximum age of the index before transitioning to warm phase	`"30d"` (default)

Example: Hot Phase Configuration

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0m",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          }
        }
      }
    }
  }
}

4. Configuring Warm Phase

Setting	Description	Values	Notes
min_age	Time after which the index moves to warm phase	`1d`, `7d`	Must be specified in days or milliseconds
max_age	Maximum age before moving to cold phase	`30d`, `90d`	Optional, defaults to never
min_size	Minimum size before moving to warm phase	`50gb`, `100gb`	Optional, defaults to never
warm_phase	Configuration for warm phase actions	`{}`	Can include allocation, routing, and other settings

Example: Warm Phase Configuration

{
  "warm": {
    "min_age": "1d",
    "max_age": "30d",
    "warm_phase": {
      "min_size": "50gb"
    }
  }
}

Note: The warm phase is designed to reduce storage costs by moving older indices to less powerful hardware or storage tiers.

5. Configuring Cold Phase

Property	Description	Default Value	Notes
min_age	Minimum age before transitioning to the cold phase	0ms	Must be specified in ISO 8601 format (e.g., "1d", "7d")
max_size	Maximum size of the index before transitioning to the cold phase	-	Optional; can be used in combination with min_age
max_docs	Maximum number of documents before transitioning to the cold phase	-	Optional; can be used in combination with min_age
min_index_age	Minimum age of the index before transitioning to the cold phase	-	Alternative to min_age, used for more granular control

Example: Cold Phase Configuration

{
  "policy": {
    "phases": {
      "cold": {
        "min_age": "30d",
        "actions": {
          "freeze": {},
          "set_priority": {
            "priority": 10
          }
        }
      }
    }
  }
}

Note: The freeze action is available in Elasticsearch 7.12+ and helps reduce memory usage by freezing the index.

6. Configuring Frozen Phase

Property	Description	Default Value	Notes
freeze	Freezes the index to prevent further writes and reduce resource consumption	false	Can only be set during index creation or via index template
min_index_age	Minimum age before transitioning to frozen phase	0d	Must be specified in days (e.g., `30d`)
max_age	Maximum age before transitioning to frozen phase	0d	Must be specified in days (e.g., `90d`)

Example: Index Template with Frozen Phase Configuration

{
  "index_patterns": ["frozen-logs-*"],
  "settings": {
    "index": {
      "lifecycle": {
        "policy": "frozen-policy"
      }
    }
  },
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "30d"
          }
        }
      },
      "frozen": {
        "min_age": "30d",
        "actions": {
          "freeze": {}
        }
      }
    }
  }
}

Note: Frozen indices can only be read, not written to. They are optimized for long-term storage and querying.

7. Configuring Delete Phase

Property	Description	Default Value	Notes
delete.min_age	Minimum age of the index before it is deleted	0ms	Must be a time value (e.g., 7d, 30m)
delete.indices	List of indices to delete	None	Supports wildcards and aliases
delete.query	Query to match documents before deletion	None	Used for conditional deletion based on document content

Example: Delete Phase Configuration

{
  "delete": {
    "min_age": "30d",
    "indices": [
      "logs-2024.01.*",
      "logs-2024.02.*"
    ],
    "query": {
      "match": {
        "status": "inactive"
      }
    }
  }
}

Note: The delete phase removes indices after they meet the specified conditions. Ensure that data is backed up or archived before configuring this phase to avoid irreversible data loss.

8. Attaching Policies to Indices

Policy	Index Pattern	Attach Method	Notes
hot-warm-delete	`logs-*`	Using Index Templates	Applies to all matching indices
rollover-policy	`logs-*`	Using Index Templates	Applies to indices with size > 50GB

Example: Applying Policy via Index Template

{
  "index_patterns": ["logs-*"],
  "settings": {
    "index": {
      "lifecycle": {
        "policy_name": "hot-warm-delete"
      }
    }
  }
}

Note: The policy must exist before attaching it to indices.

Warning: Attaching a policy to an existing index may not immediately apply lifecycle actions.

9. Using Rollover Actions

Action	Description	Parameters	Notes
rollover	Creates a new index and updates aliases to point to it	`max_age`, `max_docs`, `max_size`	Used to manage index lifecycle by rotating indices based on size or age

Example: Configure rollover policy

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          }
        }
      }
    }
  }
}

Note: The rollover action is part of the Index Lifecycle Management (ILM) policy and can be configured to trigger based on size or age thresholds.

10. Setting Phase Transitions

Phase	Transition Criteria	Notes
hot	`index.lifecycle.waiting_for_migration`	Index is actively being written to and queried
warm	`index.lifecycle.waiting_for_migration`	Index is read-only and optimized for search
cold	`index.lifecycle.waiting_for_migration`	Index is read-only and optimized for archival
delete	`index.lifecycle.waiting_for_migration`	Index is deleted after retention period

Example: Define Phase Transitions

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "readonly": {}
        }
      },
      "cold": {
        "min_age": "60d",
        "actions": {
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

11. Monitoring ILM Status

Property	Description	Value
ilm.status	Current status of ILM policy	`running`, `stopped`, `failed`
ilm.policy_name	Name of the ILM policy in use	`my-policy`
ilm.phase	Current phase of the ILM lifecycle	`hot`, `warm`, `cold`, `delete`

Example: Check ILM status via API

GET /_ilm/status

Example: Check ILM status for a specific index

GET /my-index/_ilm/status

Note: The ILM status endpoint returns information about the current state of ILM policies and phases for indices managed by those policies.

12. Retrying Failed Phases

Phase	Status	Description	Action
warm	NEW	Index is in warm phase	Retry failed operations
cold	DEPRECATED	Index is in cold phase	Migration to new index
delete	NEW	Index is in delete phase	Retry deletion if failed

Example: Retry Configuration

{
  "policy": {
    "phases": {
      "warm": {
        "min_age": "10d",
        "actions": {
          "retry": {
            "count": 3,
            "delay": "1h"
          }
        }
      }
    }
  }
}

Note: Retry configuration is available in Elasticsearch 8.0+.