Elasticsearch index configuration
To setup the PoV elasticsearch index we apply an Index Lifecycle Management (ILM) policy with rollover and automated deletion, gaining several benefit:-
Automatic data growth management
- With rollover (max_age: 1d or max_size: 50gb), you don’t need to manually monitor index size or age.
- As soon as an index reaches the threshold, Elasticsearch creates a new one (poc_traffic-000002, etc.) and automatically updates the alias poc_traffic.
-
Better query and update performance
- Oversized indices slow down searches and updates.
- By splitting them regularly, shards remain smaller, keeping queries, aggregations, and writes efficient.
-
Automatic cleanup of old data
- The delete phase (min_age: 34d) removes indices older than 34 days.
- No need for external jobs (cron, scripts) to enforce data retention → lower risk of wasting disk space.
-
Resource usage optimization
- number_of_shards: 1 and number_of_replicas: 0 reduce overhead when high availability is not required.
- index.translog.flush_threshold_size: 512mb and refresh_interval: 30s optimize ingestion performance compared to immediate search.
- Prevents the cluster from being overloaded with either too many small shards or oversized ones.
-
Easier management with index templates
- With an index template (poc_traffic_template), each new rollover index automatically inherits the same settings.
- No need to reapply configurations like refresh_interval or max_result_window manually.
-
Elasticity and scalability
- Ideal for time-series data (like logs or traffic data) that continuously grows.
- The combination of alias + rollover + ILM is the recommended Elastic pattern for scalable data management.