Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.onefirewall.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Elasticsearch index configuration

To setup the PoV elasticsearch index we apply an Index Lifecycle Management (ILM) policy with rollover and automated deletion, gaining several benefit:
  1. Automatic data growth management
    • With rollover (max_age: 1d or max_size: 50gb), you don’t need to manually monitor index size or age.
    • As soon as an index reaches the threshold, Elasticsearch creates a new one (poc_traffic-000002, etc.) and automatically updates the alias poc_traffic.
  2. Better query and update performance
    • Oversized indices slow down searches and updates.
    • By splitting them regularly, shards remain smaller, keeping queries, aggregations, and writes efficient.
  3. Automatic cleanup of old data
    • The delete phase (min_age: 34d) removes indices older than 34 days.
    • No need for external jobs (cron, scripts) to enforce data retention → lower risk of wasting disk space.
  4. Resource usage optimization
    • number_of_shards: 1 and number_of_replicas: 0 reduce overhead when high availability is not required.
    • index.translog.flush_threshold_size: 512mb and refresh_interval: 30s optimize ingestion performance compared to immediate search.
    • Prevents the cluster from being overloaded with either too many small shards or oversized ones.
  5. Easier management with index templates
    • With an index template (poc_traffic_template), each new rollover index automatically inherits the same settings.
    • No need to reapply configurations like refresh_interval or max_result_window manually.
  6. Elasticity and scalability
    • Ideal for time-series data (like logs or traffic data) that continuously grows.
    • The combination of alias + rollover + ILM is the recommended Elastic pattern for scalable data management.
curl -XPUT "http://localhost:9200/_ilm/policy/poc_traffic_policy" -H "kbn-xsrf: reporting" -H "Content-Type: application/json" -d'
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_size": "50gb"
          }
        }
      },
      "delete": {
        "min_age": "34d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}'

curl -XPUT "http://localhost:9200/poc_traffic-000001" -H "kbn-xsrf: reporting" -H "Content-Type: application/json" -d'
{
  "aliases": {
    "poc_traffic": {
      "is_write_index": true
    }
  },
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0,
    "refresh_interval": "30s",
    "index.lifecycle.name": "poc_traffic_policy",
    "index.lifecycle.rollover_alias": "poc_traffic",
    "index.translog.flush_threshold_size": "512mb",
    "max_result_window": 100000
  }
}'

curl -XPUT "http://localhost:9200/_index_template/poc_traffic_template" -H "kbn-xsrf: reporting" -H "Content-Type: application/json" -d'
{
  "index_patterns": ["poc_traffic-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "refresh_interval": "30s",
      "index.lifecycle.name": "poc_traffic_policy",
      "index.lifecycle.rollover_alias": "poc_traffic",
      "index.translog.flush_threshold_size": "512mb",
      "max_result_window": 100000
    }
  },
  "priority": 500
}'