Wazuh Performance Tuning: EPS, Heap, Shards and Queues

Performance work on Wazuh 5.0 starts by identifying which of five stages is actually the constraint: the HTTPS intake on the manager, the normalisation engine, the indexer connector, indexer ingest, or detector scheduling. They present with different symptoms and they are fixed in different places. The single highest-return change in most deployments is not a tuning parameter at all, it is cutting the volume of events you index, because 5.0 indexes normalised events as a matter of course where 4.x indexed alerts.

What you need

A running deployment with real traffic on it. Tuning against synthetic load teaches you very little about your own decoder and rule mix.
Indexer credentials with read access, and root on the indexer nodes for heap changes.
The layout from Wazuh cluster and distributed architecture if you are past one node, because several of these changes are per node.

Where the 5.0 pipeline actually bottlenecks

Stage	What it does	How it fails under load	Where you fix it
HTTPS intake on 1517	Receives event batches, inventory sessions and control calls	Sheds capacity with HTTP `503`, never `429`	Add manager workers, or reduce what agents send
Normalisation engine	Decoder tree, optional enrichment, filters, output	CPU bound on the manager	Fewer active policies, pre-filters, trimmed collection
Indexer connector	Ships documents to the indexer	Buffers in memory and discards on manager restart	Size the indexer for the rate, do not restart busy managers
Indexer ingest	Writes to data streams	Heap pressure, slow refresh, disk watermarks	Heap, replicas, shard and rollover settings
Detectors	Scheduled Sigma evaluation producing findings	Findings arrive late, not missing	Detector schedule and rule selection

Two things follow from that table. First, if agents are logging 503, the fix is upstream capacity or less traffic, not a bigger queue, because the HTTPS channel does not have one. Second, if findings are late but events are present, nothing is overloaded; you are looking at a detector interval.

Cut the volume before you buy hardware

This section is first on purpose. Everything below it is cheaper once there is less data.

Turn off raw event indexing unless you need it

wazuh-events-raw-v5 stores incoming events before processing and normalisation. It is the closest thing 5.0 has to 4.x archives, and it is the largest single contributor to volume. The Setup module exposes one setting for it:

curl -sk -u <WAZUH_INDEXER_USERNAME>:<WAZUH_INDEXER_PASSWORD> -X PUT \
  "https://127.0.0.1:9200/_plugins/_setup/settings" \
  -H 'Content-Type: application/json' \
  -d '{"engine": {"index_raw_events": false}}'

The documented success response is {"message": "Settings updated successfully.", "status": 200}. Turn it on for an investigation window, turn it off again afterwards. Default retention on that stream is 10 minutes, so it is a short-window buffer by design rather than an archive, but at high EPS it still costs shards and write throughput.

Drop events you will never look at, before decoding

Filters are engine-level content, defined in YAML and stored in the wazuh-threatintel-filters-a index. A pre-filter evaluates events before decoding and discards them immediately if they do not meet its conditions, which avoids the decoding work entirely. A post-filter runs after enrichment or decoding and discards before the outputs. Without a pre-filter every event goes to decoding unconditionally, which is the default.

In terms of return on effort, a pre-filter on a chatty, low-value source beats every heap setting on this page. The candidates are usually the same: health-check chatter from load balancers, a backup agent writing a line per file, and one verbose application that somebody enabled debug logging on and forgot.

Collect less at the agent

The cheapest event is the one never sent. Agent-side <localfile> scope, file integrity monitoring scope and inventory intervals are all configured through the group’s shared agent.conf now, which makes them adjustable per role rather than per host. That is covered in agent groups and centralised configuration and log collection.

Know what each active policy costs

The normalisation engine gives each active policy its own copy of every incoming event, and each copy can produce its own output document. Adding a custom policy alongside the standard one is not free; it roughly doubles the output for the events both policies handle. Enrichment is optional per policy and can be disabled entirely, which matters when you run a policy over a source where Geo and indicator matching add nothing.

Size the indexer

Heap

The documented rule is half of the system RAM, with the minimum and maximum set identically so the JVM never resizes the heap at runtime. In /etc/wazuh-indexer/jvm.options, for an 8 GB host:

-Xms4g
-Xmx4g

Memory locking

A swapping indexer looks exactly like an undersized one, and it is a far more common cause of a slow dashboard than shard counts. Enable locking in /etc/wazuh-indexer/opensearch.yml:

bootstrap.memory_lock: true

Raise the system limit alongside it:

mkdir -p /etc/systemd/system/wazuh-indexer.service.d/

cat > /etc/systemd/system/wazuh-indexer.service.d/wazuh-indexer.conf << EOF
[Service]
LimitMEMLOCK=infinity
EOF

Restart, then confirm:

curl -k -u <INDEXER_USERNAME>:<INDEXER_PASSWORD> "https://<INDEXER_IP_ADDRESS>:9200/_nodes?filter_path=**.mlockall&pretty"

The documented healthy value is "mlockall" : true. If the request fails, Unable to lock JVM Memory appears in /var/log/wazuh-indexer/wazuh-indexer.log.

Disk

The documented per-endpoint figures over 90 days:

Monitored endpoint	EPS	Indexer storage per endpoint, 90 days
Servers	0.25	3.7 GB
Workstations	0.1	1.5 GB
Network devices	0.5	7.4 GB

The worked example in the documentation is 80 workstations, 10 servers and 10 network devices needing 231 GB for 90 days. Use it to sanity-check an estimate, then replace it with your own measurement as soon as you have one, because the spread between a quiet fleet and a noisy one is much wider than the spread between these three categories.

Shards, rollover and retention

The Setup module creates the index templates and Index State Management policies, and manual edits to Wazuh-managed templates can be overwritten when the indexer restarts. The shipped settings on a findings template include "number_of_shards": "1", "number_of_replicas": "0", "auto_expand_replicas": "0-1", "refresh_interval": "2s" and the zstd codec.

Rollover is at 20 GB primary shard size or 200 million documents, whichever comes first. Reading the Index State Management policies off a 5.0.0-beta5 install confirms the document figure and the retention ages:

curl -sk -u <INDEXER_USERNAME>:<INDEXER_PASSWORD> "https://127.0.0.1:9200/_plugins/_ism/policies"

On the beta5 lab stack this returns, with everything but the policy names and their ages trimmed away:

        "policy_id" : "ai-assistant-sessions-policy",
                  "min_doc_count" : 200000000,
                  "min_index_age" : "1d",
                  "min_index_age" : "7d"
        "policy_id" : "stream-active-responses-policy",
                  "min_doc_count" : 200000000,
                  "min_index_age" : "3d"
        "policy_id" : "stream-events-policy",
                  "min_doc_count" : 200000000,
                  "min_index_age" : "1h"
        "policy_id" : "stream-findings-policy",
                  "min_doc_count" : 200000000,
                  "min_index_age" : "90d"
        "policy_id" : "stream-metrics-policy",
                  "min_doc_count" : 200000000,
                  "min_index_age" : "30d"
        "policy_id" : "stream-raw-events-policy",
                  "min_doc_count" : 200000000,
                  "min_index_age" : "10m"

Retention defaults, verified on the 5.0.0-beta5 lab:

Data stream	Retention
`wazuh-events-raw-v5`	10 minutes
`wazuh-events-v5-*`	1 hour
`wazuh-findings-v5-*`	90 days
`wazuh-active-responses`	3 days
`wazuh-metrics-*`	30 days
`wazuh-ai-assistant-sessions`	1 day, then deleted at 7

Read the events row again if it surprises you. Normalised events are kept for an hour by default; findings are what you keep for 90 days. If your team expects to hunt over last month’s raw telemetry, that is an explicit retention change and a matching storage budget, not something the defaults give you.

On a multi-node cluster, set replicas through cluster.default_number_of_replicas so the Setup module applies it as a persistent cluster setting, rather than editing the templates by hand.

Check what you actually have:

curl -sk -u <INDEXER_USERNAME>:<INDEXER_PASSWORD> "https://127.0.0.1:9200/_cat/indices/wazuh-*?v&s=store.size:desc"

curl -sk -u <INDEXER_USERNAME>:<INDEXER_PASSWORD> "https://127.0.0.1:9200/_index_template/wazuh-*"

Measure your own events per second

Every estimate above is a placeholder for a measurement. Count documents in the event data streams over a known window and divide:

GET /wazuh-events-v5-*/_search
{
  "size": 0,
  "query": {
    "range": { "@timestamp": { "gte": "now-1h", "lt": "now" } }
  },
  "aggs": {
    "per_minute": {
      "date_histogram": { "field": "@timestamp", "fixed_interval": "1m" }
    },
    "by_agent": {
      "terms": { "field": "wazuh.agent.name", "size": 20 }
    }
  }
}

Run it from Indexer management > Dev Tools on the dashboard, or as a curl against :9200. The bucket counts divided by 60 give EPS per minute; the agent terms aggregation gives you the handful of endpoints producing most of it. Measure events, not findings: a finding only exists when a rule matched, so counting findings measures detection output rather than ingest rate.

Three data streams exist specifically for this kind of question. On beta5 they are wazuh-metrics-agents for operational metrics from monitored endpoints, wazuh-metrics-comms-v4 for communication and performance metrics on the manager side, and wazuh-metrics-normalization for the engine itself. Note the -v4 suffix on the communications stream: it is the one metrics stream that carries a version number, and a dashboard or script written against a bare wazuh-metrics-comms name matches nothing.

Detector scheduling and finding latency

Detectors evaluate normalised events against rules at scheduled intervals and produce findings. You set the interval when you create a detector, so latency between an event and its finding is at minimum that interval plus indexing and refresh time. This is the part of 5.0 that most surprises teams coming from 4.x, where rule evaluation happened inline on the manager.

The shipped content sets the baseline. A stock beta5 install runs 22 alerting monitors before you create anything of your own:

curl -sk -u <INDEXER_USERNAME>:<INDEXER_PASSWORD> -X GET \
  "https://127.0.0.1:9200/_plugins/_alerting/monitors/_search" \
  -H 'Content-Type: application/json' -d '{"query":{"match_all":{}},"size":50}'

Reduced to names and schedules, the beta5 lab stack returns:

monitors: 22
 - apache-http | None | {"period": {"interval": 1, "unit": "MINUTES"}}
 - wazuh-rootcheck | None | {"period": {"interval": 1, "unit": "MINUTES"}}
 - cloud-detection | None | {"period": {"interval": 2, "unit": "MINUTES"}}
 - wazuh-sca | None | {"period": {"interval": 2, "unit": "MINUTES"}}
...
 - wazuh-it-hygiene | None | {"period": {"interval": 5, "unit": "MINUTES"}}
 - wazuh-fim | None | {"period": {"interval": 1, "unit": "MINUTES"}}
 - wazuh-ioc | None | {"period": {"interval": 1, "unit": "MINUTES"}}
 - wazuh-generic-2 | None | {"period": {"interval": 2, "unit": "MINUTES"}}

Three period lengths, and nothing faster than a minute. The time-sensitive content runs at one minute (file integrity, indicator matching, rootcheck, manager events), most detection content at two, and IT hygiene at five. That is the floor on finding latency for shipped content, before your own detectors and before indexing and refresh. Plan response times against those numbers rather than against 4.x behaviour.

Practical consequences:

Do not promise sub-minute alerting on a detector you scheduled at a longer interval. Set the interval to the response time you actually promise.
Active Response monitors have a documented maximum execution interval of one minute, must target the wazuh-findings-v5-* pattern, and support only DocumentLevelTrigger.
Reducing the rule set a detector evaluates is a real performance lever, because you choose which rules are active per detector when you create it.

The document-level monitor settings that govern this work are dynamic cluster settings under plugins.alerting.*: doc_level_monitor_shard_fetch_size defaults to 10000 documents per shard, percolate_query_max_num_docs_in_memory to 50000, percolate_query_docs_size_memory_percentage_limit to 10 percent of JVM heap, and both doc_level_monitor_fanout_max_duration and doc_level_monitor_execution_max_duration to 5 minutes. Raise them only with a symptom that points at one of them.

curl -sk -u <USERNAME>:<PASSWORD> -X PUT \
  "https://localhost:9200/_cluster/settings" \
  -H "Content-Type: application/json" \
  -d '{"persistent": {"plugins.alerting.monitor.max_monitors": 20}}'

Symptoms and where to look

Symptom	Most likely cause	First check
Agents log `503` from the manager	Intake capacity, not a queue	Manager CPU, then agent count per worker
Agents log `401` intermittently	Clock skew, not load. Tokens live 60 seconds	NTP on managers and agents
Events present, findings late	Detector schedule	The detector’s configured interval
Dashboard slow on wide time ranges	Indexer heap or swapping	`mlockall`, then heap size
Disk filling faster than planned	Raw event indexing on, or retention changed	`engine.index_raw_events`, then `_cat/indices`
Documents missing after a manager restart	In-memory connector buffer discarded	Nothing to recover; change the restart procedure
Indexer red or yellow after a node loss	Replicas at the template default of zero	`cluster.default_number_of_replicas`

Common errors

`Unable to lock JVM Memory`

In /var/log/wazuh-indexer/wazuh-indexer.log. Cause: bootstrap.memory_lock: true is set but the system limit was not raised, so the request to lock failed. Fix: add LimitMEMLOCK=infinity through a systemd drop-in, reload the daemon and restart the indexer, then re-check mlockall.

HTTP `503` from the manager to agents

Cause: the manager is shedding capacity. The HTTPS channel does this deliberately rather than buffering into a fixed queue, which is why <queue_size> does not apply to it. Fix: add manager workers or reduce collection. A 429 seen by an agent came from something between the agent and the manager, not from the manager.

Startup fails after editing `wazuh-manager.conf`

Cause: an element the 5.0 manager does not know. Unknown elements are a startup error, and several 4.x tuning options moved rather than disappeared, including <queue_size>, <protocol> and <local_ip>, all of which now live under <remote><legacy>. Fix: validate before restarting with wazuh-manager-analysisd -t, or check the configuration through GET /cluster/configuration/validation on the manager API.

Index templates revert after a restart

Cause: hand-edited Wazuh-managed templates. The Setup module updates them to match the installed release on startup. Fix: use the settings the Setup module exposes, such as cluster.default_number_of_replicas, or create your own template at a higher priority for your own indices rather than editing Wazuh’s.

Where to go next

The tuning settings the indexer exposes are in the official Wazuh indexer tuning guide, and the rollover and retention behaviour is documented on the Wazuh indices page.

If the measurement above shows a small number of rules generating most of your findings, the next lesson to read is cutting false positives, because tuning content is usually cheaper than adding nodes. Retention and snapshot mechanics are in upgrade, backup and index retention, and the node layout that these settings apply to is in Wazuh cluster and distributed architecture.

Next: Multi-Tenant Wazuh for MSSPs.

Files for this lesson

Every file this lesson refers to, exactly as used. Open a file to read it in the browser, or download the set as a zip. Download wazuh-performance-tuning.zip. The full set for all lessons is at wazuh-tutorial-files.zip.

Frequently asked questions

How much heap should the Wazuh indexer have?

Half of the system RAM, with the minimum and maximum set to the same value in /etc/wazuh-indexer/jvm.options. On an 8 GB host that is -Xms4g and -Xmx4g. Matching the two prevents heap resizing at runtime, and memory locking has to be enabled alongside it so the JVM is never swapped out.

How do I measure events per second in Wazuh 5.0?

Count documents in the event data streams over a known window with a date histogram on the indexer, then divide by the window in seconds. Do not count findings: a finding is only produced when a rule matches, so it measures detection output rather than ingest rate.

Why is Wazuh 5.0 storing more data than 4.x?

Wazuh 4.x indexed alerts by default and archives only if you enabled them. Wazuh 5.0 indexes normalised events into wazuh-events-v5 data streams as a matter of course, and can additionally index every raw event into wazuh-events-raw-v5. The raw stream is controlled by the engine.index_raw_events setting on the Setup module.

How long does it take for a finding to appear after the event?

It is the detector's schedule plus indexing and refresh time, not the sub-second path people expect from 4.x. Detectors are scheduled jobs on the indexer and you choose the interval when you create one. The 22 monitors shipped with 5.0.0-beta5 run at one, two or five minutes, so that is the floor for shipped content.

Where do Wazuh data streams roll over?

At 20 GB primary shard size or 200 million documents, whichever comes first, managed by Index State Management policies created by the Setup module. Default retention is 10 minutes for raw events, 1 hour for normalised events, 90 days for findings, 3 days for active responses and 30 days for metrics.