OPNsense HA with CARP: 3 Real-World Pitfalls

OPNsense HA with CARP is a five-minute setup on paper: two identical appliances, a WAN VIP, a LAN VIP, a dedicated pfSync interface, XMLRPC sync enabled — and the cluster is live. In practice, when we take over existing installations, we see the same three problem areas over and over. They either silently prevent failover or cause partial outages when the real thing happens. This post covers the pitfalls we most often clean up in customer projects — and how to eliminate them permanently with reasonable effort.

We assume a classic active/passive setup with OPNsense 25.7 or 26.1 on two hardware appliances (typically a DEC750, DEC2685 or generic x86 boxes with i226 NICs). The sync link is a dedicated 1G or 10G interface, VIPs are configured per subnet, and High Availability Sync is active under System > High Availability > Settings.

Pitfall 1: Silent config drift between master and backup

OPNsense XMLRPC sync replicates a selected list of areas — firewall rules, aliases, NAT, DHCP reservations, IPsec, OpenVPN, users, certificates. What it does not sync is equally important: interface assignments, VLAN definitions, physical addresses, gateway definitions, Suricata rulesets (only partially), NetFlow, system tunables, package installations and in many cases the ordering of firewall categories.

In practice this means: as soon as an administrator “just quickly” adds a VLAN on the master, relabels an interface or installs a plugin without repeating the same steps on the backup, drift builds up that only becomes visible during failover — typically ending up in the ticket system as “why does VLAN 47 not work any more?”.

Our approach is two-pronged. First, a monthly automated config diff: /conf/config.xml is pulled from both nodes via SSH, sanitised at defined spots (timestamps, RRD data, CARP passwords) and compared with diff. Any line that is not a pure HA delta is a finding.

#!/bin/sh
# carp-drift-check.sh -- compare master vs backup
MASTER=fw01.intern.example.com
BACKUP=fw02.intern.example.com
TMPDIR=$(mktemp -d)

for HOST in $MASTER $BACKUP; do
  ssh root@$HOST 'cat /conf/config.xml' \
    | sed -E 's/<time>[0-9]+<\/time>//g' \
    | sed -E 's/<lastchange>[0-9]+<\/lastchange>//g' \
    > "$TMPDIR/$(basename $HOST).xml"
done

diff -u "$TMPDIR/$MASTER.xml" "$TMPDIR/$BACKUP.xml" \
  | grep -vE '^(---|\+\+\+|@@|.*carp_.*password)' \
  > /var/log/carp-drift.log

test -s /var/log/carp-drift.log && \
  mail -s "CARP drift on $MASTER/$BACKUP" ops@example.com < /var/log/carp-drift.log

Second: interface and VLAN changes always run manually on both nodes, in the same order, with the backup first. That way the master is guaranteed to keep running if something goes wrong on the backup during a change.

Pitfall 2: Sync interface under load — when pfSync sabotages its own HA

pfSync replicates the state table between nodes in real time. On an SMB cluster with 20,000 active states this is completely invisible. On a cluster holding 500,000 states — typical for terminal-server farms, VoIP trunks with many parallel RTP streams or backup windows with many small SMB sessions — the sync connection can become the bottleneck.

We see two symptoms in practice: first sporadic “pfsync: requesting bulk update” messages in the log because the backup node has fallen behind; second, short reset waves during failover because the state table on the taking-over node was incomplete.

The causes are almost always one of these:

Cause	Symptom	Fix
Sync link over shared switch with VLAN	Latency spikes, bulk updates	Direct cable node-to-node, no intermediate switches
1G sync interface, >250k states	pfsync queue fills up	Upgrade to 10G or LACP two 1G links
MTU mismatch (1500 vs 9000)	Bulk update breaks	Both sync interfaces on 1500 or both on 9000
Firewall rule blocks pfsync	Backup shows “SYNC” instead of “MASTER” after failover	Sync interface on “any/any allow” or explicit rule for protocol 240

Our rule of thumb from customer projects: for anything above 100,000 states we recommend a dedicated 10G sync link, directly cabled, with MTU 9000 on both sides. Configuration is under Interfaces > [SYNC] > MTU — and, importantly, also on the physical NIC driver via System > Settings > Tunables if the driver does not adopt the MTU dynamically.

For monitoring, a simple Zabbix or Checkmk check has proven itself: query pfctl -s states | wc -l and the pfsync queue depth on both nodes. If the queue exceeds 500 entries for more than 30 seconds, sync capacity is exhausted.

Pitfall 3: A DHCP failover that is not really a failover

The most frequently overlooked point: many existing clusters run isc-dhcp (or by now Dnsmasq or Kea) in “sync” mode but without a real DHCP failover peer. That means: leases are pushed from the master to the backup via XMLRPC, but the backup does not hand out new leases itself. When the master fails, newly connected devices do not receive an IP address — existing sessions keep working, but everything new stalls.

From OPNsense 25.7 onwards, Kea DHCP is the recommended server, and Kea supports real HA mode (hot-standby or load-balancing). Our standard setup for new clusters:

# Kea HA configuration, conceptual
ha:
  mode: hot-standby
  peers:
    - name: fw01
      url: https://10.99.0.1:8000/
      role: primary
    - name: fw02
      url: https://10.99.0.2:8000/
      role: standby
  heartbeat-delay: 10000
  max-response-delay: 60000

In the OPNsense GUI you configure this under Services > Kea DHCP > [Subnet] > High Availability. Important: the Kea control interface must be reachable on the sync link, not via the VIP. Otherwise HA communication itself runs over the network that is being switched during failover — a classic split-brain trigger.

For clusters still on ISC DHCP we recommend migrating to Kea in the next maintenance window. ISC DHCP is upstream EOL and OPNsense will remove support in the medium term.

Health checks that actually mean something

A CARP status of “MASTER” on a node says nothing about whether the cluster will work when it counts. Our minimum checklist for the monthly HA test:

1. Simulated failover via CLI: pfctl -k on the master, plus configctl carp maintenance on. The backup takes over all VIPs within 3 seconds. New TCP connections go through, DHCP hands out new leases, IPsec tunnels rebuild.
2. State-sync verification: before failover, compare the state count on both nodes — the backup should be within 5% of the master.
3. Config drift report: the script from pitfall 1 runs and reports zero diffs (except HA-specific fields).
4. DHCP failover test: connect a new client while the master is in maintenance; the client receives a lease.
5. DNS resolver test: Unbound on the backup answers with the same overrides as the master.

Anyone running these five points quarterly finds 90% of all HA errors before they become production-relevant.

Conclusion

OPNsense HA with CARP is stable when three fundamentals are in place: identical configuration on both nodes, a properly dimensioned sync link and a DHCP server in true HA mode. Most outages we take over are not due to bugs in OPNsense but to setups that were installed once and never touched again. A monthly config diff and a quarterly failover test are the difference between a cluster that saves you downtime and one that doubles it when it matters.

DATAZONE supports you in the design, migration and operation of OPNsense HA clusters — from hardware selection through OPNsense consulting to long-term monitoring. If you are taking over an existing cluster or planning a new one, get in touch: contact us.