Repository navigation
Troubleshooting Guide
This guide is organized from the most common bring-up problems to more specific and obscure ones. Each section can be read on its own, but the diagnostic tools introduced at the top are used throughout.
Important
This guide assumes working knowledge of Linux networking (addresses, routes, VLANs, VRFs, ARP/NDP) and of DPDK concepts (hugepages, device binding, poll-mode drivers). It focuses on grout-specific behaviour; generic networking or DPDK setup problems are only covered where grout does something unusual.
Before diving into specific symptoms, these are the commands you will use to
observe what grout is doing. All of them can be typed at the interactive
grcli prompt or run one-shot as grcli <command...>.
| What you want to see | Command |
|---|---|
| Interfaces, flags, MTU, MAC |
grcli interface show / grcli interface show name NAME
|
Per-interface counters (incl. RX_DROPS/TX_ERRORS) |
grcli interface stats |
| Same, sampled per second | grcli interface rates |
| Per-node software stats (drop counters live here) | grcli stats show |
| NIC hardware xstats | grcli stats show hardware |
| Trace a packet's exact path through the graph |
grcli trace enable all then grcli trace show
|
| Routing table / which route a dest uses |
grcli route show / grcli route get DEST
|
| FIB usage (for "FIB full") | grcli route config show |
ARP/NDP neighbors (nexthops with origin learn) |
grcli nexthop show |
| L2 forwarding database | grcli fdb show |
| Worker/queue to CPU mapping | grcli affinity qmap show |
| Control/datapath CPU split | grcli affinity cpus show |
| Live event stream | grcli events show |
Note
grcli commands follow a <noun> <verb> order, e.g. interface add port ..., address add ..., stats show.
The single most useful debugging sequence when traffic misbehaves is:
grcli trace enable all
# ... reproduce the problem, send a few packets ...
grcli trace show
grcli trace disable allgrcli trace show prints, for each packet, every graph node it traversed and where
it ended, including the terminating drop node. That tells you why a packet
was dropped, which the Packet drop reasons section
explains.
Instead of grcli trace enable all, you can scope tracing to a single
interface with grcli trace enable iface NAME - useful on a busy box so you
only capture the path you care about.
Caution
Never leave log packets enable or trace enable ... enabled in
production. Both inspect every packet on the datapath and severely degrade
forwarding performance. They are last-resort, interactive tools: turn them
on only while you actively reproduce a problem, and turn them back off
immediately afterwards. For ongoing or non-invasive debugging, use the
per-node drop counters (grcli stats show, see Packet drop
reasons) - they are always on and have no performance
cost.
When the counters tell you drops are happening but not why a specific packet
is affected, turn on full packet logging and raise the log verbosity of the
relevant subsystem. This makes grout log every ingress/egress packet (including
the [drop <node>] <n> packets lines) to its log stream, which is invaluable
for correlating drops with control-plane events:
grcli log packets enable
grcli trace enable all
# ... reproduce the problem ...
grcli trace show
# when done, turn it back off (it is verbose):
grcli log packets disable
grcli trace disable allRaise the log level of a specific subsystem to see more detail (pattern is
<type>:<level>):
grcli log level set grout.infra.port:debug
grcli log level show # list log types and their current levelsgrcli log packets enable is the runtime equivalent of starting the daemon
with -x/--trace-packets/GROUT_TRACE_PACKETS; use the runtime command so
you can switch it on only while reproducing, since it is very chatty under
load. Do not ship a production configuration with GROUT_TRACE_PACKETS set.
grout is a DPDK application. Most first-time failures are host setup problems,
not grout bugs. grout does not bind kernel drivers itself and does not
probe PCI devices at startup; devices are hot-plugged later with grcli interface add port. The prerequisites (IOMMU, hugepages, vfio-pci binding,
CPU isolation) must be in place on the host first.
Reference host setup:
dnf install driverctl tuned-profiles-cpu-partitioning
# Enable IOMMU on boot.
grubby --update-kernel ALL --args "iommu=pt intel_iommu=on"
# Reserve hugepages.
grubby --update-kernel ALL --args "default_hugepagesz=1GB hugepagesz=1G hugepages=16"
# Isolate CPUs for grout datapath (see the CPU isolation section).
echo "isolated_cores=2-19,22-39" > /etc/tuned/cpu-partitioning-powersave-variables.conf
echo "max_power_state=C6|170" >> /etc/tuned/cpu-partitioning-powersave-variables.conf
tuned-adm profile cpu-partitioning-powersave
grubby --update-kernel ALL --args "isolcpus=2-19,22-39"
systemctl reboot
# Bind the devices you intend on using with grout to vfio-pci.
# NB: NVIDIA/Mellanox NICs should remain bound to their default driver.
driverctl set-override 0000:8a:00.0 vfio-pci
driverctl set-override 0000:8a:00.1 vfio-pcigrout composes the EAL arguments itself and logs them at startup:
MAIN: dpdk_init: DPDK 25.11.3
MAIN: dpdk_init: EAL arguments: -l 0 --no-telemetry -a pci:0000:00:00.0 --in-memory
Any EAL failure (hugepages, VFIO/IOMMU, permissions) is reported by the DPDK
EAL and printed through grout's logging (grout uses the DPDK logging backend)
with a <LEVEL>: <TYPE>: ... prefix, or via syslog with --syslog. When grout
fails to start, systemctl status grout.service and the journal are the first
place to look.
Symptom: grcli interface add port ... fails with an error such as No such device (ENODEV), or grout starts but the port cannot be created.
Cause: the PCI device is still bound to its kernel driver, or was never bound
to vfio-pci.
Fix:
driverctl set-override 0000:8a:00.0 vfio-pci
# verify:
lspci -ks 0000:8a:00.0 # or driverctl list-overridesThen create the port:
grcli interface add port p0 devargs 0000:8a:00.0 rxqs 1 qsize 2048Exceptions:
-
NVIDIA/Mellanox (mlx5) NICs use a bifurcated driver and must stay bound to their kernel driver - do not rebind them to vfio-pci. Instead, the kernel netdev must live in the same network namespace as grout, because the mlx5 DPDK driver talks to it over netlink. If the netdev is in a different netns, traffic will silently fail (packets received on the far side, no replies). Move the netdev into grout's netns/container.
-
SR-IOV VFs (Intel iavf) may log benign errors on bring-up that do not prevent the port from working, e.g.
rte_eth_dev_set_link_up: Operation not supported (95)oriface_loopback_init: open No such file or directory. If the interface reachesup runningand passes traffic, these can be ignored.
Symptom: VFIO binding fails, or the port cannot be probed, sometimes with VFIO "IOMMU group" errors from DPDK.
Check and fix:
grep -o 'intel_iommu=[a-z]* iommu=[a-z]*' /proc/cmdline # expect: intel_iommu=on iommu=pt
# if missing:
grubby --update-kernel ALL --args "iommu=pt intel_iommu=on"
systemctl rebootIf your platform genuinely has no IOMMU and you use VFIO no-IOMMU mode, grout
detects it and forces --iova-mode=pa automatically.
Two distinct hugepage problems show up in grout's log at startup.
The following message means pages are reserved but the hugetlbfs is not mounted
(there is nothing at /dev/hugepages for that page size):
EAL: 8000 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size
A mempool allocation failure when adding a port or starting a worker
(propagated as ENOMEM, with DPDK's "Cannot allocate mempool" in the log)
means there are not enough usable hugepages.
Reserve hugepages (and make sure the hugetlbfs is mounted):
grep Huge /proc/meminfo
# reserve, e.g. 1G pages (recommended):
grubby --update-kernel ALL --args "default_hugepagesz=1GB hugepagesz=1G hugepages=16"
systemctl rebootIf you use 1G pages (recommended), the warning EAL: No free 2048 kB hugepages reported on node 0 is harmless and can be ignored: it only means no 2M pages
are available, which grout does not need.
Warning
Test mode is not suitable for production. Hugepages are only optional in
test mode (-t / GROUT_TEST_MODE, which adds --no-huge). Test mode uses
a fixed anonymous heap meant for CI and local development only. Never run
a production datapath in test mode.
grout needs write access to /dev/hugepages, /dev/vfio, and to
/proc/sys/net. A read-only /proc/sys/net causes address and VRF setup to
fail with confusing downstream symptoms, e.g.:
CTLPLANE: sysctl_write: fopen(/proc/sys/net/ipv6/conf/vniXXX/keep_addr_on_down): Read-only file system
ADDRESS: ip6_iface_llocal_init: iface_addr_add: Address already in use (98)
Run the container privileged (so /proc/sys/net is writable) and give it the
hugepage and vfio devices:
podman run --privileged \
-v /dev/hugepages:/dev/hugepages \
-v /dev/vfio:/dev/vfio \
-v /sys/bus/pci:/sys/bus/pci \
--device /dev/net/tun \
--device /dev/infiniband \
... groutSymptom: pings to the VLAN gateway time out even though the port is up.
Cause: the IP address was configured on the parent port instead of the VLAN sub-interface, so routes egress on the untagged port and nothing uses the VLAN interface.
Fix: put the address on the VLAN interface.
grcli interface add vlan vlan404 parent p0 vlan_id 404
grcli address add 192.168.15.60/26 iface vlan404 # NOT iface p0Symptom: setting an interface MTU fails with Numerical result out of range
(ERANGE).
Cause: the requested MTU is below 1280 or above the daemon-wide maximum
(-u/--max-mtu/GROUT_MAX_MTU, default 1800). The per-interface MTU cannot
exceed the daemon max.
Fix: raise the daemon max MTU and restart, then set the interface MTU:
# systemd EnvironmentFile (/etc/grout.default) or command line
GROUT_MAX_MTU=9200grcli interface set port p0 mtu 9000Note
Raising --max-mtu has a memory cost: it sets the size of every mbuf, so
all packets consume a full-sized buffer regardless of their actual length
(e.g. with --max-mtu 9200 even a 64-byte packet uses a ~9K buffer). Set it
only as high as you actually need.
grout does not support multi-segment mbufs: a packet must fit in a single
mbuf. If your MTU is larger than the default mbuf data size, the only solution
is to size the mbufs at startup with --max-mtu / GROUT_MAX_MTU set at or
above your MTU.
Symptom: route insertion fails with Not enough memory (ENOMEM) well before
you expect, or a route add fails.
Cause: the FIB has a configured maximum. Check current usage:
grcli route config showFix: raise the limit (no rebuild needed on current grout) via env var at startup or at runtime:
GROUT_MAX_ROUTES=1048576 # startup default$ grcli interface set vrf main rib4-routes 1048576
$ grcli route config show vrf main
VRF FAMILY ROUTES TBL8
main ipv4 4/1048576 (0.0%) 0/2112 (0.0%)
main ipv6 2/65536 (0.0%) 10/262144 (0.0%)grcli route config show reports two IPv4 limits, and either can fill up:
-
routes - the total number of routes (
rib4-routes/GROUT_MAX_ROUTES). -
tbl8 - the IPv4 FIB is a DIR-24-8 table; only prefixes longer than /24 consume a
tbl8group. A network with many long (> /24) prefixes can exhausttbl8before hitting the route limit.
The number of tbl8 groups is derived automatically from rib4-routes, so
raising the route limit raises tbl8 too. Adding more hugepages does not
raise either limit - the knob does.
grout runs each datapath worker as a pthread pinned to a single CPU core, busy
running rte_graph_walk() in a tight loop. For deterministic forwarding
performance, the kernel must not preempt those cores for anything else. The
goal of this section is: zero involuntary context switches on datapath worker
CPUs.
At startup, grout inherits the CPU affinity mask it was launched with, uses the first available CPU as the main lcore (control plane), and gives all remaining CPUs to datapath workers. It logs the split:
MAIN: dpdk_init: running control plane on CPU 0
MAIN: dpdk_init: datapath workers allowed on CPUs 1-9,20-29
So the CPUs grout may use are set by the launch affinity (systemd
CPUAffinity=, taskset, or cgroup), not by an EAL -l flag. You can also
set the split explicitly at runtime:
$ grcli affinity cpus set control 0,20,1,21 datapath 10,30,12,32,14,34,16,36
$ grcli affinity qmap show # which rxq runs on which CPU
CPU_ID IFACE RXQ_ID ENABLED
10 p0 0 true
12 p0 1 true
14 p0 2 true
16 p0 3 true
30 p1 0 true
32 p1 1 true
34 p1 2 true
36 p1 3 trueTo pin a specific receive queue to a specific worker CPU:
grcli affinity qmap set p0 rxq 0 cpu 16Note
The target CPU must be in the datapath set - otherwise you get EBUSY if it
is a control CPU, or ERANGE if it is outside the datapath mask.
Use isolcpus plus the tuned cpu-partitioning-powersave profile so the
scheduler, RCU callbacks, and the timer tick stay off the worker cores (the
profile pulls in nohz_full and rcu_nocbs for the isolated set):
dnf install tuned-profiles-cpu-partitioning
echo "isolated_cores=10-19,30-39" > /etc/tuned/cpu-partitioning-powersave-variables.conf
tuned-adm profile cpu-partitioning-powersave
grubby --update-kernel ALL --args "isolcpus=10-19,30-39"
systemctl rebootMake sure the datapath CPUs you isolate match the CPUs you give grout (the
datapath set above).
Important
Always assign all hyperthreads of the same physical core to grout. Never
give one sibling thread to grout and the other to a non-DPDK application:
sibling threads share the core's execution units and caches, so the other
workload will steal resources from the busy-polling worker and hurt its
performance unpredictably. Check the sibling pairs with lscpu -e or cat /sys/devices/system/cpu/cpuN/topology/thread_siblings_list, and keep both
threads of each datapath core in the grout datapath set (and isolated).
Even with isolcpus, device interrupts can still land on the isolated cores
and preempt a worker. Keep irqbalance from touching them:
# /etc/sysconfig/irqbalance
IRQBALANCE_BANNED_CPULIST=2-19,22-39
systemctl restart irqbalanceThen verify no device IRQ is still pinned to a worker core and move any that are:
# find IRQs whose smp_affinity_list still includes an isolated CPU
for i in /proc/irq/[0-9]*; do
printf '%s %s\n' "${i##*/}" "$(cat "$i/smp_affinity_list")"
done
# move a stray IRQ off the datapath cores, e.g. onto CPU 0:
echo 0 > /proc/irq/<N>/smp_affinity_listA correctly isolated worker thread should show essentially zero non-voluntary context switches over time. Any that accumulate mean the kernel is preempting a busy-polling worker - a latency/throughput problem.
First identify the worker thread IDs. grout logs one per worker at startup:
GRAPH: gr_datapath_loop: [CPU 10] starting tid=634073
GRAPH: gr_datapath_loop: [CPU 12] starting tid=634074
GRAPH: gr_datapath_loop: [CPU 14] starting tid=634075
GRAPH: gr_datapath_loop: [CPU 16] starting tid=634076
GRAPH: gr_datapath_loop: [CPU 30] starting tid=634077
GRAPH: gr_datapath_loop: [CPU 32] starting tid=634078
GRAPH: gr_datapath_loop: [CPU 34] starting tid=634079
GRAPH: gr_datapath_loop: [CPU 36] starting tid=634080
Then watch the non-voluntary switch rate per thread with pidstat (-t per
thread, -u CPU, -w context switches, -h one line per sample):
$ pidstat -wuth -G grout 1 1
Time UID TGID TID %usr %system %guest %wait %CPU CPU cswch/s nvcswch/s Command
08:47:26 PM 0 634048 - 795.15 0.00 0.00 0.00 795.15 20 1.94 0.00 grout:main
08:47:26 PM 0 - 634048 0.00 0.00 0.00 0.00 0.00 20 1.94 0.00 |__grout:main
08:47:26 PM 0 - 634073 100.00 0.00 0.00 0.00 100.00 10 0.00 0.00 |__grout:w10
08:47:26 PM 0 - 634074 100.00 0.00 0.00 0.00 100.00 12 0.00 0.00 |__grout:w12
08:47:26 PM 0 - 634075 100.00 0.00 0.00 0.00 100.00 14 0.00 0.00 |__grout:w14
08:47:26 PM 0 - 634076 100.00 0.00 0.00 0.00 100.00 16 0.00 0.00 |__grout:w16
08:47:26 PM 0 - 634077 99.03 0.00 0.00 0.00 99.03 30 0.00 0.00 |__grout:w30
08:47:26 PM 0 - 634078 100.00 0.00 0.00 0.00 100.00 32 0.00 0.00 |__grout:w32
08:47:26 PM 0 - 634079 99.03 0.00 0.00 0.00 99.03 34 0.00 0.00 |__grout:w34
08:47:26 PM 0 - 634080 100.00 0.00 0.00 0.00 100.00 36 0.00 0.00 |__grout:w36Watch the nvcswch/s column for the worker threads (grout:w<num>) - it
should stay at ~0. Non-zero means something is preempting the worker.
Without pidstat, read the counter straight from /proc and diff it over time:
for pid in $(pidof grout); do
for t in /proc/$pid/task/*; do
while read -r field value; do
case "$field" in
Name:) name=$value ;;
Pid:) pid=$value ;;
Cpus_allowed_list:) cpus=$value ;;
nonvoluntary_ctxt_switches:) ctx=$value ;;
esac
done < $t/status
echo $name $pid $cpus $ctx
done
done | column -t -N THREAD,PID,CPU_AFFINITY,NONVOL_CTX_SWITCHES
# sample again a few seconds later; any increase of NONVOL_CTX_SWITCHES on a grout:w* thread is badIf you see increases, the usual culprits are: a CPU that is not actually in
isolcpus, a device IRQ still pinned to that core (see irqbalance above), or
another process/cgroup allowed to run there.
By default an idle grout worker does not sit at a hard 100%: when there is no traffic it ramps an adaptive micro-sleep (with tightened timer slack) and resets on any packet activity. A busy worker still uses a full core. There are three modes, trading power against latency:
-
Default (adaptive micro-sleep) - good balance. Idle cores sleep in small increments; latency is slightly higher than pure polling but should be safely lower than the time it takes for RX queues to fill before wakeup.
-
-p/--poll-mode/GROUT_POLL_MODE- disables the micro-sleep. Workers busy-poll at 100% CPU always. Lowest latency, highest power. Use only if you have packet drops due to wakeup latency. -
-a/--adaptive-irq/GROUT_ADAPTIVE_IRQ- when idle, workers arm RX interrupts and block on epoll instead of polling, so the core can actually go idle and enter deep C-states. This gives the lowest power at idle. It requires the PMD to support RX interrupts; if unsupported it logs a NOTICE and falls back to the micro-sleep ramp (and it is ignored if--poll-modeis also set). To avoid the core being down-clocked while blocked, grout pinsuclamp_minto max viasched_setattr; if the kernel does not support it you will see[CPU N] uclamp_min unavailable: ...(harmless, just no frequency floor).
To minimize power draw, combine --adaptive-irq with a tuned profile that
allows deep C-states on the isolated cores. The cpu-partitioning-powersave
profile plus max_power_state controls how deep the isolated cores may sleep:
echo "isolated_cores=10-19,30-39" > /etc/tuned/cpu-partitioning-powersave-variables.conf
echo "max_power_state=C6|170" >> /etc/tuned/cpu-partitioning-powersave-variables.conf
tuned-adm profile cpu-partitioning-powersaveTrade-off summary: --poll-mode (max performance, max power) → default
(balanced) → --adaptive-irq + powersave tuned profile (min power, slightly
higher wake-up latency).
grout counts drops per reason. Each drop reason is a named node in the packet processing graph, and its counter is visible in the stats. To see drops:
grcli stats show pattern '*drop*' zero # all drop counters, including zero
grcli stats show pattern 'arp_input_request_drop' # one specific reason
grcli interface stats # RX_DROPS / TX_ERRORS at the NIC boundaryinterface stats shows RX_DROPS/TX_ERRORS at the hardware/interface
boundary (NIC ran out of descriptors, link errors). stats show shows the
software drop nodes - the reason grout itself discarded a packet.
These counters are always on and cost nothing, so they are the preferred way to diagnose drops in production. Reach for the per-node counters first; only escalate to packet logging/tracing (below) when you must understand an individual packet, and never leave those enabled (see Deep-dive debugging).
To find which reason a specific packet hits, trace it (interactively, then turn it back off):
grcli trace enable all
# ... send the packet ...
grcli trace show
grcli trace disable allThe last node in a traced packet's path is where it died. If packet logging is
on (grcli log packets enable, or the daemon -x/--trace-packets flag), the
daemon also logs TRACE: [drop <node>] <n> packets.
| Drop node | Meaning / what to check |
|---|---|
error_no_headroom |
Not enough mbuf headroom to prepend a header (encap/ICMP build). Usually a sizing/config issue. |
eth_input_unknown_type |
Unhandled ethertype on ingress (usually harmless). |
eth_output_no_mac |
Destination MAC not resolved - the neighbor is unresolved. Check nexthop show state (pending/failed) and ARP/ND reachability. |
iface_input_admin_down |
Packet received on an admin-down interface. Bring it up. |
iface_input_unknown_vlan |
Tagged frame with no matching VLAN sub-interface. Create the VLAN iface. |
ip_input_other_host / ip6_input_other_host
|
Strong-host model: destination is not a local address on the ingress interface. |
ip_input_bad_checksum / *_bad_length / *_bad_version
|
Malformed header. |
ip_error_ttl_exceeded / ip6_error_ttl_exceeded
|
TTL/hop-limit hit 0; grout sends ICMP Time Exceeded. |
ip_error_no_route / ip6_error_no_route
|
No matching route; grout sends ICMP Destination Unreachable. Check route get DEST. |
ip_blackhole / ip6_blackhole
|
Route points at a blackhole nexthop (intentional). |
ip_admin_prohibited / ip6_admin_prohibited
|
Route points at a reject nexthop. |
ip6_output_too_big |
Egress MTU exceeded (would send ICMPv6 Packet Too Big). |
ip_output_drop |
Includes NAT verdict DROP. |
arp_input_request_drop |
ARP request dropped (e.g. not for a local exposed address). Common in VXLAN/EVPN ARP problems. |
bridge_input_hairpin |
Frame would be sent back out its ingress port. |
bridge_input_flood_disabled |
BUM flooding disabled on the bridge. |
bridge_neigh_suppress_drop |
ARP/ND suppression consumed the packet (EVPN neigh-suppress). |
vxlan_input_no_tunnel |
Received VNI has no matching local VXLAN interface. |
vxlan_output_no_route |
No route to the remote VTEP in the encap VRF. |
sr6_local_* |
SRv6 SID/behavior validation failure on decap. |
sr6_source_no_route |
No route for the SRv6-encapsulated packet. |
Full list of the ~98 drop nodes: grcli graph show full (each drop/error node
appears in the pipeline). grcli stats show pattern '*drop*' shows their live
counts.
If packets are lost but no software drop node is counting them, the loss is
usually at the NIC: the receive rings fill up faster than the workers drain
them, so the hardware drops packets before grout ever sees them. These do not
appear in stats show (software) - they show up as RX_DROPS on the interface
and as *miss* counters in the hardware xstats:
$ grcli interface rates # look at non-zero RX_DROPS/S
INTERFACE RX_PACKETS/S RX_BYTES/S RX_DROPS/S TX_PACKETS/S TX_BYTES/S TX_ERRORS/S
main 0 0 0 0 0 0
p0 31475008 1888500480 5741713 31501312 1890078720 0
p1 31501312 1890078720 5715462 31475008 1888500480 0
$ grcli stats show hardware pattern '*miss*'
p0.rx_missed_errors: 531937234
p1.rx_missed_errors: 529713662A non-zero and increasing rx_miss/imissed means the datapath cannot keep
up: the worker is not draining the RX queue fast enough (CPU saturated,
preempted, or too few queues/workers for the load).
To watch the load live, poll the per-second rates in a loop and watch for
RX_DROPS/S appearing while RX_PACKETS/S is near line rate:
yes interface rates | GROUT_PAGER= grcliWhat to check and do:
-
Confirm the workers are not being preempted - see CPU isolation and the involuntary context-switch check. A worker that is context-switched off its core cannot drain the ring.
-
Give the port more receive queues and spread them across more worker CPUs, so the load is shared:
grcli interface set port p0 rxqs 4 grcli affinity qmap set p0 rxq 0 cpu 2 grcli affinity qmap set p0 rxq 1 cpu 4 # ... grcli affinity qmap show
-
Increase the queue size (
qsize) to absorb short bursts (this smooths micro-bursts but does not fix a sustained overload):grcli interface set port p0 qsize 4096 -
Check per-CPU where the cycles go:
grcli stats show cpu CPUshowsCYCLES/PKTper node on that worker - a node with an unusually high cost points at the expensive part of the pipeline.
Important
If the offered load simply exceeds what the assigned cores can forward, the fix is more cores/queues (scale out) - no amount of tuning makes a single saturated core forward more than it can.
Symptom in the debug log:
NEXTHOP: nh4_resolve_cb: 172.16.0.2 hold queue full
Cause: the nexthop is unresolved (nobody answering ARP/ND), and grout holds
packets awaiting resolution up to a maximum (default 256), then drops. This is
working as designed - it means the next hop is unreachable. Check L2
reachability and nexthop show for that address (pending/failed). The hold
depth is tunable:
grcli nexthop config set held-packets 1024grout integrates with FRR through the dplane_grout zebra plugin, which
replaces the kernel forwarding plane. FRR must run in the same network
namespace as grout, and grout should be started first - the plugin retries
the connection every second and does a full sync before normal operation. See
man 7 grout-frr for the full model.
Enable the plugin in /etc/frr/daemons:
zebra_options="-A 127.0.0.1 -s 90000000 -M dplane_grout"This is the highest-impact FRR gotcha, reported repeatedly.
Symptom: ss -tn shows sessions stuck at SYN-SENT/SYN-RECV toward :179;
grout traces show the SYN but bgpd never completes. Numbered BGP over the plain
kernel interface works, but not over grout-fed netdevs.
Cause: strict reverse-path filtering (rp_filter=1) on the control-plane TAP
interfaces drops the return traffic. Note rp_filter resolves with MAX
semantics - setting net.ipv4.conf.all.rp_filter=0 does not help if the
per-interface value is 1.
Fix (preferred): let grout set loose RP filter (rp_filter=2) on its control
plane TAPs:
GROUT_OVERRIDE_RP_FILTER=onThis is required when net.ipv4.conf.all.rp_filter is 1 (strict) in grout's
netns. As a manual alternative/diagnosis, set it per grout interface:
sysctl -w net.ipv4.conf.p0.rp_filter=0
sysctl -w net.ipv4.conf.p1.rp_filter=0Symptom: grout's default route is ignored because another default route (e.g. from the CNI) has priority.
Fix:
GROUT_OVERRIDE_DEFAULT_ROUTE=ongrout then overrides the existing default route in the main table with metric 0.
Because grout mirrors interface addresses as host routes, BGP neighbors must
point update-source at the grout local IP. This is not needed for OSPF/OSPFv3
(which use multicast/link-local).
Symptom: repeated Unable to find specified Interface for default in the zebra
log when applying addresses from frr.conf at startup.
Cause: zebra tries to configure interfaces and addresses before the grout plugin has finished its initial interface sync. These early-boot messages are expected due to sync timing and resolve on their own once the sync completes.
Create a VXLAN interface and program remote MACs/VTEPs:
grcli interface add vxlan vni777 vni 777 local 100.0.2.2 domain br777 encap_vrf underlay
grcli fdb add MAC iface IFACE vtep VTEP # remote (EVPN) MAC
grcli flood vtep add ADDR vni 777 vrf underlay # BUM flood targetThis is the most common VXLAN config error, and the errno is informative once you know what to look for.
Symptom, one of:
-
Address not available(EADDRNOTAVAIL) - thelocal(VTEP) address is not a known local address in the encap VRF. -
Address already in use(EADDRINUSE) - the VNI+encap_vrf key is already used, or in VRF mode (L3 VNI) only one VXLAN per VRF is allowed. -
Network is unreachable(ENETUNREACH) - no route to reach the VTEP address in the encap VRF. -
Numerical result out of range(ERANGE) - VNI is 0 or > 16777215.
Cause and fix: the VXLAN local/VTEP address must exist on an interface and be
reachable in the target encap_vrf. If encap_vrf is omitted it defaults to
main. Configure the underlay first:
grcli interface add vrf underlay
grcli interface add port phy devargs 0000:18:00.2 vrf underlay
grcli address add 100.0.2.2/32 iface underlay
grcli interface add bridge br777
grcli interface add vxlan vni777 vni 777 local 100.0.2.2 domain br777 encap_vrf underlay
grcli interface add vlan hub.777 parent hub vlan_id 777 domain br777Symptom: ARP for a bridged/anycast IP is only answered by one node; ARP/ND requests are not seen on the far node.
Cause: EVPN ARP/ND suppression is a per-VXLAN-interface function
(neigh_suppress on) that deliberately blocks ARP/ND across tunnels. In grout
the drop shows as bridge_neigh_suppress_drop. To restore flood behavior, turn
it off when creating the bridge/vxlan interface (neigh_suppress off).
For anycast gateways, use the same MAC on each L2VNI across all VTEPs plus a static FDB entry; without the static FDB entry only one node answers for the MAC.
Symptom: ARP reply from the remote side is not seen on the grout port; trace
shows arp_input_request_drop. Reproduced on both VF and PF ports and still
open (DPDK/grout#545). Likely hardware L2/multicast filtering or VF
trust/spoof settings (spoof checking on). Check the VF trust/spoof
configuration on the PF. Note also that VF-to-VF traffic on the same PF may not
switch as expected (broadcast/ARP from one VF reaching the PF instead of the
sibling VF).
Symptom: with net_virtio_user/net_vhost in a bridge/VXLAN path, ARP and
ICMP look fine but TCP sessions stall or guest traffic fails.
Cause: grout enables RX checksum offload on all ethdevs including virtio/vhost; the vhost peer sends partial-checksum packets that are not completed before egress. This is an open issue (DPDK/grout#724, #756) with no merged fix at time of writing - track those issues.
Configure SRv6 encap/decap:
grcli tunsrc set 2001:db8::1
grcli nexthop add srv6 seglist 2001:db8::a 2001:db8::b encap h.encaps.red
grcli nexthop add srv6-local behavior end.dt6 table 100
grcli nexthop show type srv6
grcli nexthop show type srv6-localgrout implements a subset of SRv6 endpoint behaviors: End, End.X, End.T,
End.DT4, End.DT6 and End.DT46. Configuring an unsupported behavior in FRR
(e.g. End.DX4, End.DX6, End.DX2 or End.B6) is rejected, with the
seg6local action number in the zebra log:
ZEBRA: GROUT: unsupported seg6local action 6
(action 6 is End.DX4; the numbering is enum seg6local_action_t in FRR's
lib/srv6.h). Fix: use one of the supported behaviors, or drop the unsupported
SID from the FRR configuration.
When adding SRv6 nexthops/local SIDs, the errno tells you what is wrong:
-
EINVAL- empty seglist, or a non-IPv6 segment, or unknown behavior name. -
E2BIG- too many segments in the seglist. -
ENODEV- theEND.Xoutput interface does not exist. -
EPROTOTYPE- the resolvedEND.Xnexthop is not an L3 nexthop. -
EAFNOSUPPORT- theEND.XL3 nexthop is not IPv6. -
EDOM/ERANGE- fornext-csid, block/csid bits are not byte-multiples, or their sum exceeds 128.
Datapath drops to look for: sr6_local_invalid, sr6_local_no_transit,
sr6_local_not_allowed_upper, sr6_local_unexpected_upper, sr6_pkt_invalid,
sr6_source_no_route.