After the nftables rule ordering incident, I decided I was done testing router changes in production. Network namespaces give you a full Linux networking stack in isolation, for free, on a laptop. This is the pattern I use now.

The concept

A network namespace is a kernel-level isolated copy of the networking subsystem: its own interfaces, its own routing table, its own netfilter rules, its own sockets. You can build a tiny “network” of three namespaces connected by veth pairs, simulating client -> router -> server, apply your real rules on the router namespace, and run real traffic between client and server. Confidence in a ruleset goes up a lot when you have actually seen a SYN go through it.

The script

Here is my standard harness, saved as netns-harness.sh:

#!/bin/bash
set -e

# tear down if it exists
for ns in client router server; do
  ip netns del "$ns" 2>/dev/null || true
done

# create namespaces
ip netns add client
ip netns add router
ip netns add server

# veth pairs
ip link add veth-cr-c type veth peer name veth-cr-r
ip link add veth-rs-r type veth peer name veth-rs-s

# move ends into namespaces
ip link set veth-cr-c netns client
ip link set veth-cr-r netns router
ip link set veth-rs-r netns router
ip link set veth-rs-s netns server

# client side
ip -n client link set lo up
ip -n client link set veth-cr-c up
ip -n client addr add 10.0.1.2/24 dev veth-cr-c
ip -n client route add default via 10.0.1.1

# router side
ip -n router link set lo up
ip -n router link set veth-cr-r up
ip -n router link set veth-rs-r up
ip -n router addr add 10.0.1.1/24 dev veth-cr-r
ip -n router addr add 10.0.2.1/24 dev veth-rs-r
ip netns exec router sysctl -w net.ipv4.ip_forward=1

# server side
ip -n server link set lo up
ip -n server link set veth-rs-s up
ip -n server addr add 10.0.2.2/24 dev veth-rs-s
ip -n server route add default via 10.0.2.1

echo "harness up. test with: sudo ip netns exec client ping 10.0.2.2"

Run with sudo ./netns-harness.sh and you have a three-node network. sudo ip netns exec client ping 10.0.2.2 should work.

Applying real rules

The router namespace can load any nftables ruleset. The trick is that nft -f rules.nft from inside the namespace applies to that namespace:

sudo ip netns exec router nft -f /etc/nftables.d/edge.nft
sudo ip netns exec router nft list ruleset

This is the key. The ruleset you would deploy to production gets tested here first. If it drops the client’s pings, you know before production.

Testing a real scenario

Pretend I want to add a rule to block outgoing SYN to port 25 from any client behind the router (simulating blocking mail spam). I add to my ruleset:

table inet filter {
    chain forward {
        type filter hook forward priority 0; policy accept;
        tcp flags & (syn|ack) == syn tcp dport 25 drop
    }
}

Apply in the router netns. Then from the client:

sudo ip netns exec client curl -v telnet://10.0.2.2:25 -m 3
# curl: (28) Connection timeout
sudo ip netns exec client curl -v telnet://10.0.2.2:2525 -m 3
# curl: (7) Failed to connect

Wait, connection timeout for 25 is expected (SYN dropped), but “failed to connect” for 2525 is also expected if nothing is listening. Let me start something on the server:

sudo ip netns exec server python3 -m http.server 2525 &
sudo ip netns exec client curl -v http://10.0.2.2:2525
# HTTP/1.0 200 OK

Good. And port 25 is still blocked. Real evidence the rule does what I expect.

Testing failover or asymmetric routing

Namespaces are great for this too. I can add a second router namespace with its own veth links and simulate HA failover:

ip netns add router2
ip link add veth-c2 type veth peer name veth-c2r2
ip link set veth-c2 netns client
ip link set veth-c2r2 netns router2
# ... etc

Then inside client I add a second default route with a higher metric, and watch what happens when I bring router1 down. This is harder to do on a live router without risking an outage.

Capturing packets

tcpdump works inside a namespace too. I often run:

sudo ip netns exec router tcpdump -i any -n -vv

On a real router I sometimes cannot run tcpdump at all (read-only device, or paranoid about the load). In a namespace it is just a process.

A few gotchas

  • conntrack state is per-namespace. If your rules depend on established/related, make sure you exercise the rules with real flows, not just one-off probes.
  • ip netns exec changes many things besides the network stack; it also changes /proc/net, /sys/class/net, etc. Some tools get confused if they expect host-global sysfs.
  • If you use Docker or podman, they create their own namespaces that can collide with yours in weird ways. Run the harness in a clean environment.
  • IPv6 works the same way but you have to set addresses and routes explicitly; the harness above is IPv4-only for brevity.

My workflow

  1. Every ruleset change: apply in the netns first, run a small acceptance script that tests the cases I care about.
  2. If the change involves forwarding or routing, the acceptance script builds the full three-namespace topology and checks traffic end to end.
  3. The acceptance script is idempotent and lives next to the ruleset in git. CI runs it on every PR.

This has caught two bugs in the past six months that would have been edge incidents otherwise. The harness setup is a one-time 20-line script. The acceptance tests are specific to each change. Both are cheap. The payoff is massive peace of mind.

Reflection

Running a router in a namespace is not the same as running it in production. The namespace has no NICs, no real driver quirks, no link flapping. But for testing rulesets, routing logic, and policy changes, it is close enough to matter. Pair this with the occasional real-hardware burn-in test and you have a robust change-management story.

Related: see my post on nftables rule ordering for the reason I wrote all this in the first place.