Conntrack and the case of the vanishing SYN
Conntrack, the Linux kernel's connection tracking layer, is supposed to see every packet that traverses the network stack. But a simple experiment raises a puzzling question: when a firewall drops a TCP SYN packet, why does no corresponding entry appear in the conntrack table?
A quick test makes the mystery concrete. Set up a VM, connect over SSH, and inspect the conntrack table with conntrack -L. Despite the active SSH session sending packets constantly, the table is empty—even with the conntrack module loaded.
Why are there no entries in the conntrack table for SYN packets dropped by the firewall?
Conntrack doesn't see packets directly
The first surprise is that conntrack isn't called directly by the IPv4 or IPv6 network stacks. Walking the receive path step by step, you won't find any direct invocation of conntrack code. Instead, conntrack registers callbacks with the Netfilter framework, which embeds hooks into the network stack.
int ip_rcv(struct sk_buff *skb, struct net_device *dev, …)
{
…
return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING,
net, NULL, skb, dev, NULL,
ip_rcv_finish);
}Netfilter defines five hooks for the INET family (IPv4 and IPv6), and conntrack doesn't register its callbacks at module load time by default. That only happens when the enable_hooks parameter is set, which has been available since Linux v5.1. With the module loaded without this flag, callbacks remain unregistered and no traffic is tracked.
[vagrant@ct-vm ~]$ modinfo nf_conntrack
…
parm: enable_hooks:Always enable conntrack hooks (bool)Reloading the module with enable_hooks set changes everything. The SSH session now appears in the conntrack table, confirming that conntrack is invoked through Netfilter hooks. With that mechanism established, attention turns back to the original question: what happens when a firewall drops a SYN?
Observing hook activity
To test this, add an iptables rule to drop packets arriving on port tcp/2570. Then attempt a connection from outside the VM—for instance, with nc -z 192.168.122.204 2570—and list the conntrack table. No new entries appear. The dropped SYN generates no flow record.
[vagrant@ct-vm ~]$ sudo iptables -t filter -A INPUT -p tcp --dport 2570 -j DROPBut that doesn't mean conntrack never processed the packet. To find out, you can examine which callbacks conntrack has registered with each Netfilter hook. Netfilter stores these in instances of struct nf_hook_entries, reachable via the network namespace's Netfilter state (struct netns_nf).
struct netns_nf {
…
struct nf_hook_entries __rcu *hooks_ipv4[NF_INET_NUMHOOKS];
struct nf_hook_entries __rcu *hooks_ipv6[NF_INET_NUMHOOKS];
…
}struct nf_hook_entries has an unusual memory layout: a count followed by two arrays of equal size. One array holds hook function pointers, the other holds pointers to struct nf_hook_ops, which include priority information that determines invocation order.
[vagrant@ct-vm ~]$ sudo drgn
drgn 0.0.8 (using Python 3.9.1, without libkdumpfile)
…
>>> pre_routing_hook = prog['init_net'].nf.hooks_ipv4[0]
>>> for i in range(0, pre_routing_hook.num_hook_entries):
... pre_routing_hook.hooks[i].hook
...
(nf_hookfn *)ipv4_conntrack_defrag+0x0 = 0xffffffffc092c000
(nf_hookfn *)ipv4_conntrack_in+0x0 = 0xffffffffc093f290
>>>Using drgn, a programmable C debugger for the Linux kernel, you can walk this structure to list all callbacks for each hook. The output reveals that conntrack registers two callbacks with PRE_ROUTING: ipv4_conntrack_defrag and ipv4_conntrack_in.
[vagrant@ct-vm ~]$ sudo /vagrant/tools/list-nf-hooks
? ipv4 PRE_ROUTING
-400 → ipv4_conntrack_defrag ☜ conntrack callback
-300 → iptable_raw_hook
-200 → ipv4_conntrack_in ☜ conntrack callback
-150 → iptable_mangle_hook
-100 → nf_nat_ipv4_in
? ipv4 LOCAL_IN
-150 → iptable_mangle_hook
0 → iptable_filter_hook
50 → iptable_security_hook
100 → nf_nat_ipv4_fn
2147483647 → ipv4_confirm
…Tracing the callbacks
To confirm that these callbacks actually run when a SYN arrives on port tcp/2570, attach BPF kprobes to the function entries. bpftrace makes this straightforward with a high-level scripting language inspired by AWK, avoiding the usual C and clang -target bpf compilation cycle.
kprobe:ipv4_conntrack_defrag,
kprobe:ipv4_conntrack_in
{
$skb = (struct sk_buff *)arg1;
$iph = (struct iphdr *)($skb->head + $skb->network_header);
$th = (struct tcphdr *)($skb->head + $skb->transport_header);
if ($iph->protocol == 6 /* IPPROTO_TCP */ &&
$th->dest == 2570 /* htons(2570) */ &&
$th->syn == 1) {
time("%H:%M:%S ");
printf("%s:%u > %s:%u tcp syn %s\n",
ntop($iph->saddr),
(uint16)($th->source << 8) | ($th->source >> 8),
ntop($iph->daddr),
(uint16)($th->dest << 8) | ($th->dest >> 8),
func);
}
}Running the trace while connecting from outside shows both callbacks are triggered by the SYN. The conntrack callbacks clearly process the packet. Yet no flow ends up in the table. So what happens inside these callbacks that prevents a persistent entry?
Following the call chain
To answer that, trace the call chain entering the conntrack callback using the function_graph tracer built into Ftrace. Since all incoming traffic passes through PRE_ROUTING—including SSH traffic, which would pollute the trace—switch to a serial console connection via virsh before starting the trace.
host $ virsh -c qemu:///session list
Id Name State
-----------------------------------
1 conntrack_default running
host $ virsh -c qemu:///session console conntrack_default
Once connected to the console and logged into the VM, we can record the call chain using the trace-cmd wrapper for Ftrace:
[vagrant@ct-vm ~]$ sudo trace-cmd start -p function_graph -g ipv4_conntrack_defrag -g ipv4_conntrack_in
plugin 'function_graph'
[vagrant@ct-vm ~]$ # … connect from the host with `nc -z 192.168.122.204 2570` …
[vagrant@ct-vm ~]$ sudo trace-cmd stop
[vagrant@ct-vm ~]$ sudo cat /sys/kernel/debug/tracing/trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
1) 1.219 us | finish_task_switch();
1) 3.532 us | ipv4_conntrack_defrag [nf_defrag_ipv4]();
1) | ipv4_conntrack_in [nf_conntrack]() {
1) | nf_conntrack_in [nf_conntrack]() {
1) 0.573 us | get_l4proto [nf_conntrack]();
1) | nf_ct_get_tuple [nf_conntrack]() {
1) 0.487 us | nf_ct_get_tuple_ports [nf_conntrack]();
1) 1.564 us | }
1) 0.820 us | hash_conntrack_raw [nf_conntrack]();
1) 1.255 us | __nf_conntrack_find_get [nf_conntrack]();
1) | init_conntrack.constprop.0 [nf_conntrack]() { ❷
1) 0.427 us | nf_ct_invert_tuple [nf_conntrack]();
1) | __nf_conntrack_alloc [nf_conntrack]() { ❶
…
1) 3.680 us | }
…
1) + 15.847 us | }
…
1) + 34.595 us | }
1) + 35.742 us | }
…
[vagrant@ct-vm ~]$The trace shows an allocation, __nf_conntrack_alloc() (❶), inside init_conntrack() (❷). This function creates a struct nf_conn, the object that represents a tracked connection. Critically, the trace then shows this object being pushed onto a list of unconfirmed connections.
The conntrack documentation explains that an "unconfirmed" connection has been seen but hasn't yet passed through all the checks needed to be marked as confirmed. That suggests the dropped SYN might be visible in a separate, unconfirmed table rather than the main conntrack view.
The unconfirmed table holds no answers
Checking the unconfirmed table yields bad news: the flow isn't there either. If a struct nf_conn object was added to the unconfirmed list, something must have removed it before the table was inspected. That points to nf_ct_del_from_dying_or_unconfirmed_list(), the counterpart to the function that adds entries to the list.
A BPF tracing program that records kernel stack traces can reveal when and where this deletion function runs. Triggering a connection attempt while tracing shows a clear chain:
[vagrant@ct-vm ~]$ sudo bpftrace -e 'kprobe:nf_ct_del_from_dying_or_unconfirmed_list { @[kstack()] = count(); exit(); }'
Attaching 1 probe...
@[
nf_ct_del_from_dying_or_unconfirmed_list+1 ❹
destroy_conntrack+78
nf_conntrack_destroy+26
skb_release_head_state+78
kfree_skb+50 ❸
nf_hook_slow+143 ❷
ip_local_deliver+152 ❶
ip_sublist_rcv_finish+87
ip_sublist_rcv+387
ip_list_rcv+293
__netif_receive_skb_list_core+658
netif_receive_skb_list_internal+444
napi_complete_done+111
…
]: 1
[vagrant@ct-vm ~]$The captured stack trace shows the packet being destroyed on the local delivery path (❶)—specifically at the LOCAL_IN Netfilter hook (❷)—during sk_buff destruction (❸), which prompts conntrack to remove the unconfirmed flow entry (❹).
That iptables -j DROP rule has a significant side effect: it cleans up the conntrack unconfirmed table. The flow entry exists for only the briefest moment before being removed along with the dropped packet. This is why the dropped SYN never leaves a trace in the conntrack table, no matter which table you inspect.
Confirming a flow: too little, too late?
Earlier, the hook listing showed a second conntrack callback that we set aside: ipv4_confirm. Now it takes center stage. This is the “confirmation point” from the conntrack(8) man page—the step where a flow graduates from the unconfirmed table into the main conntrack table.
What makes this callback special is its registration priority: 2,147,483,647. That is the maximum positive value of a 32-bit signed integer, and, in Netfilter terms, the lowest possible priority. Registering at this level guarantees that ipv4_confirm runs last among all callbacks.
Why insist on running last? The design intent is that a flow should only be promoted to the main table once its packet has cleared the firewall. Confirmation before that point would record flows that never actually made it through.
Playing with priorities
Netfilter allows more than one callback to share a priority. When that happens, registration order decides the execution order. That opens up an interesting experiment—purely educational.
iptables cannot help here: its Netfilter callbacks come with hard-coded priorities. nftables, the successor to iptables, offers more flexibility. It lets us create rule chains at arbitrary priorities.
The trick is to register our nftables chain after the firewall filters but before conntrack registers its hooks. In the hook list, still ordered by priority, we can slide our earlier-registered drop rule to the end of the sort order.
After removing the iptables drop rule for tcp/2570 and unregistering conntrack:
vm # iptables -t filter -F
vm # rmmod nf_conntrack_netlink nf_conntrackAdd the nftables drop rule at the lowest possible priority:
vm # nft add table ip my_table
vm # nft add chain ip my_table my_input { type filter hook input priority 2147483647 \; }
vm # nft add rule ip my_table my_input tcp dport 2570 counter drop
vm # nft -a list ruleset
table ip my_table { # handle 1
chain my_input { # handle 1
type filter hook input priority 2147483647; policy accept;
tcp dport 2570 counter packets 0 bytes 0 drop # handle 4
}
}Now re-register the conntrack hooks:
vm # modprobe nf_conntrack enable_hooks=1The LOCAL_IN hook now shows the following callback order:
vm # /vagrant/tools/list-nf-hooks
…
? ipv4 LOCAL_IN
-150 → iptable_mangle_hook
0 → iptable_filter_hook
50 → iptable_security_hook
100 → nf_nat_ipv4_fn
2147483647 → ipv4_confirm, nft_do_chain_ipv4
…With the firewall rule dropping traffic to tcp/2570, what happens to conntrack’s view of that packet?
vm # conntrack -L
tcp 6 115 SYN_SENT src=192.168.122.1 dst=192.168.122.204 sport=54868 dport=2570 [UNREPLIED] src=192.168.122.204 dst=192.168.122.1 sport=2570 dport=54868 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
conntrack v1.4.5 (conntrack-tools): 1 flow entries have been shown.Conntrack moved the flow into the main table even though the packet was dropped. The chain order inversed the usual precedence: what normally runs first (conntrack) now runs after the drop, and the rule that decides a packet’s fate executes last.
Beyond the trick
The takeaway here is not a recommendation to reorder Netfilter hooks in production. Describing conntrack internals was a reason to demonstrate the bigger point: modern tools make it straightforward to inspect the Linux network stack in action. Utilities like drgn, bpftrace, and Ftrace, combined with a kernel source cross-referencer, let you watch a live system and see exactly where events fire or fail to fire.
The kernel admits those who dig. Just expect it to be absorbing.
