Linux / Fedora Core and nf_conntrack:

While testing and benchmarking some new features in Apache Traffic Server on my dev box, I started having really odd problems with inconsistently lost connections or really slow performance. Upon examing the logs, I found a large number of kernel barf like

Oct 29 11:53:51 loki kernel: nf_conntrack: table full, dropping packet.
Oct 29 11:53:51 loki kernel: nf_conntrack: table full, dropping packet.
Oct 29 11:53:51 loki kernel: nf_conntrack: table full, dropping packet.
Oct 29 11:53:51 loki kernel: nf_conntrack: table full, dropping packet.

Clearly this could not be good. I poked around, and could not find any ways to turn off this kernel module, at least on my Fedora Core 13 it seems compiled straight into the kernel (i.e. no .ko). There are ways to increase the table size (sysctl's) but that seemed like a band aid at best. Bummer. So, I went to the mountain (Noah F.) and asked for advice. After some poking around, we found that forcing iptables to ignore the conntracking helps, a lot. E.g.

iptables -t raw -I OUTPUT -j NOTRACK

 seems to do the trick for my particular case. This turns it off for all protocols, but you could probably limit it further (e.g. for tcp only with a "-p tcp" option). Also, you might want to do this on the input as well, e.g.

iptables -t raw -I PREROUTING -j NOTRACK

We found this old report against Fedora Core 10 as well, which describes the same problem, but no solution.

Hacking: 

Comments

thanks for this. i remember

thanks for this. i remember this issue killing our linux box. conntrack iis meant for linux firewalls and not for production systems that need to scale. anyhow, they bought 2 more linux servers and f5 big ip that now distributes the traffic to 4 linux boxes.