Wormhole traffic being blocked by iptables

wormhole
#1

I am having issues with relevant Wormhole traffic being blocked by the OS firewall. In my case I am using:

  • CentOS 7.7.1908
  • firewalld 0.6.3-2
  • wireguard-dkms 1:0.0.20200128-1
  • wireguard-tools 1:1.0.20191226-1
  • one “real” network interface managed by the OS
  • that interface is in the firewalld zone called “internal”
    ** the “internal” zone has a default rule/target that “rejects” packets without a defined rule allowing it
  • i have defined the following rules on the master node:
    ** allow ssh
    ** allow 6443, 2379, 2380, 10250, 10251, 10252 TCP for Kubernetes traffic
    ** allow 9806 for Wireguard
  • i have defined the following rules on the worker nodes:
    ** allow ssh
    ** allow 10250 and 30000-32767 TCP for Kubernetes traffic
    ** allow 9806 for Wireguard

NOTE: Sorry for the non-linked URLs below. Apparently as a new user to this forum I can only include 2 real links.

I installed one master and two (2) workers using Kubernetes 1.17 based on the Kubernetes ‘Installing kubeadm’ page. Next up, I installed Wireguard according to their ‘Installation - Wireguard’ page. Followed by the Wormhole instructions on the Github page. I did have to modify the instructions a bit due to the now deprecated PodSecurity API location. You can see my Github issue #17 with the Wormhole project for details on that.

This leaves me with a mostly working cluster. However when I look closely to kubectl get pods --all-namespaces output I see that my two (2) coredns pods are NOT ready.

root@kube-master00:~$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                    READY   STATUS    RESTARTS   AGE
kube-system   coredns-6955765f44-dx2h4                0/1     Running   4          20d
kube-system   coredns-6955765f44-jpgjz                0/1     Running   4          20d

If I stop the ‘firewalld’ service. Within a minute the two (2) coredns pods spin up and everything works as expected.

root@kube-master00:~$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                    READY   STATUS    RESTARTS   AGE
kube-system   coredns-6955765f44-dx2h4                1/1     Running   4          20d
kube-system   coredns-6955765f44-jpgjz                1/1     Running   4          20d

When I have ‘firewalld’ running and I am logging denied packets. I see a lot of this:

root@kube-master00:~$ tail -f /var/log/messages 
Feb  3 14:23:24 kube-master00 kernel: FINAL_REJECT: IN=wormhole-br0 OUT=ens33 
PHYSIN=vethb8345c2f MAC=b6:47:98:45:58:84:72:70:f9:6a:f5:16:08:00 SRC=10.244.0.19 
DST=172.16.222.2 LEN=45 TOS=0x00 PREC=0x00 TTL=63 ID=32767 DF PROTO=UDP 
SPT=38741 DPT=53 LEN=25 
Feb  3 14:23:24 kube-master00 kernel: FINAL_REJECT: IN=wormhole-br0 OUT= 
PHYSIN=vethb8345c2f MAC=b6:47:98:45:58:84:72:70:f9:6a:f5:16:08:00 SRC=10.244.0.19 
DST=172.16.222.129 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=64816 DF PROTO=TCP 
SPT=33154 DPT=6443 WINDOW=43690 RES=0x00 SYN URGP=0

I have tried permitting specific traffic that I see being blocked. Every time I open one up, another seems to now be blocked. I was hoping someone out here or someone from Gravitational could give me a list of the required iptables rules. The only documentation I see on this forum, or the Github page, mention allowing port 9806 for Wireguard. Which I have open already.

Oh, and I have tested this on multiple hosting platforms. GCP, VMware Fusion, and bare metal. They all result in the same outcome. The GCP through me for a bit of a loop cause they put their interfaces in the “trusted” zone. This zones uses a default target of “ACCEPT” which basically allows all traffic. Per specific rules on my network I can’t do that or leave the firewall off.

#2

Hi @brian-mantech thank you for the detailed question, it really helps to get an idea of what’s going on. And thanks for the github issue as well, I should be able to patch that later this week.

I haven’t tested wormhole explicitly with firewalld, but it looks like what’s going on is the virtual bridge interface created by wormhole is getting caught in the firewalld policy of default deny.

While you only have a single “real” interface towards the physical network, wormhole creates a linux bridge called wormhole-br0 as well as the wireguard interface (wormhole-wg0). CNI also creates a number of veth pairs for connecting pod network namespaces to the host network namespace for routing.

Stealing a diagram I use for training on how flannel operates (wormhole is very similar to flannel in the OS configuration):

image
image2663×1371 253 KB

cni0 = wormhole-br0
flannel.1 = wormhole-wg0

In the diagram, we can see that pods within their own network namespaces, are connected through the linux kernel using a veth pair to a virtual bridge. And from the virtual bridge traffic is routes towards the host network (eth0) or the overlay network (flannel.1/wormhole-wg0) based on the routing table.

If the virtual bridge is being set by iptables (firewalld) to deny traffic, all pod-to-pod or pod-to-host traffic will be caught by these firewall rules. Which is why you have to individually enable lots of rules, the firewall is basically acting as a firewall on all traffic from or to pods, even within the overlay network, by applying to these internal interfaces.

What suggests to me that this is the case, is in the /var/log/messages you included, we see that the IN=wormhole-br0 interface. And specifically this appears to be a pod trying to connect outwards to a DNS server running on IP 172.16.222.2.

I’m not super familiar with firewalld, but as I recall it’s also much more geared as an endpoint firewall, so it may be writing rules in a way that doesn’t expect traffic to be “routed” through the kernel. What we’re doing in building an overlay network, is using the linux kernel to route packets between the overlay/pod network and the host network for outgoing/incoming traffic, such as reaching a DNS server. This traffic doesn’t originate from the host in the same way as a local query, from the kernel perspective it’s routing the traffic from a source interface to a destination interface.

You mentioned running in a restrictive environment. The options depends a bit on what specifically those restrictions are.

The simplest would be to place the wormhole-br0 and probably the wormhole-wg0 interfaces into a trusted zone within firewalld, or write an iptables rule to allow all traffic from those interfaces. This would effectively “trust” all traffic originating on those interfaces, which may mean pods can connect outwards to anywhere on the network, and any internal traffic within the cluster is unfiltered. Then on the physical network, only the wormhole port is required by the firewall.

The next slightly more complicated option would be to install a kubernetes network policy controller. You may still need to tweak how the firewall rules are configured on the virtual interfaces so they don’t interfere with the network policy, but the network policy controller then set’s up the iptable rules to specifically allow or block internal traffic as per the policy given to kubernetes. While wormhole is designed to work with commonly available network policy controllers, I haven’t had the opportunity to test out this configuration (hopefully soon!).

I hope this helps / makes sense.

#3

Thanks, that made sense. Here is what I did to solve it.

firewall-cmd --zone=trusted --change-interface=wormhole-br0 --permanent
firewall-cmd --zone=trusted --change-interface=wormhole-wg0 --permanent
firewall-cmd --reload

Like you said I could also just accept all traffic from the source IP/network of the overlay network. Which I assigned to 10.244.0.0/16 when I initialized the cluster. kubeadm init --pod-network-cidr=10.244.0.0/16

firewall-cmd --zone=internal --add-rich-rule='rule family="ipv4" source address="10.244.0.0/16" accept' --permanent
firewall-cmd --zone=internal --change-interface=wormhole-br0 --permanent
firewall-cmd --zone=internal --change-interface=wormhole-wg0 --permanent
firewall-cmd --reload

Thanks again.