How to enable IPVS mode on K8s (Kube-proxy)

gravity
#1

I’d like to enable IPVS on Kube-proxy so that my Calico CNI, installed via networkInstall, will use it. I’ve tried creating a custom Planet with a --proxy-mode=ipvs set within the kube-proxy.service file but Calico doesn’t seem to notice Kube-proxy in this mode and the kube-ipvs0 interface (for example) is not created for Services.

Getting the following error within Planet for the kube-proxy service:

server_others.go:259] can't determine whether to use ipvs proxy, error: error getting ipset version, error: executable file not found in $PATH

Are there additional steps to enable IPVS?

#2

So it looks like the binary ipset is not included within Planet’s $PATH. Manually testing using gravity enter and apt-get update / apt-get install ipset and then restarting kube-proxy with service kube-proxy restart solved the problem. Looking at journalctl -u kube-proxy shows IPVS mode enabled, and confirmed with the kube-ipvs0 interface enabled on the host.

So now the question is, if I wanted to modify the Planet container to come with ipset by default, instead of running a script during install to do this, which Planet files would I need to modify?

#3

Hi @mmelin,

Information about building custom system container(Planet) you can find here.

#4

Thank you for pointing this out!

#5

In Gravity 5.xx, Planet’s kubernetes services are referenced in /lib/systemd/system/kube-<service-name>.service. I can use sed within a Dockerfile to update these flags in these files (such as --myflag==foobar).

In Planet 6.1.4 I see some of these flags within the files have changed to variables. How would I customize say APIServer flags when the data I’d like to customize is located within these variables?

For example I’d like to disable a feature-gate, but these are now located within what looks like the $KUBE_APISERVER_FLAGS within the /lib/systemd/system/kube-apiserver.service file.

I tried using the --config=envars.yaml option and creating a new $KUBE_COMPONENTS_FLAG env variable, but this did not work and now the API server has two such variables:

  CGroup: /system.slice/-planet-4b0725e6-0748-4365-82fd-4d356ef16e2a.scope/system.slice/kube-apiserver.service
               └─1178 /usr/bin/kube-apiserver --insecure-port=0 <...truncated...> --feature-gates=AllAlpha=true,APIResponseCompression=false,BoundServiceAccountTokenVolume=false,CSIDriverRegistry=false,CSINodeInfo=false,KubeletPodResources=false,ServerSideApply=false --feature-gates=AllAlpha=true,APIResponseCompression=false,BoundServiceAccountTokenVolume=false,CSIDriverRegistry=false,CSINodeInfo=false,KubeletPodResources=false

node01:/$ cat /etc/container-environment
KUBE_COMPONENT_FLAGS="--feature-gates=AllAlpha=true,APIResponseCompression=false,BoundServiceAccountTokenVolume=false,CSIDriverRegistry=false,CSINodeInfo=false,KubeletPodResources=false,ServerSideApply=false --feature-gates=AllAlpha=true,APIResponseCompression=false,BoundServiceAccountTokenVolume=false,CSIDriverRegistry=false,CSINodeInfo=false,KubeletPodResources=false"