Deep dive in externals IPs of kubernetes
Written by Jacques Roussel, Consultant Cloud @Objectif Libre
Today, I will speak about services objects present in Kubernetes. I will focus on a externals IPs on bare metal deployment because it’s already manage on cloud deployment, and I will show how they work.
Then I will speak about a new project called Metallb that can avoid to use externals IPs, which may easy your life.
Prepare your environment
To follow this tutorial, you will need a functionnal minikube. The commands below work with Ubuntu 16.04:
(on your host) $ sudo apt-get update && sudo apt-get install -y virtualbox curl (on your host) $ curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.24.1/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/ (on your host) $ minikube start --memory 4096
You should now have a working kubernetes installation. To make sure everything works as expected, run:
(on your host) $ kubectl get no NAME STATUS AGE VERSION minikube Ready 15h v1.8.0
If minikube is ready, you can follow this article. So let’s go !
Externals IPs
First of all, we need a deployement and a service to play with:
(on your host) $ kubectl run nginx --image=nginx --port=80 --replicas=3 (on your host) $ kubectl expose deployment nginx --type NodePort
You can go on when the pods are running :
(on your host) $ kubectl get po -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-85dfb4bc54-9vjhn 1/1 Running 1 17h 172.17.0.3 minikube nginx-85dfb4bc54-hxcql 1/1 Running 1 17h 172.17.0.4 minikube nginx-85dfb4bc54-l2jxp 1/1 Running 1 17h 172.17.0.8 minikube
It’s time to explore the service:
(on your host) $ kubectl get svc nginx NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx 10.99.239.22 <nodes> 80:30916/TCP 8s
At this stage, there is no external IP setup.
Let’s configure it:
(on your host) $ kubectl patch svc nginx --patch '{"spec": {"externalIPs": ["10.22.0.0"]}}'
(on your host) $ kubectl get svc nginx
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx 10.99.239.22 10.22.0.0 80:30916/TCP 48s
That looks good. We now have an external IP associated to our service.
And so ? We know that we can’t reach this IP from outside the cluster.
Maybe we can from inside?
(on your host) $ minikube ssh (on minikube) $ curl http://10.22.0.0 curl: (7) Failed to connect to 10.22.0.0 port 80: Network is unreachable
It doesn’t work. But:
(on minikube) $ curl http://10.99.239.22
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
It’s working! Ok but we don’t care about the CLUSTER-IP. We want to use an EXTERNAL-IP.
So let’s do it:
(on minikube) $ sudo ip a a 10.22.0.0 dev lo
And now:
(on minikube) $ curl http://10.22.0.0
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
Our external IP is working! But how?
(on minikube) $ sudo iptables-save |grep "10.22.0.0" -A KUBE-SERVICES -d 10.22.0.0/32 -p tcp -m comment --comment "default/nginx: external IP" -m tcp --dport 80 -j KUBE-MARK-MASQ -A KUBE-SERVICES -d 10.22.0.0/32 -p tcp -m comment --comment "default/nginx: external IP" -m tcp --dport 80 -m physdev ! --physdev-is-in -m addrtype ! --src-type LOCAL -j KUBE-SVC-4N57TFCL4MD7ZTDA -A KUBE-SERVICES -d 10.22.0.0/32 -p tcp -m comment --comment "default/nginx: external IP" -m tcp --dport 80 -m addrtype --dst-type LOCAL -j KUBE-SVC-4N57TFCL4MD7ZTDA
The first rule sends paquets to the KUBE-MARK-MASQ target:
(on minikube) $ sudo iptables-save |grep "A KUBE-MARK-MASQ" -A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
This mark is not important for our topic. The second rule matches only when paquets come from a pod, so deals with internal communication.
The last rule is used when the communication comes from outside. In our case, from the minikube host.
(on minikube) $ sudo iptables-save|grep "A KUBE-SVC-4N57TFCL4MD7ZTDA" -A KUBE-SVC-4N57TFCL4MD7ZTDA -m comment --comment "default/nginx:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-RETR7JMLNGKOJSQK -A KUBE-SVC-4N57TFCL4MD7ZTDA -m comment --comment "default/nginx:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-UZXILYFQQ2IZUWN5 -A KUBE-SVC-4N57TFCL4MD7ZTDA -m comment --comment "default/nginx:" -j KUBE-SEP-Z5JB7O4CUBWMYZUU
These rules load-balance the traffic between our pods. For example, “KUBE-SEP-RETR7JMLNGKOJSQK”:
(on minikube) $ sudo iptables-save|grep "A KUBE-SEP-RETR7JMLNGKOJSQK" -A KUBE-SEP-RETR7JMLNGKOJSQK -s 172.17.0.3/32 -m comment --comment "default/nginx:" -j KUBE-MARK-MASQ -A KUBE-SEP-RETR7JMLNGKOJSQK -p tcp -m comment --comment "default/nginx:" -m tcp -j DNAT --to-destination 172.17.0.3:80 (on your host) $ kubectl get po -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-85dfb4bc54-9vjhn 1/1 Running 1 17h 172.17.0.3 minikube nginx-85dfb4bc54-hxcql 1/1 Running 1 17h 172.17.0.4 minikube nginx-85dfb4bc54-l2jxp 1/1 Running 1 17h 172.17.0.8 minikube
Here the target is the pod with IP 172.17.0.3.
That is the trick: when you add an EXTERNAL-IP to a service, k8s add iptables rules on each host to expose it. And that’s it. The cluster doesn’t manage the setup of the EXTERNAL-IP by default.
So we just learnt how an external ip works on k8s.
A problem stands. If you want to use it, you have to manage different things:
1. Add the IP on a node
2. Route the IP to the correct node
3. Possibly record a DNS entry
Metallb
Metallb is a new project, opensourced by Google end of 2017. Its goal is to manage external IPs on k8s bare-metal deployments.
Metallb will add the IP on a host and annonces this IP to your routers using the BGP protocol.
Let’s clean our environment:
(on your host) $ kubectl delete svc nginx service "nginx" deleted
Let’s setup metallb:
(on your host) $ kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.2.1/manifests/metallb.yaml namespace "metallb-system" created clusterrole "metallb-system:controller" created clusterrole "metallb-system:bgp-speaker" created role "config-watcher" created serviceaccount "controller" created serviceaccount "bgp-speaker" created clusterrolebinding "metallb-system:controller" created clusterrolebinding "metallb-system:bgp-speaker" created rolebinding "config-watcher" created deployment "controller" created daemonset "bgp-speaker" created
In order to check that everything is ok:
(on your host) $ kubectl get po -n metallb-system NAME READY STATUS RESTARTS AGE bgp-speaker-w7cn6 1/1 Running 0 55s controller-7fb5785458-ls9vp 1/1 Running 0 55s
Now we can configure the service:
(on your host) $ cat <<EOF > config-metallb.yml apiVersion: v1 kind: ConfigMap metadata: namespace: metallb-system name: config data: config: | peers: - peer-address: 10.0.0.1 peer-asn: 100 my-asn: 42 address-pools: - name: default cidr: - 192.168.10.0/24 advertisements: - aggregation-length: 32 EOF
In the peers section, you declare a list of the BGP routers that will learn routes. Then you configure differents address pools that will be used in your services.
(on your host) $ kubectl create -f config-metallb.yml configmap "config" created
To keep this tutorial simple, we stay focused on external IPs. If you want to check the BGP announcement, you can configure a host with Bird for example.
(on your host) $ kubectl expose deployment nginx --type LoadBalancer (on your host) $ kubectl get svc nginx NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx 10.96.58.234 192.168.10.0 80:32116/TCP 26s (on your host) $ minikube ssh (on minikube) $ curl http://192.168.10.0 (on minikube) $ sudo iptables-save |grep "192.168.10.0" -A KUBE-SERVICES -d 192.168.10.0/32 -p tcp -m comment --comment "default/nginx: loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-4N57TFCL4MD7ZTDA (on minikube) $ sudo iptables-save |grep "A KUBE-FW-4N57TFCL4MD7ZTDA" -A KUBE-FW-4N57TFCL4MD7ZTDA -m comment --comment "default/nginx: loadbalancer IP" -j KUBE-MARK-MASQ -A KUBE-FW-4N57TFCL4MD7ZTDA -m comment --comment "default/nginx: loadbalancer IP" -j KUBE-SVC-4N57TFCL4MD7ZTDA -A KUBE-FW-4N57TFCL4MD7ZTDA -m comment --comment "default/nginx: loadbalancer IP" -j KUBE-MARK-DROP
The rule “-A KUBE-SERVICES -d 192.168.10.0/32 -p tcp -m comment –comment “default/nginx: loadbalancer IP” -m tcp –dport 80 -j KUBE-FW-4N57TFCL4MD7ZTDA” is a bit different from previous rules because it allows an IP that is not local. So we don’t need to set the IP on the host:
(on minikube) $ sudo ip a|grep -E "192.168.10.0|10.22.0.0" inet 10.22.0.0/32 scope global lo (on minikube) $ curl http://192.168.10.0
Conclusion
If you choose to use standalone EXTERNAL-IP configurations, you have to manage local IPs on your hosts, and manage routing with externals tools.
But if you prefer automation (like I do), you can have Metallb do everything for you. But be careful: it’s a brand new project, and if you use it now,you will be early adopters… with everything implied.
