IP externes

Deep dive in externals IPs of kubernetes

Written by Jacques Roussel, Consultant Cloud @Objectif Libre

Today, I will speak about services objects present in Kubernetes. I will focus on a externals IPs on bare metal deployment because it’s already manage on cloud deployment, and I will show how they work.

Then I will speak about a new project called Metallb that can avoid to use externals IPs, which may easy your life.

Prepare your environment

To follow this tutorial, you will need a functionnal minikube. The commands below work with Ubuntu 16.04:

(on your host) $ sudo apt-get update && sudo apt-get install -y virtualbox curl
(on your host) $ curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.24.1/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
(on your host) $ minikube start --memory 4096

You should now have a working kubernetes installation. To make sure everything works as expected, run:

(on your host) $ kubectl get no
NAME STATUS AGE VERSION
minikube Ready 15h v1.8.0

If minikube is ready, you can follow this article. So let’s go !

 

Externals IPs

First of all, we need a deployement and a service to play with:

(on your host) $ kubectl run nginx --image=nginx --port=80 --replicas=3
(on your host) $ kubectl expose deployment nginx --type NodePort

You can go on when the pods are running :

(on your host) $ kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-85dfb4bc54-9vjhn 1/1 Running 1 17h 172.17.0.3 minikube
nginx-85dfb4bc54-hxcql 1/1 Running 1 17h 172.17.0.4 minikube
nginx-85dfb4bc54-l2jxp 1/1 Running 1 17h 172.17.0.8 minikube

It’s time to explore the service:

(on your host) $ kubectl get svc nginx
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx 10.99.239.22 <nodes> 80:30916/TCP 8s

At this stage, there is no external IP setup.

Let’s configure it:

(on your host) $ kubectl patch svc nginx --patch '{"spec": {"externalIPs": ["10.22.0.0"]}}'
(on your host) $ kubectl get svc nginx
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx 10.99.239.22 10.22.0.0 80:30916/TCP 48s

That looks good. We now have an external IP associated to our service.

And so ? We know that we can’t reach this IP from outside the cluster.

Maybe we can from inside?

(on your host) $ minikube ssh
(on minikube) $ curl http://10.22.0.0
curl: (7) Failed to connect to 10.22.0.0 port 80: Network is unreachable

It doesn’t work. But:

(on minikube) $ curl http://10.99.239.22
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
 body {
 width: 35em;
 margin: 0 auto;
 font-family: Tahoma, Verdana, Arial, sans-serif;
 }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

It’s working! Ok but we don’t care about the CLUSTER-IP. We want to use an EXTERNAL-IP.

So let’s do it:

(on minikube) $ sudo ip a a 10.22.0.0 dev lo

And now:

(on minikube) $ curl http://10.22.0.0
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
 body {
 width: 35em;
 margin: 0 auto;
 font-family: Tahoma, Verdana, Arial, sans-serif;
 }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Our external IP is working! But how?

(on minikube) $ sudo iptables-save |grep "10.22.0.0"
-A KUBE-SERVICES -d 10.22.0.0/32 -p tcp -m comment --comment "default/nginx: external IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.22.0.0/32 -p tcp -m comment --comment "default/nginx: external IP" -m tcp --dport 80 -m physdev ! --physdev-is-in -m addrtype ! --src-type LOCAL -j KUBE-SVC-4N57TFCL4MD7ZTDA
-A KUBE-SERVICES -d 10.22.0.0/32 -p tcp -m comment --comment "default/nginx: external IP" -m tcp --dport 80 -m addrtype --dst-type LOCAL -j KUBE-SVC-4N57TFCL4MD7ZTDA

The first rule sends paquets to the KUBE-MARK-MASQ target:

(on minikube) $ sudo iptables-save |grep "A KUBE-MARK-MASQ"
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000

This mark is not important for our topic. The second rule matches only when paquets come from a pod, so deals with internal communication.

The last rule is used when the communication comes from outside. In our case, from the minikube host.

(on minikube) $ sudo iptables-save|grep "A KUBE-SVC-4N57TFCL4MD7ZTDA"
-A KUBE-SVC-4N57TFCL4MD7ZTDA -m comment --comment "default/nginx:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-RETR7JMLNGKOJSQK
-A KUBE-SVC-4N57TFCL4MD7ZTDA -m comment --comment "default/nginx:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-UZXILYFQQ2IZUWN5
-A KUBE-SVC-4N57TFCL4MD7ZTDA -m comment --comment "default/nginx:" -j KUBE-SEP-Z5JB7O4CUBWMYZUU

These rules load-balance the traffic between our pods. For example, “KUBE-SEP-RETR7JMLNGKOJSQK”:

(on minikube) $ sudo iptables-save|grep "A KUBE-SEP-RETR7JMLNGKOJSQK"
-A KUBE-SEP-RETR7JMLNGKOJSQK -s 172.17.0.3/32 -m comment --comment "default/nginx:" -j KUBE-MARK-MASQ
-A KUBE-SEP-RETR7JMLNGKOJSQK -p tcp -m comment --comment "default/nginx:" -m tcp -j DNAT --to-destination 172.17.0.3:80

(on your host) $ kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-85dfb4bc54-9vjhn 1/1 Running 1 17h 172.17.0.3 minikube
nginx-85dfb4bc54-hxcql 1/1 Running 1 17h 172.17.0.4 minikube
nginx-85dfb4bc54-l2jxp 1/1 Running 1 17h 172.17.0.8 minikube

Here the target is the pod with IP 172.17.0.3.

That is the trick: when you add an EXTERNAL-IP to a service, k8s add iptables rules on each host to expose it. And that’s it. The cluster doesn’t manage the setup of the EXTERNAL-IP by default.

So we just learnt how an external ip works on k8s.

A problem stands. If you want to use it, you have to manage different things:

1. Add the IP on a node
2. Route the IP to the correct node
3. Possibly record a DNS entry

Metallb

Metallb is a new project, opensourced by Google end of 2017. Its goal is to manage external IPs on k8s bare-metal deployments.

Metallb will add the IP on a host and annonces this IP to your routers using the BGP protocol.

Let’s clean our environment:

(on your host) $ kubectl delete svc nginx
service "nginx" deleted

Let’s setup metallb:

(on your host) $ kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.2.1/manifests/metallb.yaml
namespace "metallb-system" created
clusterrole "metallb-system:controller" created
clusterrole "metallb-system:bgp-speaker" created
role "config-watcher" created
serviceaccount "controller" created
serviceaccount "bgp-speaker" created
clusterrolebinding "metallb-system:controller" created
clusterrolebinding "metallb-system:bgp-speaker" created
rolebinding "config-watcher" created
deployment "controller" created
daemonset "bgp-speaker" created

In order to check that everything is ok:

(on your host) $ kubectl get po -n metallb-system
NAME READY STATUS RESTARTS AGE
bgp-speaker-w7cn6 1/1 Running 0 55s
controller-7fb5785458-ls9vp 1/1 Running 0 55s

Now we can configure the service:

(on your host) $ cat <<EOF > config-metallb.yml
apiVersion: v1
kind: ConfigMap
metadata:
 namespace: metallb-system
 name: config
data:
 config: |
 peers:
 - peer-address: 10.0.0.1
 peer-asn: 100
 my-asn: 42
 address-pools:
 - name: default
 cidr:
 - 192.168.10.0/24
 advertisements:
 - aggregation-length: 32
EOF

In the peers section, you declare a list of the BGP routers that will learn routes. Then you configure differents address pools that will be used in your services.

(on your host) $ kubectl create -f config-metallb.yml 
configmap "config" created

To keep this tutorial simple, we stay focused on external IPs. If you want to check the BGP announcement, you can configure a host with Bird for example.

(on your host) $ kubectl expose deployment nginx --type LoadBalancer
(on your host) $ kubectl get svc nginx
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx 10.96.58.234 192.168.10.0 80:32116/TCP 26s
(on your host) $ minikube ssh
(on minikube) $ curl http://192.168.10.0
(on minikube) $ sudo iptables-save |grep "192.168.10.0"
-A KUBE-SERVICES -d 192.168.10.0/32 -p tcp -m comment --comment "default/nginx: loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-4N57TFCL4MD7ZTDA
(on minikube) $ sudo iptables-save |grep "A KUBE-FW-4N57TFCL4MD7ZTDA"
-A KUBE-FW-4N57TFCL4MD7ZTDA -m comment --comment "default/nginx: loadbalancer IP" -j KUBE-MARK-MASQ
-A KUBE-FW-4N57TFCL4MD7ZTDA -m comment --comment "default/nginx: loadbalancer IP" -j KUBE-SVC-4N57TFCL4MD7ZTDA
-A KUBE-FW-4N57TFCL4MD7ZTDA -m comment --comment "default/nginx: loadbalancer IP" -j KUBE-MARK-DROP

The rule “-A KUBE-SERVICES -d 192.168.10.0/32 -p tcp -m comment –comment “default/nginx: loadbalancer IP” -m tcp –dport 80 -j KUBE-FW-4N57TFCL4MD7ZTDA” is a bit different from previous rules because it allows an IP that is not local. So we don’t need to set the IP on the host:

(on minikube) $ sudo ip a|grep -E "192.168.10.0|10.22.0.0"
 inet 10.22.0.0/32 scope global lo
(on minikube) $ curl http://192.168.10.0

Conclusion

If you choose to use standalone EXTERNAL-IP configurations, you have to manage local IPs on your hosts, and manage routing with externals tools.

But if you prefer automation (like I do), you can have Metallb do everything for you. But be careful: it’s a brand new project, and if you use it now,you will be early adopters… with everything implied.