MobyDocker

This blog post is an introduction to LinuxKit, a component of the Moby Project to build secure, portable, and lean operating systems based on containers.

Intended audience: cloud developer and architects, any tech desiring to hack with containers, cloud, immutable OS and CI/CD.

By Maxime Cottret, Cloud Consultant.

LinuxKit: a Dockerfile for building VM ?

Introduction

Most cloud user may one day have to create a VM image dedicated to a specific application (i.e. a virtual appliance).

What are the alternatives to do this:

  • the manual way: boot a VM, configure it, save a snapshot. This is obviously not the most efficient method (even if you can improve it by using Ansible ou Puppet for provisionning) and you are stuck with a specific VM image format.
  • the Packer way: Among all the good stuff from Hashicorp, Packer allows building identical machine images for multiple platforms from a single source configuration and can easily be integrated in CI/CD. Packer actually does not provide a complete abstraction and you may have to explicitely provide configuration for each plateforms you want to support (kickstart for Redhat/Centos, preseed for Ubuntu/Debian, etc). Alternatives are Diskimage-builder, Oz, VeeWEee, etc. Note that all this tools allows to build dedicated VM images from traditionnal linux OS bases (ubuntu, centos, arch, etc)

Coping with the difficulty to easily provision the Docker Plateform on different systems like MacOS, Windows, AWS, GCE, etc, Docker decided to create its own dedicated VM image builder with the help of containers and unikernels, LinuxKit:

  • Build images from scratch to the bare minimum needed, leading to very small image with a minimal attach surface.
  • Everything in the image runs in containers, from init to services
  • Stateless and immutable images, but persistent storage can be attached
  • Simple single yaml file configuration.

Follow the Moto “build, ship, run”, Docker tries to bring the ease of use of a Dockerfile and docker image build to the VM world.

To illustrate this blog post, we need to choose a specific application to install and configure in our virtual appliance image … I find LDAP to be a good candidate so let’s try it !!

 

Installation

Linuxkit provides releases on its Github repository but can also be installed directly as a Go package or using Homebrew on MacOS X.

Check LinuxKit Github Repo and choose the method that suit you the best.

You also need to install Docker and a local hypervisor (qemu, vbox, etc) for testing.

Building our LDAP appliance

First, let see how a LinuxKit image is designed.

Linuxkit images are built on a minimal kernel and init system whose only purpose is to run the containerD daemon. Any further processes or services are run as containers.

LinuxKit defines three types of containers:

  • on-boot and shutdown: those containers are run-once processes used to configure the image instance on boot or clean up on shutdown.
  • service: service container are long-run processes

The purpose of LinuxKit is clearly to build dedicated linux system and not generic OS like Ubuntu or Arch.

The initial Docker use case was the creation of a minimal linux OS dedicated to run containers for the VM bundled in Docker4Max: for this, you just need a kernel and the minimal subsystem to run the Docker engine.

Image specification

To specify LinuxKit image, you just need a yaml file describing all the components listed above with LinuxKit packages or  OCI images (i.e. classic Docker images).

LinuxKit packages are dedicated OCI images built by the LinuxKit community and providing basics linux functionalities like sysctl configuration management, dhcpd, volume mount, etc.

For LDAP, we’ll use an OpenLDAP alpine based image available on the dockerhub: objectiflibre/slapd. See the documentation on the repository page for usage information.

The yaml file contains the following sections:

  • kernel: contains information about the kernel to use
  • init: contains a list of containers used at the image init
  • onboot: contains a list of containers run at boot (in the order provided in the yaml file).
  • services: list of long time running containers
  • onshutdown: list of containers run at shutdown(in the order provided in the yaml file).
  • files: list of files to be imported from localhost during the image build
  • trust: list of organisations or images for which Docker Content trust should be enforced.

The file structure is quite simple and logic once you have the image design in mind. Therefore, container parameters are mostly what the Docker plateform provides, like command, binds, network, environment variables, etc.

The final yaml for the LDAP appliance looks like this:

kernel:
  image: linuxkit/kernel:4.9.75
  cmdline: "console=ttyS0"
init:
  - linuxkit/init:5a577d070817b4f17821657823082651baafd4ed
  - linuxkit/runc:abc3f292653e64a2fd488e9675ace19a55ec7023
  - linuxkit/containerd:e58a382c33bb509ba3e0e8170dfaa5a100504c5b
  - linuxkit/ca-certificates:de21b84d9b055ad9dcecc57965b654a7a24ef8e0
onboot:
  - name: sysctl
    image: linuxkit/sysctl:4c1ef93bb5eb1a877318db4b2daa6768ed002e21
  - name: dhcpcd
    image: linuxkit/dhcpcd:0d59a6cc03412289ef4313f2491ec666c1715cc9
    command: ["/sbin/dhcpcd", "--nobackground", "-f", "/dhcpcd.conf", "-1"]
  - name: format
    image: linuxkit/format:e945016ec780a788a71dcddc81497d54d3b14bc7
  - name: mount
    image: linuxkit/mount:b346ec277b7074e5c9986128a879c10a1d18742b
    command: ["/usr/bin/mountie", "/var/lib/openldap"]
  - name: metadata
    image: linuxkit/metadata:2af15c9f4b0e73515c219b7cc14e6e65e1d4fd6d
    command: ["/usr/bin/metadata", "openstack"]
services:
  - name: rngd
    image: linuxkit/rngd:94e01a4b16fadb053455cdc2269c4eb0b39199cd
  - name: sshd
    image: linuxkit/sshd:ac5e8364e2e9aa8717a3295c51eb60b8c57373d5
    binds:
     - /etc/resolv.conf:/etc/resolv.conf
     - /run:/run
     - /tmp:/tmp
     - /etc:/hostroot/etc
     - /usr/bin/ctr:/usr/bin/ctr
     - /usr/bin/runc:/usr/bin/runc
     - /containers:/containers
     - /var/log:/var/log
     - /var/config:/var/config
     - /dev:/dev
     - /sys:/sys
     - /var/config/ssh/authorized_keys:/root/.ssh/authorized_keys
  - name: slapd
    image: objectiflibre/demo-linuxkit-slapd
    env:
     - SUFFIX_FILE=/var/config/ldap_suffix
     - ORGANISATION_NAME_FILE=/var/config/ldap_organisation
     - ROOT_PW_FILE=/var/config/ldap_rootpass
    binds:
     - /var/config:/var/config
     - /var/lib/openldap:/var/lib/openldap/openldap-data
     - /etc/resolv.conf:/etc/resolv.conf
    capabilities:
     - CAP_NET_BIND_SERVICE
     - CAP_CHOWN
     - CAP_SETUID
     - CAP_SETGID
     - CAP_DAC_OVERRIDE
trust:
  org:
    - linuxkit
    - library

For a detail reference of the yaml format, see the official documentation on Github.

The most difficult part today is to understand how core packages like systctl, dhcpd or metadata work: packages come with no documentation and digging in package sources (and even application code sources) is the only way. You may use the default linuxkit.yml or the example files as a start for your own image.

You’ll notice that the yaml contains no reference to a final image format or destination cloud, providing a true portable image definition.

Note

All materials used in this post is available on Objectiflibre’s Github

Build, local run, ship, run a LinuxKit image

As said in the introduction, LinuxKit follow the moto “build, ship, run”.

The toolkit supports several image format (qcow2, raw, iso, etc) and several target plateforms (Azure, Openstack, etc). See command line help to adapt to command to your platform.

Build

To build an image, you just need to specify a name, an image format and a yaml file. For example:

linuxkit build -format qcow2-bios -name linuxkit-ldap ldap.yml

You have now a minimal linux system with LDAP installed of only 80 Mo.

Local run

You can test your image on a compatible local hypervisor, with qemu for example:

linuxkit run qemu --mem 1024 --publish 8389:389 linuxkit-ldap.qcow2

Here the LDAP port 389 of the virtual machine is binded to the 8389 port of the local machine.

Then you can test your LDAP instance with ldap-client and the following parameters:

  • url: localhost:8389
  • admin dn: cn=admin,dc=example,dc=org
  • admin passwd: password

Ship

If you want to use your image on a cloud, you can upload yourself the image to your cloud image repository or use the LinuxKit command.

I’ll use Objectif Libre’s internal Openstack Cloud in this example but you can easily adapt it to use another cloud.

$ linuxkit push openstack -img-name=linuxkit-ldap -authurl=$OS_AUTH_URL \
  -domain=$OS_USER_DOMAIN -username=$OS_USERNAME -project=$OS_PROJECT_NAME \
  -password=$OS_PASSWORD linuxkit-ldap.qcow2

Run

Once the image is uploaded to your cloud, you can use it as any other image with your cloud tools or use the LinuxKit command to spawn an instance by providing the necessary information (instance flavor, network, security group, ssh keys, etc)

$ linuxkit run openstack -authurl=$OS_AUTH_URL -domain=$OS_USER_DOMAIN \
  -username=$OS_USERNAME -project=$OS_PROJECT_NAME -password=$OS_PASSWORD \
  -flavor=m1.small -instancename=demo-ldap -keyname=mcottret -sec-groups="openbar" \
  -network=d5c237e1-0376-4d68-87b6-e208f6dc210d linuxkit-ldap

You have now an fully operationnal LDAP server running in your cloud.

To connect to the ldap instance, get the instance public ip (add a floating ip if necessary) and use the default parameter with ldap-client:

  • url: LDAP_IP:389
  • admin dn: cn=admin,dc=example,dc=org
  • admin passwd: password

Since the image was built with an sshd service, you can check if the instance is up and running:

$ ssh root@LDAP_IP
$ ctr c ls # list containers
$ ctr t ls # list tasks

If any problem, logs can be found in /var/log

And if we want to customize the basic ldap configuration ?

Now that we have a usable image, how can we configure it. LinuxKit builds immutable image, so you can not just ssh to the instance and make modifications (Of course, sshd is running IN a container).

Fortunately, our slapd OCI image can be configured using environment variables. We also specify in the yaml file the package metadata in the boot section that can gather user-init metadata provided by Openstack when creating an instance.

Thus, you have multiple ways to configure your LDAP instance:

  • modify the ldap.yml to embed your env variables, rebuild and push the new image.
  • modifiy the ldap.yml to load local files in your image (using the files key), rebuild and push the new image.
  • provide a json metadata file at runtime with the following schema:
    {
      "ldap_suffix": {
        "content": "dc=ol,dc=lab",
        "perm": "0644"
      },
      "ldap_rootpass": {
        "content": "totolitoto",
        "perm": "0644"
      },
      "ldap_organisation": {
        "content": "OL Lab",
        "perm": "0644"
      }
    }
    

What about LinuxKit packages ?

LinuxKit can use packages to assemble a system image. As said earlier, packages are OCI images with extra-metadata, natively multi-arch and signed.

Most packages are built and maintained by the LinuxKit community but anyone can create one.

Let’s see how we can build an ldap packages to use instead the slapd image.

Package metadata

First, we need a build.yml containing metadata:

image: ldap
org: oldemo
arches:
  - amd64
gitrepo: https://github.com/ObjectifLibre/linuxkit-ldap-demo.git
network: yes
disable-content-trust: yes
config:
  capabilities:
   - CAP_NET_BIND_SERVICE
   - CAP_CHOWN
   - CAP_SETUID
   - CAP_SETGID
   - CAP_DAC_OVERRIDE

The only mandatory key is the image containing the name of the image.

The org indicates the namespace for the hub (the image will be pushed to docker.io/ORG/IMAGE). The default value is linuxkit and is used only for core packages maintained by the LinuxKit project.

arches specify the list of images the tool may find to build the multi-arch manifest.

The network key indicates the need for internet connection during the build.

For demo purpose, content trust (ie. image signing) is disable.

Finally, runtime configuration are provided in the config key. Those values will be set to the label org.mobyproject.config in the image and used by Linuxkit as default container runtime configuration. This way, package can be used without further notice about a necessary set of capabilities, mountpoints, etc.

For more information and update, see Official documenation

Package OCI image

then, we need a Dockerfile to specify how to build the OCI image:

FROM alpine AS mirror

RUN mkdir -p /out/etc/apk && cp -r /etc/apk/* /out/etc/apk/
RUN apk add --no-cache --initdb -p /out \
    alpine-baselayout \
    busybox \
    bash \
    ca-certificates \
    musl \
    tini \
    openldap \
    openldap-back-mdb \
    openldap-clients \
    && true
# we really do not want a rogue inittab here
RUN rm -rf /out/etc/inittab
COPY scripts/* /out/etc/openldap/
COPY slapd.sh /out/

FROM scratch
ENTRYPOINT ["/sbin/tini","-s","-v","--"]
WORKDIR /
COPY --from=mirror /out/ /
CMD ["/slapd.sh"]

The full package source is available on ObjectifLibre github

Building the package

To build the package, just use the following command in the source folder:

$ linuxkit pkg build .

By default, the image’s tag is the HEAD of the current git tree.

To use it, you just need to change the image reference for the ldap service in the yaml file to use the freshly built image.

You can also share your package by pushing the image to the hub with docker or the LinuxKit command:

$ linuxkit pkg push .

Note

On linux, be sure to not use any docker-credentials-helper since the tool will look at the base64 encoded user/password in ~/.docker/config.json in order to push the image on the hub

Note

For multi-arch, you need to build and push the image in sequence on each plateform. The tool will update the multi-arch manifest at each pass.

Conclusion

LinuxKit is an attempt to provide the same lean development and run process for hosts (physical or virtual) as the one proposed by Docker for containers.

The LinuxKit team have done a tremendous piece of work in a short time (the Project has not even one year old) and the project is already used in public products from Docker or IBM.

The project suffers from its youth: api and core packages are definitively not stable and the lack of a good documentation can be challenging for the new comer.

Nevertheless, the approach proposed by LinuxKit clearly fits the DevOps way and you should keep an eye on it.