Logo Clair

This blog post is a guide to use CoreOS’s clair vulnerability scanner in GitLab.

Intended audience: System administrators, DevOps familiar docker and its ecosystem. Bonus points if you have already used GitLab CI/CD.

By Quentin Anglade, professional tinkerer and security freak @ Objectif Libre

Scanning docker images with clair and gitlab

So you have a shiny CI with gitlab, building some awesome software that will ultimately run inside containers. Or you don’t, but you’re curious? In any way, why not add some CD with a docker registry, and, most importantly, scan your containers for known security vulnerabilities? Well, you could do this with gitlab enterprise edition (Gitlab EE). But you may prefer open-source software, especially when it comes to security.

Requirements

You will need an up and running gitlab, preferably on a public IP with a domain for quick and working https with Let’s encrypt. I’ll recommend doing it first on a dummy gitlab for testing purposes before doing it for real. I will use gitlabtest.objectif-libre.com as my domain. Here is an example:

sudo docker run --detach \
--hostname gitlabtest.objectif-libre.com \
--publish 443:443 --publish 80:80 --publish 2222:22 --publish 4443:4443 \
--name gitlab \
--restart always \
--volume /data/gitlab/config:/etc/gitlab \
--volume /data/gitlab/logs:/var/log/gitlab \
--volume /data/gitlab/data:/var/opt/gitlab \
gitlab/gitlab-ce:latest

This will run the official gitlab docker container, with some things to keep in mind:

  • Storage: I have chosen to store all gitlab data on /data/gitlab, so feel free to adjust the path to where you want on your server.
  • Network: ports 80 and 443 are exposed for http(s), port 2222 on the host is mapped to port 22 in the container to prevent ssh conflict. SSH is useful to clone projects without having to log in with your gitlab credentials. Port 4443 is the port where I decided to host my gitlab docker registry to store the scanned docker images.

Don’t forget to set letsencrypt['enable'] = true and external_url 'https://gitlabtest.objectif-libre.com' in /data/gitlab/config/gitlab.rb if you want automatic let’s encrypt (https) certificates.

Now, set the root password at https://gitlabtest.objectif-libre.com and log in to see if everything works. I would also create a new user to avoid using the root account. I called my new user “dev1”.

Enabling docker registry in gitlab

The first thing we’ll need is to enable the docker registry that comes with gitlab. To do so, just set registry_external_url 'https://gitlabtest.objectif-libre.com:4443' in /data/gitlab/config/gitlab.rb. You could also use another domain but you will have to add certificates in /data/gitlab/config/ssl. Check out the gitlab docs for more information.

A gitlab-ctl reconfigure from the gitlab container or a docker restart gitlab is needed to launch the registry. You should now see that the Container Registry light is green in the Gitlab admin dashboard https://gitlabtest.objectif-libre.com/admin:

Now let’s try to login to the registry with your gitlab credentials:

docker login gitlabtest.objectif-libre.com:4443

Using the registry

The gitlab docker registry is like any other docker registry, you will probably use it this way:

docker build -t myawesomeimage .
docker tag myawesomeimage gitlabtest.objectif-libre.com:4443/dev1/awesomeproject:tag
docker push gitlabtest.objectif-libre.com:4443/dev1/awesomeproject:tag

You can also push images on different levels, useful if your projects use multiple docker images:

gitlabtest.objectif-libre.com:4443/dev1/awesomeproject:tag
gitlabtest.objectif-libre.com:4443/dev1/awesomeproject/microservice1:tag
gitlabtest.objectif-libre.com:4443/dev1/awesomeproject/microservice1/image1:tag

Setting up gitlab runner

Now let’s use this registry with gitlab CI/CD to automagically build, scan and push images to the registry, but only if they don’t have any severe vulnerabilities. You need a gitlab runner to use gitlab CI. A runner runs outside gitlab and is used to run things needed for your CI pipeline. Let’s deploy one quickly:

docker run -d --name gitlab-runner --restart always --privileged \
  -v /path/on/host/runner:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
   gitlab/gitlab-runner:latest

Note that I choose to share the host’s docker socket as I’m not a fan of docker in docker, alias dind. More info on this here. Next, you need to register your runner. Get your token from the admin runners page: https://gitlabtest.objectif-libre.com/admin/runners. See gitlab docs about runners for more details.

docker exec gitlab-runner gitlab-runner register \
 --non-interactive \
 --url https://gitlabtest.objectif-libre.com \
 --registration-token TOKEN\
 --description "Docker Runner" \
 --tag-list "docker" \
 --executor docker \
 --docker-image "docker:latest" \
 --docker-volumes /var/run/docker.sock:/var/run/docker.sock \
 --docker-privileged

The docker-volumes option is used to share the docker socket from the host to the container, and since our runner will manage other containers it needs the privileged flag.

Scanning the image with clair

Clair is a tool made by CoreOS/Redhat that scans docker images and reports known vulnerabilities. We’ll see how to use it to prevent vulnerable docker containers from ever seeing the light of production environments.

Let’s create a project called broken that will just build a security-wise broken docker image, so that we will see if it contains vulnerabilities. Here is a sample Dockerfile:

FROM ubuntu:14.04

RUN apt-get update

RUN apt-get install -y wget

RUN wget https://launchpad.net/~ubuntu-security/+archive/ubuntu/ppa/+build/7531893/+files/openssl_1.0.1-4ubuntu5.31_amd64.deb && dpkg -i openssl_1.0.1-4ubuntu5.31_amd64.deb

Next, we need to setup the .gitlab-ci.yml file to build our image and scan it, using clair. Remember that clair runs as a server with a database that stores the vulnerabilities databases. Since I’m lazy enough not to want to wait for the clair database to sync all vulnerabilities, I use arminc’s daily built clair databases. This is not ideal, please use a more durable clair server instead of this in production. Here is a sample of the magic gitlab-ci.yml file:

build_image:
  image: docker:git
  script:
    - docker build --no-cache -t gitlabtest.objectif-libre.com:4443/dev1/broken:testing .

test_image:
  image: docker:git
  script:
    - docker run gitlabtest.objectif-libre.com:4443/dev1/broken:testing bash -c "echo 'This is where you can run your tests'"

scan_image:
  image: docker:git
  before_script:
    - apk update && apk add coreutils
    - docker network create scanning
    - docker run -p 5432:5432 -d --net=scanning --name db arminc/clair-db:$(date -d "yesterday" '+%Y-%m-%d') ; sleep 10
    - docker run -p 6060:6060  --net=scanning --link db:postgres -d --name clair arminc/clair-local-scan:v2.0.1 ; sleep 10
  script:
    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN gitlabtest.objectif-libre.com:4443
    - docker run --net=scanning --rm --name=scanner --link=clair:clair -v '/var/run/docker.sock:/var/run/docker.sock' objectiflibre/clair-scanner --clair="http://clair:6060" --ip="scanner" -t Medium gitlabtest.objectif-libre.com:4443/dev1/broken:testing
    - docker tag gitlabtest.objectif-libre.com:4443/dev1/broken:testing gitlabtest.objectif-libre.com:4443/dev1/broken:latest
    - docker push gitlabtest.objectif-libre.com:4443/dev1/broken:latest
  after_script:
    - docker rm -vf db clair
    - docker network rm scanning

This may seem overkill, but let me explain. First, we launch the clair database and clair server, we take the database from yesterday to be sure it has been built. Again, in production, please use a persistent clair server. We only use this because it takes a while for clair to download all vulnerabilities databases. Also, we need a docker network because our scanner container needs to communicate with the clair server, and the default docker network doesn’t allow container to container network communications. Then, we scan the image using a containerized version of arminc’s clair-scanner and push it if it’s not vulnerable.

This is just an example, in a more realistic workflow you probably want to run different pipelines depending on the branch the commit is. GitLabs’s CI/CD variables will probably help you.

You should see a list of vulnerabilities in the gitlab job console that looks like this:

Conclusion

Setting up security scans is a bit tedious, but its integration in CI/CD workflows can prevent some serious flaws without doing actively anything. Scanning docker images is like an antivirus software: it’s clearly not enough to be confident about security as a whole, but it’s still essential. Clair only tells you if there are known vulnerabilities in the software your container uses. You still need to respect security best practices.