By Luka Peschke, DevOps at Objectif Libre.Portrait-LPE

 

More than 10 000 jobs are launched by OpenStack’s Continuous Integration system every day (source openstack.org).

 

Several projects have been developed as part of the OpenStack project infrastructure to manage these jobs.

As the first approach can be a bit confusing, we felt the need to explain the main components of this system in this post.

 

You can consider this post as a tutorial about adding a job to the CI system.

 

If, like Jon Snow, you know nothing about the whole system (and you don’t want to spend hours reading documentation) : keep calm and go on reading.

 

May you want to go deeper, or even want to add a job yourself, we recommend you to read the official infra documentation along with the Developer’s Guide.

Content

1. Gerrit

2. Zuul

3. Jenkins Job Builder

4. Add a job

4.1. cloudkitty.yaml
4.2. projects.yaml
4.3. layout.yaml

 

1. Gerrit

When a developer wants to apply a patch/change on one of OpenStack projects, he will commit it with git-review on the Gerrit server hosted at http://review.openstack.org.

For each project, there are several job pipelines. They are defined in the project-config repo’s layout.yaml file.

 

The most important ones are the check and gate pipelines. When a patch/change is submitted to Gerrit, the check pipeline’s jobs are executed by Jenkins. These jobs can be voting or not. Non voting jobs have the -nv suffix. Depending on the failure -or not- of the check pipeline’s voting jobs, Jenkins will add a +1 or -1 vote with the verified mention to the patch/change. Only non-interactive members of Gerrit (Jenkins and others) can add a vote with the verified mention. Reviewers can add a -1, +1 or 0 vote.

 

Core reviewers can also add -2 and +2 votes. A patch/change has to get two positive reviews from core reviewers to be accepted. When the patch/change has gotten two +2 reviews, a core reviewer can give it the approved mention. Once this mention has been given, the gate pipeline’s jobs are executed. If every voting job of this pipeline succeeds, the patch/change is merged into the project’s repo.

 

There are other pipelines too, like the post pipeline, which is executed as soon as a patch/change is applied (its jobs can for example update the project’s documentation); or the experimental pipeline, which is meant to test new jobs. To trigger this pipeline, you need to leave a “check experimental” review comment.

 

2. Zuul

Zuul is a project developed for the needs of the OpenStack CI. It reads Gerrit’s event stream, launches a project’s jobs depending on this stream, and merges the patches/changes which pass the gate tests. Depending on the state of a patch/change, Zuul will launch the jobs of a specific pipeline.

 

Zuul is configured via the layout.yaml file. The triggers of each pipeline are defined in the trigger: gerrit: section, and the jobs to launch are given in the projects: section of the file.

 

Project templates are also defined in layout.yaml . Project templates allow you to define several jobs in several pipelines, and to apply these jobs to various projects.

 

For example, the merge-check template is defined like this:

- name: merge-check
  merge-check:
    - noop

 

If you apply the merge-check template to a project, the noop job will be called in the merge-check pipeline. This pipeline’s description is Each time a change merges, this
pipeline verifies that all open changes on the same project are still mergeable. The noop job is internal to Zuul and always returns SUCCESS. If there isn’t at least one successful job,
a patch/change can’t be applied, so that’s the point of noop.

 

Here is CloudKitty’s definition in layout.yaml:

- name: openstack/cloudkitty
    template:
      - name: merge-check
      - name: python-jobs
      - name: python34-jobs
      - name: python35-jobs-nv
      - name: openstack-server-publish-jobs
      - name: openstack-server-release-jobs
    post:
      - cloudkitty-coverage-ubuntu-trusty
      - cloudkitty-coverage-ubuntu-xenial

 

3. Jenkins Job Builder

Jenkins jobs are configured in .xml files. To ease and to automate the creation of these, the Jenkins Job Builder project was created. It converts .yml and .yaml files into .xml files. The point of these yaml files is to allow the creation of job templates and macros for Jenkins jobs.

 

If you look at Cloudkitty’s definition in layout.yaml, you can see that two coverage jobs are called in the post pipeline. These jobs are defined in python-jobs.yaml.
In this file, you have the {name}-coverage-{node} job template:

- job-template:
    name: '{name}-coverage-{node}' # <1>

    wrappers: # <2>
      - build-timeout:
          timeout: 40
      - timestamps

    builders: # <3>
      - print-template-name:
          template-name: "{template-name}"
      - zuul-git-prep-upper-constraints
      - install-distro-packages
      - revoke-sudo
      - coverage:
          env: cover

    publishers: # <4>
      - scp:
          site: 'static.openstack.org'
          files:
        - target: 'logs/$LOG_PATH'
          source: 'cover/**'
          keep-hierarchy: true
          copy-after-failure: true
        - test-results
        - console-log

    node: '{node}' # <5>

<1> The template’s name (obviously)

<2> Wrappers allow you to modify the way a job is executed. You can compare them to python decorators.

<3> Builders are the equivalent of a function. They are defined in the macros.yaml file.

<4> Publishers are executed by Jenkins after a build. In this case, their role is to publish the job’s results.

<5> The node on which you want the job to be executed.

 

A few lines later, you can find the coverage-jobs job group:

- job-group:
    name: coverage-jobs
    node:
      - ubuntu-trusty
      - ubuntu-xenial
    jobs:
      - '{name}-coverage-{node}'	

In this case, JJB will build a '{name}-coverage-ubuntu-trusty' and a '{name}-coverage-ubuntu-xenial' job.

 

For each OpenStack project, you also need an entry in projects.yaml. Jobs belonging to each project are defined in this file.

layout.yaml is the file used by Zuul, and projects.yaml is the file used by JJB. Zuul’s job is to choose which jobs to launch, JJB’s is to build them.

 

Here is CloudKitty’s definition in projects.yaml:

- project:
    name: cloudkitty
    tarball-site: tarballs.openstack.org
    doc-publisher-site: docs.openstack.org

    jobs:
      - coverage-jobs
      - python-jobs
      - openstack-publish-jobs
      - openstack-server-release-jobs
      - 'gate-{name}-python35{suffix}':
          suffix: '-nv'

As you can see, the coverage-jobs are declared here. When JJB will see this, it will replace {name} in the job’s name by the content of the project’s name variable. Once the jobs are declared, you can call cloudkitty-coverage-ubuntu-trusty and cloudkitty-coverage-ubuntu-xenial from the post pipeline in the openstack/cloudkitty section of layout.yaml.

 

4. Add a job

CloudKitty needs a job which will try to install the project with devstack. To do this, we will build a dsvm (DevStack Virtual Machine) job. These jobs use the devstack gate project and are launched on nodes managed by Nodepool.

4.1. cloudkitty.yaml

There isn’t a job specific to CloudKitty yet, so we will have to create the cloudkitty.yaml file to define a job template.

 

We want to have timestamps, and we are going to set a timeout to 120 minutes:

- job-template:
    name: '{pipeline}-cloudkitty-dsvm-install{job-suffix}'
    node: '{node}'

    wrappers:
      - timeout:
          timeout: 120
      - timestamps

Once we have set the wrappers, we are going to set the builders. First, we are going to call devstack-checkout, which uses Zuul cloner to clone the devstack-gate repo. It is defined in macros.yaml like this:

- builder:
    name: devstack-checkout
    builders:
      - shell: |
          #!/bin/bash -xe
          cat > clonemap.yaml << EOF
          clonemap:
            - name: openstack-infra/devstack-gate
              dest: devstack-gate
          EOF
          /usr/zuul-env/bin/zuul-cloner -m clonemap.yaml --cache-dir /opt/git \
              git://git.openstack.org \
	      openstack-infra/devstack-gate			

We did set the following builders in cloudkitty.yaml:

 

  builders:
     - devstack-checkout
     - shell: |
         #!/bin/bash -xe
         DEVSTACK_LOCAL_CONFIG="enable_plugin ceilometer git://git.openstack.org/openstack/ceilometer"
         DEVSTACK_LOCAL_CONFIG+=$'\n'"enable_plugin cloudkitty https://github.com/stackforge/cloudkitty master" # <1>

	 export DEVSTACK_LOCAL_CONFIG
         export PROJECTS="openstack/cloudkitty $PROJECTS"
	 export PROJECTS="openstack/python-cloudkittyclient $PROJECTS"
	 export PROJECTS="openstack/cloudkitty-dashboard $PROJECTS" # <2>
	 export BRANCH_OVERRIDE={branch-override}

	 if [ "$BRANCH_OVERRIDE" != "default" ] ; then  # <3>
	     export OVERRIDE_ZUUL_BRANCH=$BRANCH_OVERRIDE
	 fi

	 cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh # <4>
	 ./safe-devstack-vm-gate-wrap.sh

<1> By default, DevStack doesn’t install CloudKitty, so you have to add the project to the config. CloudKitty uses Ceilometer, so we add it too.

<2> Specifying the additional repos to clone

<3> If the user has specified a branch to use during the cloning, we add it to the config.

<4> devstack-vm-gate-wrap.sh defines all kinds of environment variables needed by DevStack’s installation script (stack.sh), and launches it.

 

At the end, we specify that we want to have the build’s logs.

publishers:
  - devstack-logs
  - console-log

4.2. projects.yaml

We add our new job to cloudkitty in projects.yaml like this:

- '{pipeline}-cloudkitty-dsvm-install{job-suffix}':
    pipeline: 'gate' # <1>
    job-suffix: '-nv' # <2>
    branch-override: default # <3>
    node: ubuntu-xenial

<1> Our job will be a gate job.

<2> Our job hasn’t been tested on the OpenStack infra yet, so it will be non-voting for now.

<3> We use the default branches.

4.3. layout.yaml

At last, we have to update layout.yaml, to have our job launched when we want it to. We are going to add it to the experimental pipeline until it is tested. This is how our final entry for Cloudkitty in layout.yaml looks like.

- name: openstack/cloudkitty
    template:
      - name: merge-check
      - name: python-jobs
      - name: python34-jobs
      - name: python35-jobs-nv
      - name: openstack-server-publish-jobs
      - name: openstack-server-release-jobs
    experimental:
      - gate-cloudkitty-dsvm-install-nv
    post:
      - cloudkitty-coverage-ubuntu-trusty
      - cloudkitty-coverage-ubuntu-xenial