Logo OpenSCAP

This blog post presents the setup of a security rules checkup system on OpenStack.

Intended audience : System Administrators with basic knowledge on Puppet and OpenStack.

By Adrien Beudin, System Administrator @Objectif Libre

Compliance enforcement on an OpenStack cloud with Puppet, Foreman and OpenSCAP

Introduction

How can you be sure that every instance of your cloud is properly configured to respect best practices in order to avoid security threats? This is a legitimate question, especially in self-provisioned environments where users can spawn instances at will.

This is where compliance management comes in, to create a security repository and enforce it across your cloud. And unless you have an army of gnomes that can check every instance of your cloud, you’ll probably want to automate this process.

OpenSCAP

SCAP (Secure Content Automation Protocol) is an open standard designed enable automated compliance management, including vulnerability management, measurement, and policy compliance evaluation.

OpenSCAP is an open-source implementation of SCAP, approved by National Institute of Standards and Technology (NIST).

Why Puppet?

Puppet is a client-server configuration management tool. Puppet clients automatically check (every 30 minutes by default) their compliance with the server repository.

This repository (a Puppet manifest) is centralized on a server that compiles the client configuration to produce a Puppet catalog. A catalog is basically a document that describes the desired state for each managed resource. The clients will apply the catalog configuration and send a report to the server.

Why use Puppet for compliance enforcement? Well, its agent-server architecture with a regular check fits well our need. This way we can be sure that all clients are always in the desired state.

To have an overview of Puppet and detailed reports we will use Foreman. It is a provisioning, monitoring and configuration tool, a “complete lifecycle management tool”. It integrates well with several configuration management tools including Puppet.

Last but not least, there is an OpenSCAP plugin for Foreman. It can be used to configure OpenSCAP client on hosts and collect reports. Note that this plugin does not support self-remediation, so it is necessary to write Puppet code for each
selected rule.

Deployment

Infrastructure

Puppet and Foreman must scale since as the number of instances grows, Puppet servers will have more requests to handle.

One crucial element is the Puppet certificate authority. Puppet certificates are signed by certificate authority that stores files in a simple directory. To overcome this limitation, several solutions exist, as discussed during the FOSDEM 2018.

The simplest solution est to set up a Foreman server as a certificate authority and to have multiple Puppet master servers.

Infrastructure example:

Policy setup

Several steps are required to set up a compliance policy. First, we define rules that are interesting to check.

To do so, we use OpenSCAP tools and verify which profile fits our needs. OpenSCAP has two tools:

  • SCAP Security Guide, an implementation of the security guides recommended by PCI DSS, STIG, and USGCB
  • SCAP Workbench, a graphic interface for OpenSCAP

Once we selected the profile we can modify it to:

  • disable some rules
  • change threshold values

This configuration will be stored in a Tailoring file.

Next, we need to create the compliance policy that will be deployed on instances.
Everything is done via the Foreman interface where we upload the following files:

  • SCAP Content (profile)
  • Tailoring

We then define on which instances we want to attach the policy to. Two options here:

  • Do it manually using the actions of an instance in Foreman
  • Use a group in Foreman

The simplest is to use groups as it is possible to automatically assign hosts to a group using the foreman_default_hostgroup plugin.

The only drawback with this plugin is that it does not support the AND operator to assign the host to the right group.

I made a pull request to fix this issue, but it is not yet merged at the time of writing this article.

So, to assign an instance to a group, we first need to create the group in Foreman and configure the plugin.

Here is an example of a configuration:

---
:default_hostgroup:
  :facts_map:
    "Redhat7":
      "os.name": "RedHat"
      "os.release.major": "7"
    "Centos7":
      "os.name": "Centos"
      "os.release.major": "7"

With this configuration, we simply assign all Redhat / Centos 7 machines to the Redhat7 group.

Once we created the policy in Foreman, we need to use the Puppet module on all instances. This module will install and configure a SCAP client on instances.

The client execution is not managed by the Puppet run, it is a cron task (the periodicity is defined is the previously created policy).

When the client runs, it fetch the SCAP files (content & tailoring) from the Foreman server and runs the compliance checks. After the checks, the client will upload a report to Foreman.

Puppet Modules

As stated above, Foreman plugins don’t support self-remediation, so we will need either an existing plugin or write a custom one.

Problems in a cloud environment

So we saw how to use OpenSCAP with Puppet and Foreman, but there are some problems when used in a cloud environment, OpenStack in this case.

Name conflicts

The main problem is with hostname unicity. Puppet use by default the hostname of the instance to generate the certificate. As Puppet and Foreman don’t support OpenStack tenants isolation, there is a risk of conflicts.

The best way to avoid this is to use a random name. To keep things consistent, I can only recommend to use the instance ID.

But this is not enough, because if a user rebuild an instance, the ID does not change and the automatic certificate signing will not work as Puppet already has one with the same name.

We have two options in this case:

  • Enable in Puppet an option to force automatic regeneration of a certificate. This is not safe.
  • Add a random number after the OpenStack instance ID.

Puppet configuration sample:

echo "certname = `openssl rand -hex 7`-`ls /var/lib/cloud/instances/`.novalocal" >> /etc/puppet/puppet.conf

Purge

Cloud means self-service (creation and deletion of instances by users) and auto-scaling. Both of this features will cause useless entries in Foreman that will stay after the instances are deleted.

So we need to set up a purge for out of sync instances, with a simple cron task and a little python script. Here is an example for Foreman:

#!/usr/bin/python
# -*- coding: utf-8 -*-

import requests
from requests_oauthlib import OAuth1
import json
import yaml


config = yaml.load(file('/etc/foreman/settings.yaml', 'r'))

url_base = "https://localhost/api/"
client_key = config[':oauth_consumer_key']
client_secret = config[':oauth_consumer_secret']


search = 'last_report+<=+"1+days+ago"+and+status.enabled+%3D+true'

auth = OAuth1(client_key=client_key, client_secret=client_secret)

try:
  url = url_base + "hosts?&search={}".format(search)
  response = requests.get(url=url, auth=auth, verify=False)

except Exception as e:
  print "Foreman error"
  raise Exception(e)

nodes = response.json()

for node in nodes['results']:
  name = node['certname']
  url = url_base + "hosts/{}".format(name)
  try:
    response =  requests.delete(url=url, auth=auth, verify=False)
    print response.json()
  except Exception as e:
    print "Foreman error"
    raise Exception(e)

Security Guide

It is crucial to test the configuration of the Tailoring file because some SCAP rules are not compatible with a cloud usage. If activated, they could cause some issues:

  • Firewall activation with (too) strict rules
  • SSH configuration
  • etc…

Some other rules are not applicable to cloud environments:

  • Partitions isolation
  • UEFI parameters
  • etc…

Network access

One last problem due to OpenStack is the tenant’s segregation. We need to find a way for all instances to be able to reach Puppet and Foreman. Possible solutions are:

  • Create a relay instance that will forward Puppet requests
  • Use a transparent sNAT solution (with OpenStack software-defined networks, Cisco ACI for example).
  • Automatically assign a floating IP or a provider network for all instances, but say goodbye to the generic aspect of OpenStack…

Conclusion

Having a solution to enforce instances compliance is an interesting idea, even though its setup is more complicated than on a legacy non-cloud infrastructure.