Containers made the world easier to live in: no more OS-specific vs language-specific packaging systems dependency hell, easy portability across systems and platforms, isolation between services, and, most importantly, easy image sharing through registries. But what is the cost of this convenience?

Target audience: k8s / helm lovers and users, cloud-native aficionados

Written by Quentin Anglade, DevSecOps @ Objectif-Libre

Convenience vs Security: Container images, helm charts and sandwiches

Trust, I remember trust

With the rise of containers came a shift of the responsibility of software distribution maintenance and support. From distro maintainers or system administrators who used to deploy software either via their favourite distro’s package manager or doing it themselves, we went to docker itself (for official images) and strangers on the docker hub. Of course, anyone is free to use their Dockerfiles and maintain images themselves, but let’s face it: we, as humans, like convenience. Images on the docker hub have a lot of pulls, and that number by itself is already a trust factor for most users. An image downloaded 5 million times and updated 6 days ago is enough to be implicitly trusted. To twist the knife in the wound, there is no widespread mechanism to verify that an image is actually the result of the GitHub repo it is linked to (automated builds and public cryptography can help in this regard, though). This issue is not specific to the docker hub (a lot of language-specific packaging systems have the same issue), nor it is purely a technical issue. It’s just that users will too often choose convenience over security, and security is often a bit less (and sometimes a lot less) convenient.

A complex sandwich

Containers and container images are just one layer in a complex sandwich. Just like containers have brought their fair share of convenience at the application level, new tools have emerged on top, using the tools from the underlying layer, trying to solve issues from the layer on top. We can repeat the process as many layers there is in the sandwich.

The issue with sandwiches is that if only one slice of cheese went bad, it will taste bad and you could get sick even if the rest of the sandwich is fine. At least from a security point of view.

For this article the layer above containers is Kubernetes. It’s a great project that the community has and is still strongly embracing. But Kubernetes by itself is missing a package manager. Let’s add another layer to our sandwich: just like you’d prefer running apt install package instead of cloning the sources and installing a software yourself, you’d prefer using a helm chart for your Kubernetes needs. And you would be right. But who’s maintaining that chart, that chart’s container images and that chart’s container images’ software? Probably different entities. This segmentation causes a foggy trust and security environment.

Meet helm-trivy: trust, but verify

To help helm charts developers and users to quickly estimate the security of a given chart’s images, we’ve developed a helm plugin called helm-trivy. This plugin is largely inspired by Snyk.io’s helm plugin, helm-snyk, but it uses aquasec’s free trivy instead. Snyk and trivy are useful tools that can scan container images looking for known vulnerabilities. They can be used as standalone, in a CI pipeline, or here with a helm plugin.

The goal of helm-trivy is to quickly scan a given chart’s images, whether it is a public or a local one, in a CI or on your computer. To be honest, I first wanted to fork the snyk-helm repository and simply change the docker image it uses for scans, but my NodeJs love and skills pushed me to implement a new plugin in Golang instead. You can install it using helm’s great plugin system:

helm plugin install https://github.com/ObjectifLibre/helm-trivy

You can then scan any public or local chart:

# Scan a public chart
helm trivy stable/mysql

#Scan a local chart
helm trivy path/to/your/chart

# Output only high and critical vulnerabilities
helm trivy --trivyargs '--severity HIGH,CRITICAL' <chart>

# Scan a specific version and output json
helm trivy --json --version 1.2 <chart>

Feel free to contribute to helm-trivy on GitHub !