cat ~/posts/security-and-containers.md

Security and Containers

2021-06-01Containers, Docker, Kubernetes, Security, Vulnerability Management

Introduction

A container exists through a form of operating system virtualization, IBM describes this process, “containers take advantage of a form of operating system (OS) virtualization in which features of the OS are leveraged to both isolate processes and control the amount of CPU, memory, and disk that those processes have access to”. (Docker, What is a Container)

Containers are described by Docker to “package up code and all its dependencies so the application runs quickly and reliably from one computing environment to another”. (Docker, What is a Container)

In contrast to virtual machines, containers are light weight. Containers are minimal, their purpose are to only contain what is needed to run a piece of software. A container will share the OS kernel while a virtual machine runs a full OS.

VM vs Container architecture

Figure 1: the backbones of a virtual machine vs a container. (Sagi, 2019)

The standout difference in the backbones of a container and a virtual machine is that a virtual machine will use a hypervisor to virtualize the underlying hardware of the host or node, while a container will utilize a container engine downloaded on a operating system to virtualize software.

In this report I will focus on discovering container security risks and how to mitigate them, in specific I will discuss network traffic risks, access to files risks, kernel level risks, and lastly, I will conclude with discussing patching service applications and container vulnerability scanning.

Kernel Level Threats

Running container processes as root creates an attack vector for threat actors. By running a container process as root, the administrator “provides (threat actors) access to the underlying kernel” and “a direct path to attack other containers and the Docker engine”. (Securosis, 2018)

With root access, exploiting the namespace and cgroup feature used by the Linux kernel is trivial. Docker containers ran as root can be abused to escalate privileges using the mount namespace. “Mount namespaces provide isolation of the list of mount points seen by the processes in each namespace instance”. (SysTutorials, namespaces)

Docker containers by default have a cgroup policy that blocks read and write of block devices.

Saarinen explains how using a container with root privileges can be used to write and read in the node files. Presenting the container with root privileges creates a attack vector to threat actors, once an attacker has access to a container, a block device can be created within the container and given the name “sda”, which is the default block for the file union file system between the container and the node, given read, write, and execute privileges, and lastly, using a user that exists on both the container and the node the threat actor can write to the node files. (Saarinen, 2020)

Container root exploit

Figure 2: exploiting the host of a container ran with root privileges. (Saarinen, 2020)

I have discussed above that vulnerable containers with unnecessary privileges can be used to exploit the host machine through kernel, it is important to also note that a vulnerable kernel can also be exploited to gain access to containers.

The vulnerability, CVE-2016-5195, is a Linux kernel vulnerability that was disclosed on 11/10/2016. It is described to allow local users to gain privileges “by leveraging incorrect handling of a copy-on-write (COW) feature to write to a read-only memory mapping, as exploited in the wild in October 2016, aka ‘Dirty COW’”. (NVD, CVE-2016-5195 Detail)

Exploiting the kernel to gain root privileges on a host that runs containers will pose an extreme risk, the containers will be very easily compromised.

Mitigating the Threat

Kernel security systems and Linux functions can be used in conjunction to mitigate bypasses to the kernel feature controls. Linux functions limit the permissions assigned to each process so that container processes are not assigned full root privileges. Kernel security systems such as the Linux security module provides “a framework that allows the Linux kernel to support different security models” (Saarinen, 2020) such as SELinux. SELinux can be used to provide access control policies to allow or deny access to files and directories using labels.

The most important threat to mitigate from the discussed vulnerabilities is the Docker container running with root privileges. Containers should not be run as root or given a user namespace of root, Saarinen also mentions that the vulnerability the container experiences can be mitigated by running the Docker container using the “–cap-drop=MKNOD” parameter. (Saarinen, 2020)

To mitigate kernel vulnerabilities, the host OS should be hardened, and to continuously keep the OS patched and up to date.

Unrestricted Network Traffic

To understand the threat in networking between containers, an understanding of the fundamentals in networking in container provider software is required. In Docker, a Docker network is described to “define a communication trust zone where communication is unrestricted between containers in that network”. (IBM, Docker containers and network interfaces)

By default, networking between containers is limited to the default network bridge network interface, as a result, all containers connected to the bridged network can communicate with each other.

A compromised container connected to a bridged network interface allows a threat actor to conduct further attacks. Once 1 container on a bridged network interface is compromised, the threat actor will have the ability to listen to traffic on the wire and to send packets to other containers on the bridged network interface. As a result, bridged network interfaces by default enable an attack vector for threat actors. This unrestricted network traffic mode in the bridged network interfaces poses the “risk of unintended and unwanted disclosure of information to other containers”. (Lancini, Docker Threat Model)

Man-in-the-Middle Attacks

A man in the middle attack is an attack where a threat actor eavesdrops on a network conversation they are not intended to listen to. The attack is used to collect information regarding what type of packets are being sent on the wire, statistical information about the packets, and the contents of the packets.

In Kubernetes, pods are granted the “net_raw” capability to the pod’s unprivileged process. To check capabilities in a pod, the command “pscap -a” is to be run in the pod.

The “net_raw” capability is a default capability granted to Kubernetes pods to allow ICMP traffic between pods. This capability poses a risk; the capability does not only allow ICMP traffic, but it also allows threat actors to craft raw packets such as ARP and DNS.

ARP Spoofing and ARP Cache Poisoning

ARP spoofing is a man in the middle initiating technique; ARP spoofing describes the process of threat actors crafting fake ARP packets.

ARP cache poisoning is an attack where a threat actor exploits the functions of gratuitous ARP packets to trick hosts in an ongoing conversation that the threat actor pod or container is part of the ongoing conversation.

In our situation, the goal of a threat actor attempting to eavesdrops on a bridged network interface can only ARP spoof one of the pods or containers, unlike how physical machines can spoof not just endpoints, but also gateways or other networking devices.

ARP spoofing in pods or containers will allow the threat actor will be able to intercept all traffic between containers on the same bridged network interface.

Mitigating the Threat

To mitigate unrestricted network traffic between Docker containers on the same bridged network interface, the parameter “–icc=false” can be run with the Docker daemon or the Docker daemon configuration file can be edited to add the line “‘icc’:false”. As a result of applying the following, “Inter-container communication is disabled on the default network bridge”. (Man7, Capabilities)

To mitigate unrestricted network traffic between Kubernetes pods on the same bridged network interface, the administrator needs to append security settings for pods to include “securityContext”. The securityContext field in a pod specification file is a PodSecurityContext object. The securityContext has a capability object option that can be used to drop a default capability a pod runs with, the capability in question, net_raw is to be dropped.

Unrestricted Access to Files

To access node files from Docker containers, an administrator has 2 options, to use bind mounts or to use Docker volumes. When bind mounts are used, “a file or directory on the host machine is mounted into a container”. When volumes are used, “a new directory is created within Docker’s storage directory on the host machine, and Docker manages that directory’s contents”. (Docker Documentation, 2021)

A compromised container with bind mounts to critical parts of a node OS file structure, such as “/etc/” or “/proc/” will expose critical directories and files to attackers.

Valid use cases such as mounting the “/var/log” directory is used by system administrators to write to log files related to container processes directly to the node.

Mitigating the Threat

If an administrator has no direct reason or cause to use bind mounts, volumes should be used. Volumes mounted to containers will only write to blocks under the Docker directory and will eliminate the risk behind reading and writing in critical directories by using bind mounts.

Inconsistent Updates and Patching

Outdated and unpatched container service application such as Docker or Kubernetes pose a risk not just to the containers, but to the entire node. Over time, developers for container service applications will make changes to the applications the administrators use. For example, Docker Engine’s developers post “release notes” write ups of changes made to the Docker Engine application to not only document the changes but to also bring attention to issues, mitigated vulnerabilities, and improvements the Docker Engine team has made to the application for administrators.

Container images such as Docker images also pose a threat to the node. A container image may be vulnerable to already disclosed and known attacks. It is critical for administrators to develop a plan to not only harden the images as discussed previously but to also scan the images for known vulnerabilities. A container vulnerability scanner will be optimal to detect and mitigate known vulnerabilities in container images.

An Example / Security Patches for Docker Engine

A very critical patch was recently released for the vulnerability CVE-2019-5736 for Docker Engine. This vulnerability is described by the release notes to “allow specially-crafted containers to gain administrative privileges on the host”. (Docker, Docker Engine 18.09 release notes)

NVD gave the vulnerability an 8.6 CVSS score. NVD describes the attack to not require any privileges to replicate the attack, to have a total loss of confidentiality, to have a total loss of integrity, and to have a total loss of availability because of the vulnerability. (NVD, Common Vulnerability Scoring System Calculator CVE-2019-5736)

Mitigating the Threat

An administrator should deem that it is vital for the security of the node to immediately apply the patch made available by the Docker Engine team because of researching the vulnerability, CVE-2019-5736.

Furthermore, an administrator should develop a plan to periodically check for updates and to apply them or to develop automation into not just updating the service application for containers, but also to verify the integrity of the downloaded patches and to verify that the latest version of the service application is running.

Container Image Vulnerability Scanning

There are many container image vulnerabilities scanning open-source tools not only for docker but also for Kubernetes. I will discuss in specific Dagda, a Docker image vulnerability scanner.

Dagda is described by Rubio to be “a tool to perform static analysis of known vulnerabilities, trojans, viruses, malware & other malicious threats in docker images/containers and to monitor the docker daemon and running docker containers for detecting anomalous activities” (Rubio, 2020) Dagda imports vulnerability details from disclosing vulnerability repositories such as CVE and RHSA into a MongoDB collection.

Dagda then “retrieves information about the software installed into your docker image, such as the OS packages and the dependencies of the programming languages” (Rubio, 2020).

Dagda will then verify that the image or container is free of vulnerabilities and will lastly use ClamAV to scan the container or the image to detect malware. The collected information regarding this analysis will then be archived in another MongoDB collection.

Conclusion

The importance to security of not only containers, but also the node the containers run on can not be overstated. It is vital for container operation in production to not be vulnerable to already disclosed and published threats.

To sum up the points I have made and talked about in this report, I have created the following graphic. For every threat, I described the technique or concept of how the threat can be used to exploit containers.

Container threat model

Figure 4: container threat model

References