What is Docker Content Trust and How do You Secure It?

By Brandon Niemczyk (Security Researcher)

“Can they be trusted?” One of the primary security problems one must solve in a container-based system is validating that your images are correct and came from the correct source (or maliciously manipulated). One of our security predictions for 2020 discussed how malicious container images — if trusted — could have a detrimental effect on the enterprise pipeline. We have reported on attacks that involved the abuse of container images to carry out malicious activities such as scanning for vulnerable servers and cryptocurrency mining.

To help solve this, Docker provides a feature called “Content Trust.” It allows users to deploy images to a cluster or swarm confidently and verify that they are the images you expect them to be. What the Docker Content Trust (DCT) does not do is monitor your images across the swarm for changes or anything of that nature. It is strictly a one-time check done by the Docker client, not the server.

This has implications for the usefulness of DCT as a full-on integrity monitoring system. In a previous post by my colleague Magno on cloud-native systems, he mentioned using image-signing tools such as Notary to solve the question, “can they be trusted?” DCT is an attempt at providing built-in tools for Docker clients to do just that.

This article will cover four areas:

1. How DCT works
2. How to enable DCT
3. What steps can be taken to automate trust validation in the continuous integration and continuous deployment (CI/CD) pipeline
4. What are the limitations of the system

An additional goal of this article is to provide a singular in-place tutorial on getting up and running with experimenting on DCT, especially since the existing documents appear to be spread out and not centralized.

How does Docker Content Trust (DCT) work?

At its core, Docker Content Trust is very simple. It is logic inside the Docker client that can verify images you pull or deploy from a registry server, signed on a Docker Notary server of your choosing.

The Docker Notary tool allows publishers to digitally sign their collections while users get to verify the integrity of the content they pull. Through The Update Framework (TUF), Notary users can provide trust over arbitrary collections of data and manage the operations necessary to ensure freshness of content. If you have not used a Notary server before, check out Docker’s introductory guide.

The graphic in Figure 1 shows how deploying a Docker swarm or Docker build –pull allows the client to talk to the registry server to get the required images and the Notary server to see how they were signed. If you have the correct environment variables setup, it will fail to deploy unsigned images. The signing can be done on a different machine so that private keys do not need to be stored on the Docker management node used in deployment.

Figure 1. The Docker client can communicate with the registry server and Notary server

Enabling DCT

By default, DCT is disabled. We need to do a few things to set it up so that we can sign the images we want to deploy:

1. Set up our registry
2. Set up a Notary server
3. Push an image to our registry server
4. Sign the pushed image
5. Enable DCT – Set the correct environment variables on our management host so image signatures get verified by Docker commands

Step 1: Setting up our registry server

The easiest way to set up your registry server is to run the base registry image off Docker Hub. We can do this with a single command (see below). Make sure you expose port 5000 because this is what the registry server listens on.

ubuntu@ip-{BLOCKED}-20-187:~$ docker run -d -p 5000:5000 --restart always --name registry registry:2
Unable to find image 'registry:2' locally
2: Pulling from library/registry
Digest: sha256:8be26f81ffea54106bae012c6f349df70f4d5e7e2ec01b143c46e2c03b9e551d
Status: Downloaded newer image for registry:2
5df581b6eb4186edeebb40da766e7907427005d387facdb81365df35647d952d

To validate that it is running:

ubuntu@ip-{BLOCKED}-20-187:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5df581b6eb41 registry:2 "/entrypoint.sh /etc…" 4 seconds ago Up 3 seconds 0.0.0.0:5000->5000/tcp registry

Step 2: Setting up a Notary server

In addition to a registry server for storing our images, we need a Notary server to store our image signatures. Simply running a registry image from Docker Hub requires a lot of setting up, so let’s go with the simplest way to do it: by cloning the repository from the update framework.  We can then use a simple docker-compose up to deploy with their Dockerfile.

ubuntu@ip-{BLOCKED}-20-187:~$ git clone https://github.com/theupdateframework/notary[.]git
Cloning into 'notary'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 26412 (delta 0), reused 1 (delta 0), pack-reused 26409
Receiving objects: 100% (26412/26412), 35.08 MiB | 5.32 MiB/s, done.
Resolving deltas: 100% (16038/16038), done.
ubuntu@ip-{BLOCKED}-20-187:~$ cd notary/
ubuntu@ip-{BLOCKED}-20-187:~/notary$ docker-compose up -d
WARNING: The Docker Engine you're using is running in swarm mode. Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node. To deploy your application across the swarm, use `docker stack deploy`.
….. Too much output … cut out ….>
Creating notary_mysql_1 ... Creating notary_mysql_1 ... done
Creating notary_signer_1 ... Creating notary_signer_1 ... done
Creating notary_server_1 ... Creating notary_server_1 ... done

I left the warning message about not running in swarm mode on purpose. The docker-compose.yml file provided by the update framework is not swarm-compatible for a couple of reasons:

• It uses version: “2” – This is often fixable by just updating to version “3”. Other changes may be required.
• It uses a build: command – Swarm mode does not support the build operation. You will need to build these services separately and add them to a registry server. Since you will not have content trust signatures at this point, you’ll need to ensure you are not enforcing content trust on your Docker client when deploying this service.
  

As mentioned earlier, the easiest way to set up your registry server is to run the base registry image off Docker Hub with a single command. Make sure that port 5000 is open because this is what the registry server listens on.

ubuntu@ip-{BLOCKED}-20-187:~$ docker run -d -p 5000:5000 --restart always --name registry registry:2
Unable to find image 'registry:2' locally
2: Pulling from library/registry
Digest: sha256:8be26f81ffea54106bae012c6f349df70f4d5e7e2ec01b143c46e2c03b9e551d
Status: Downloaded newer image for registry:2
5df581b6eb4186edeebb40da766e7907427005d387facdb81365df35647d952d

Let’s validate that our Notary service is now up. It should also have deployed a MySQL service that it uses:

ubuntu@ip-{BLOCKED}-20-187:~/notary$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6c72520afc2e notary_server "/usr/bin/env sh -c …" 12 minutes ago Up 12 minutes 0.0.0.0:4443->4443/tcp, 0.0.0.0:32769->8080/tcp notary_server_1
862fea9019c9 notary_signer "/usr/bin/env sh -c …" 12 minutes ago Up 12 minutes notary_signer_1
8c8a05af5224 mariadb:10.4 "docker-entrypoint.s…" 12 minutes ago Up 12 minutes 3306/tcp notary_mysql_1
5df581b6eb41 registry:2 "/entrypoint.sh /etc…" 21 minutes ago Up 21 minutes 0.0.0.0:5000->5000/tcp registry

Now, we need to push an image to our repository. We do this by tagging an image with the repository URL, then calling Docker push on that tag:

ubuntu@ip-{BLOCKED}-20-187:~/notary$ docker pull ubuntu:latest
latest: Pulling from library/ubuntu
54ee1f796a1e: Already exists f7bfea53ad12: Already exists 46d371e02073: Already exists b66c17bbf772: Already exists Digest: sha256:31dfb10d52ce76c5ca0aa19d10b3e6424b830729e32a89a7c6eee2cda2be67a5
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest
ubuntu@ip-{BLOCKED}-20-187:~/notary$ docker tag ubuntu:latest localhost:5000/ubuntu:mine
ubuntu@ip-{BLOCKED}-20-187:~/notary$ docker push localhost:5000/ubuntu:mine
The push refers to repository [localhost:5000/ubuntu]
a4399aeb9a0e: Pushed 35a91a75d24b: Pushed ad44aa179b33: Pushed 2ce3c188c38d: Pushed mine: digest: sha256:6f2fb2f9fb5582f8b587837afd6ea8f37d8d1d9e41168c90f410a6ef15fa8ce5 size: 1152

For the Docker client to know to use this server, you will need to set an environment variable pointing to it:

ubuntu@ip-{BLOCKED}-20-187:~/notary$ export DOCKER_CONTENT_TRUST_SERVER=https://localhost:4443

Now, let’s sign our image. There are three steps. First, we must add a key to Docker that we can use for signing. Next, we must add that key as a signer for the Notary repository for this image, then we need to sign it.

Adding the key:

ubuntu@ip-{BLOCKED}-20-187:~/notary$ docker trust key generate sample_signer
Generating key for sample_signer...
Enter passphrase for new sample_signer key with ID f39f731: Repeat passphrase for new sample_signer key with ID f39f731: Successfully generated and loaded private key. Corresponding public key available: /home/ubuntu/notary/sample_signer.pub

Adding the key as a signer:

ubuntu@ip-{BLOCKED}-20-187:~/notary$ docker trust signer add --key sample_signer.pub sample_signer localhost:5000/ubuntu:mine
Adding signer "sample_signer" to localhost:5000/ubuntu:mine...
Initializing signed repository for localhost:5000/ubuntu:mine...
You are about to create a new root signing key passphrase. This passphrase
will be used to protect the most sensitive key in your signing system. Please
choose a long, complex passphrase and be careful to keep the password and the
key file itself secure and backed up. It is highly recommended that you use a
password manager to generate the passphrase and keep it safe. There will be no
way to recover this key. You can find the key in your config directory.
Enter passphrase for new root key with ID 65c87b3: Repeat passphrase for new root key with ID 65c87b3: Enter passphrase for new repository key with ID 10e5763: Repeat passphrase for new repository key with ID 10e5763: Successfully initialized "localhost:5000/ubuntu:mine"
Successfully added signer: sample_signer to localhost:5000/ubuntu:mine

You can now do a docker inspect and see the signer you added, but notice that no tags have been signed yet:

ubuntu@ip-{BLOCKED}-20-187:~/notary$ docker trust inspect localhost:5000/ubuntu:mine
[ { "Name": "localhost:5000/ubuntu:mine", "SignedTags": [], "Signers": [ { "Name": "sample_signer", "Keys": [ { "ID": "f39f731f1c288b66c10d70905de6d98dfa40104741c878cb2766cddc6ed52f28" } ] } ], "AdministrativeKeys": [ { "Name": "Root", "Keys": [ { "ID": "58617dd8ce70d089e7a2669bc782472e677466278d4519c2de5ec0148b681129" } ] }, { "Name": "Repository", "Keys": [ { "ID": "10e5763ffaaa7e1e606575005f460369f9f3ef49e553914b50589f1b822f695b" } ] } ] }
]

Finally, let’s sign our tag :mine.

ubuntu@ip-{BLOCKED}-20-187:~/notary$ docker trust sign localhost:5000/ubuntu:mine
Signing and pushing trust data for local image localhost:5000/ubuntu:mine, may overwrite remote trust data
The push refers to repository [localhost:5000/ubuntu]
a4399aeb9a0e: Layer already exists 35a91a75d24b: Layer already exists ad44aa179b33: Layer already exists 2ce3c188c38d: Layer already exists mine: digest: sha256:6f2fb2f9fb5582f8b587837afd6ea8f37d8d1d9e41168c90f410a6ef15fa8ce5 size: 1152
Signing and pushing trust metadata
Enter passphrase for sample_signer key with ID f39f731: Successfully signed localhost:5000/ubuntu:mine

Another inspect shows that the :mine tag has been signed by sample_signer.

ubuntu@ip-{BLOCKED}-20-187:~/notary$ docker trust inspect localhost:5000/ubuntu:mine
[ { "Name": "localhost:5000/ubuntu:mine", "SignedTags": [ { "SignedTag": "mine", "Digest": "6f2fb2f9fb5582f8b587837afd6ea8f37d8d1d9e41168c90f410a6ef15fa8ce5", "Signers": [ "sample_signer" ] } ], "Signers": [ { "Name": "sample_signer", "Keys": [ { "ID": "f39f731f1c288b66c10d70905de6d98dfa40104741c878cb2766cddc6ed52f28" } ] } ], "AdministrativeKeys": [ { "Name": "Root", "Keys": [ { "ID": "58617dd8ce70d089e7a2669bc782472e677466278d4519c2de5ec0148b681129" } ] }, { "Name": "Repository", "Keys": [ { "ID": "10e5763ffaaa7e1e606575005f460369f9f3ef49e553914b50589f1b822f695b" } ] } ] }
]

We can get the Docker client to validate that every image from your repository is signed before deploying by setting the environment variable DOCKER_CONTENT_TRUST=1. Remember, if you use sudo to run Docker, you’ll need to use the –E flag to ensure that the environment variables are preserved.

Automate trust validation in the CI/CD pipeline

Checking if an image is signed or not (and not checking for a specific signature) does not likely solve in-house security needs. Being able to inspect signatures for any image on your repository, however, makes it possible to integrate checks into your CI/CD pipeline. Your team can write code to make sure that specific images were signed by their owners, and only those owners would have access to the private keys.

This is a good way to validate that the correct responsible party has signed off any image being deployed to production. This also makes it more difficult for an attacker trying to deploy a malicious image inside your swarm, whether it be through social engineering or some technical mechanism.

A brief discussion of the limitations

One of the natural applications for a feature like this is when providing continuous monitoring of an image’s integrity. It would be incredible to be able to monitor an image for “unapproved” changes and alert you or take action as soon as they happen. Unfortunately, this would require monitoring of the daemon, kernel, and file system level, and is simply not in the scope of what DCT does as a client-only implementation. 

The Trend Micro™ Deep Security™ solution protects hosts and provides Integrity Monitoring to provide integrity of the Docker and Kubernetes configuration files running on the same host. Trend Micro Cloud One™ – Container Security has a feature that uses its own admission controller to stop the deployment of containers based on findings from Deep Security Smart Check or other container configurations (like a privileged container or one running as root).

Read More HERE