What do others use for ensuring the authenticity of images after downloading them with docker pull?
We’ve setup our CI build process to use docker for consistent, cross-platform builds. To ensure that our builds don’t use a malicious docker image (because the surface area of attack with TLS is enormous if you’re using X.509), we’ve been using DCT (Docker Content Trust).
Unfortunately, I just discovered that the official docker documentation says that DCT is being deprecated. Apparently this was announced last year, and in June this blog post was published with advice:
Cosign is not secure
We spent some time looking into cosign, but we discovered that the private keys aren’t actually in the hands of the developer.
Rather, they use this complicated setup using very insecure X.509 to issue temporary certificates.
The result is that the OIDC identity provider (e.g., GitHub) extends the vector of attack significantly – to probably tens of thousands of people – that can publish a malicious image that will be accepted by cosign as “trusted”
Notation (Notary v2)
I also looked at Notation (aka “Notary v2”), but there’s no way to bootstrap the software safely, since (perplexingly) their tool for verifying the authenticity of images using cryptographic signatures itself can’t be verified using a cryptographic signature.
Alternatives
Are there any other alternatives that I can use to replace DCT to ensure the authenticity (using cryptography) of the container images that I download – where the keys are actually held by the developer (thus significantly reducing the “insider threat” risk)?
What do you (or does your org do) to ensure that you’re not using maliciously-modified containers after pulling a new docker image?
I don’t run a complex setup as it seems you do, but if I pull a new Docker container, I closely monitor my pfsense firewall logs. A lot of times, I’ll deploy a recently released container on a small test server and just observe as I run it through it’s paces. Also, I like containers that have a rather established history. I look at things like stars, how they handle bug complaints, etc. Even when updates come out, unless it’s a dire security patch, I’ll wait until all the early adopters work out the bugs and do my work for me. Early adopters are a valuable resource.
I guess you could say it all comes down to calculated risk.
How could monitoring a firewall log protect you from a Publishing Infrastructure Compromise or MITM attack? It would just show the malicious image being downloaded from the expected source…
Anyway I was gonna write a rant about it but I’m too tired. But basically the docker ecosystem is kinda fucked in this regard, and trades security in many aspects for convenience of development and distribution. This is one of the most notable examples of this.
It’s popular because it’s convinient and easy to use in many ways. If these pitfalls are a dealbreaker for you, then there is no trivial way to add the security requirements you are asking for.
You can pay for signed images from someone else, which is a little better, but there are still disadvantages.
Can you tell me a bit more about this? Who are the major providers?
Monitoring firewall logs would show calls to suspicious IPs and domains after you pulled the Docker container. A MiTM attack is usually conducted between two communicating parties, rather than inside the server itself. The attacker intercepts traffic as it travels across a network or service path. MiTM are not always done exterior of the server, but usually. Strong ciphers are your friend. Although MiTM and PIC events can overlap, a PIC is usually an attacker gaining unauthorized access to a system. A PIC compromise occurs outside the public facing server, through a stolen administrator password, compromised developer workstation, exposed API key, or hijacked cloud account. Securing API, devices, frequent password rotation are good practices. Again, not always exterior of the server, but usually.