What do others use for ensuring the authenticity of images after downloading them with docker pull?
We’ve setup our CI build process to use docker for consistent, cross-platform builds. To ensure that our builds don’t use a malicious docker image (because the surface area of attack with TLS is enormous if you’re using X.509), we’ve been using DCT (Docker Content Trust).
Unfortunately, I just discovered that the official docker documentation says that DCT is being deprecated. Apparently this was announced last year, and in June this blog post was published with advice:
Cosign is not secure
We spent some time looking into cosign, but we discovered that the private keys aren’t actually in the hands of the developer.
Rather, they use this complicated setup using very insecure X.509 to issue temporary certificates.
The result is that the OIDC identity provider (e.g., GitHub) extends the vector of attack significantly – to probably tens of thousands of people – that can publish a malicious image that will be accepted by cosign as “trusted”
Notation (Notary v2)
I also looked at Notation (aka “Notary v2”), but there’s no way to bootstrap the software safely, since (perplexingly) their tool for verifying the authenticity of images using cryptographic signatures itself can’t be verified using a cryptographic signature.
Alternatives
Are there any other alternatives that I can use to replace DCT to ensure the authenticity (using cryptography) of the container images that I download – where the keys are actually held by the developer (thus significantly reducing the “insider threat” risk)?
What do you (or does your org do) to ensure that you’re not using maliciously-modified containers after pulling a new docker image?
You must log in or register to comment.
Assuming the risk of being naive… Why is checking the hash not enough?
Or didn’t I understand your problem?
Because the hash comes from the same source as the image itself.
So if someone compromised the publishing infrastructure (or does a MITM attack), they can trivially maliciously modify the hash as easily as they can maliciously modify the image. It provides zero security to Publishing Infrastructure compromise.
A good historical example of this happening was when monero’s release infrastructure was comprimised. And here’s a great list of historically relevant cases where this happened:
In the case of Monero, the users were able to verify that the release was malicious – because the cryptographic signature was invalid, and it was fixed very fast. If the user can only check checksums, they have literally no way to detect if the publishing infrastructure or a MITM attack is taking place.
Got it, I thought it was a last segment attack (image substition) but now I see you are aiming to a fully supply chain attack.
And if you find an nswer to that, please let me know because this things have virtually not a solution that ticks all boxes
What do you (or does your org do) to ensure that you’re not using maliciously-modified containers after pulling a new docker image?
I don’t run a complex setup as it seems you do, but if I pull a new Docker container, I closely monitor my pfsense firewall logs. A lot of times, I’ll deploy a recently released container on a small test server and just observe as I run it through it’s paces. Also, I like containers that have a rather established history. I look at things like stars, how they handle bug complaints, etc. Even when updates come out, unless it’s a dire security patch, I’ll wait until all the early adopters work out the bugs and do my work for me. Early adopters are a valuable resource.
I guess you could say it all comes down to calculated risk.
How could monitoring a firewall log protect you from a Publishing Infrastructure Compromise or MITM attack? It would just show the malicious image being downloaded from the expected source…
Anyway I was gonna write a rant about it but I’m too tired. But basically the docker ecosystem is kinda fucked in this regard, and trades security in many aspects for convenience of development and distribution. This is one of the most notable examples of this.
It’s popular because it’s convinient and easy to use in many ways. If these pitfalls are a dealbreaker for you, then there is no trivial way to add the security requirements you are asking for.
You can pay for signed images from someone else, which is a little better, but there are still disadvantages.
You can pay for signed images from someone else
Can you tell me a bit more about this? Who are the major providers?
Monitoring firewall logs would show calls to suspicious IPs and domains after you pulled the Docker container. A MiTM attack is usually conducted between two communicating parties, rather than inside the server itself. The attacker intercepts traffic as it travels across a network or service path. MiTM are not always done exterior of the server, but usually. Strong ciphers are your friend. Although MiTM and PIC events can overlap, a PIC is usually an attacker gaining unauthorized access to a system. A PIC compromise occurs outside the public facing server, through a stolen administrator password, compromised developer workstation, exposed API key, or hijacked cloud account. Securing API, devices, frequent password rotation are good practices. Again, not always exterior of the server, but usually.
I think the most popular solution is locally building and registry.
but how do you verify the sources of what you’re fetching?
Is it coming from an unsigned git repo? That seems equally vulnerable
How do you verify that nobody is holding the original developer at gunpoint making them sign their compromised software with their real key?
The point I’m trying to make: At some point, you have to trust something which you can not feasibly verify.
For most cases the github repo and an occasional look on a relevant newsfeed is good enough. If it’s not good enough for yours, the first question still stands.
A common distribution method involves multi-party signing, that is, multiple developers use keys to sign off on reviewed changes.
Multiple developers review the changes, before signing the git commit after review. Then they build the package, either locally or on CI servers, but again, multiple parties/servers sign and review, doing a reproducible build to verify across machines.
In an ideal architecture, there is never a single point of failure. You would have to compromise the computers of multiple devs, or multiple build servers that are building signed reproducible builds, in order to do it.
Although in theory, you could compromise all of them. But it’s extremely difficult.
Yes, you have to trust someone.
However, by using cryptographic signatures on release artifacts (eg container image layers), we can reduce that risk from having to trust tens of thousands of people to just one person.
That’s a hugely meaningful reduction of risk.
Also what are you building? You might be able to replace docker with nix for example.
Open-Source Software.
Can I install nix securely (eg with
apt) to run inside the free CI runners provided by GitHub, GitLab, and Codeberg?Maybe not apt…but
sudo dnf install nix nix-daemonEdit: which doesn’t answer your question about free runners… but can you trust them anyway? They are running a container afaik
We do our pre-releases on CI runners for convenience. GitHub automatically kicks-off a build when we
git push.When it comes time to do a real release, we can just run the job locally on our build machines (this is easier thanks to docker). Because our builds are reproducible, we can just check that the hash matches on our local build and the one from the free GitHub CI runners. That way we don’t have to trust the infrastructure, but we can use it for free & easy iteration before our final release.
Nix is also packaged in debian as nix-bin.
