Comment by smw
Comment by smw 3 days ago
Can it decrypt tls? Perhaps by hooking the calls to common libraries?
Comment by smw 3 days ago
Can it decrypt tls? Perhaps by hooking the calls to common libraries?
This will not work with HPKP but hopefully nothing is using that any more. ( https://en.m.wikipedia.org/wiki/HTTP_Public_Key_Pinning )
It won't work with programs that defensively validate the cert chain but those are rare.
It won't work with programs that embed their own root cert store, which is also rare but I would guess less rare than the previous one. The usual reason to do this is to minimize OS deps, and in the case of Docker containers to save on container image size by only including the roots you care about.
But yes for the vast majority of programs it should work fine.
Yep, certificate pinning is the one scenario Subtrace can't handle in my experience, but thankfully, it's fairly rare like you said. And IMO there is no general solution to the problem [1], but it's one of those very interesting problems to daydream thinking about when you're stuck in traffic or whatever :)
We still try our best by handling as much of the long tail of environments with some library/framework specific workaround (e.g. Deno bundles all TLS certs in its binary so we set the DENO_CERT env var when applicable).
in a different context, maybe: https://news.ycombinator.com/item?id=38044786 :)
kubeshark [0] is using ebpf to catch calls to openssl/go's tls lib and thus no need to juggle certs. Has pros and cons compared to your method, but an interesting comparison.
The approach you describe above is common for similar projects:
- Pixie (https://px.dev) -- which I contribute to
- Beyla (https://github.com/grafana/beyla)
- Coroot (https://github.com/coroot/coroot)
If you are interested in the details and how the strategy for this tracing has evolved, you can learn more in this blog (https://blog.px.dev/ebpf-tls-tracing-past-present-future/).
Is this a different method from the httptap [1] that was on hackernews a few weeks ago? Somebody in that post seemed to say that it also generates CA certificates on the fly.
Up front, this is not my area of expertise. You would still not want to run this on the same server as your containers since someone could inject their own cert? I think it allows someone to decrypt traffic that isn't just proxied to Subtrace?
Edit: I've reviewed the docs and it looks like you do run it on the same server. For clarity, I've used Sentry before.
Subtrace proxies the program's connection using a regular TLS connection to the upstream server. For example, if you do `subtrace run -- curl https://example.com`, curl thinks it's talking to example.com over TLS, but it's really talking to Subtrace locally. Since we injected the ephemeral root CA into the system store, curl will trust the valid TLS certificate that Subtrace presents for example.com. From within the same server, Subtrace will handle the actual TLS connection to upstream example.com. That upstream connection is undecipherable to outsiders.
Everything is exactly as secure as before Subtrace. In other words, using Subtrace doesn't make the NSA's job any easier ;)
Wow. I think more network inspection tools should adopt an approach like this. Great job.
Yes, but we've managed to do it automatically without any library/language specific hooks! It's probably one of my favourite things in Subtrace :)
We generate an ephemeral TLS root CA certificate and inject it into the system store. The generated certificate is entirely in-memory and never leaves the machine. To make this work without root privileges, we intercept the open(2) syscall to see if it's /etc/ssl/certs/ca-certificates.crt (or equivalent). If so, we append the ephemeral root CA to the list of actual CA certificates; if not, we let the kernel handle the file open like usual. This way, none of the other programs are affected, so only the program you start with `subtrace run` sees and trusts the ephemeral root CA.
After we get the program to trust the ephemeral root CA, we can proxy outgoing TLS connections through Subtrace transparently but also read the cleartext bytes.
All of this is fully automatic, of course.