Hi gophers,
I recently created a tool that uses gVisor and linux network namespaces to log the http/https requests and responses made by an arbitrary linux command, without needing root privileges, and without making global system changes that would affect other running processes. The idea is you run httptap -- <command> and you get a nice log of http and https requests made by <command>. You can print the full request bodies, or generate a HAR dump for visualization later if you want. For example:
httptap -- python -c "import requests; requests.get('https://monasticacademy.org')"
---> GET https://monasticacademy.org/
<--- 308 https://monasticacademy.org/ (15 bytes)
---> GET https://www.monasticacademy.org/
<--- 200 https://www.monasticacademy.org/ (5796 bytes)
In this example it’s a python interpreter that is being monitored, but it just as easily could have been any other linux command. for example, here is httptap printing requests made by kubectl:
httptap -- kubectl get all --insecure-skip-tls-verify
---> GET https://somecluster/api/v1/namespaces/default/pods?limit=500
<--- 200 https://somecluster/api/v1/namespaces/default/pods?limit=500 (38345 bytes)
---> GET https://somecluster/api/v1/namespaces/default/replicationcontrollers?limit=500
<--- 200 https://somecluster/api/v1/namespaces/default/replicationcontrollers?limit=500 (2509 bytes)
...
https://github.com/monasticacademy/httptap
Why does this exist?
There are many of tools out there for general network inspection (tcpdump, wireshark), but capturing https traffic and decrypting it is impossible unless I can dump TLS session secrets from the program I’m trying to trace, which is usually not simple at all.
There are some good TLS proxies out there (mitmproxy, for example), but to use these I have to manually install CA roots into my system, and manually configure a http proxy for the program I’m trying to run. A lot of programs don’t respect HTTP_PROXY/HTTPS_PROXY environment variables, so I have to dig through the docs for the particular program I’m running. Then afterwards I have to start the proxy server in one place, run the command in another place, then run a command to dump the output (or look at a web interface) in another place. I wanted something that was scoped to a single linux process, and had a command line interface analogous to, for example, strace <command>.
What are some use cases?
- My own personal use case that motivated this project was an attempt to use the Oracle Cloud API, where some of the documentation was out of date. I had the oracle CLI doing the thing I wanted, but I was having difficulty doing the same thing programmatically from Go. The APIs were clearly not meant to be secret or private so I wanted to see the exact API was that the oracle CLI was making.
- In general this tool will be useful for understanding how things work, for reverse engineering, for debugging, for penetration testing, for privacy reviews, and so on.
How does it work?
The tool actually has its own userspace TCP/IP stack, which it gets from gVisor. The subcommand is run in an isolated network namespace, and all traffic in and out is intercepted at the IP level. I then have some Go code to proxy the traffic out to the internet.
To decrypt TLS traffic I use the standard approach of generating a custom certificate authority, and telling the command to trust it. To do this I looked through all the major TLS implementations out there and made a list of environment variables that can be used to inject a custom CA. This won’t work in case of an unknown TLS implementation or in the case of a binary with pinned CAs.
Can it be ported to operating systems other than linux?
Sadly, I don’t think this will end up being ported to mac or windows in anything like its current form. Httptap makes heavy use of linux network namespace, and unless I’ve missed something neither mac nor windows has anything similar to a network namespace (please correct me if I’m wrong about this!). I would absolutely love to port httptap to other operating systems so if you have any suggestions on how it could be done then please let me know.