Httptap: View HTTP/HTTPS requests made by any Linux program

The "How it was made" section of the README was not less interesting than the tool itself:

> The way we have set things up is that we live and practice together on a bit over a hundred acres of land. In the mornings and evenings we chant and meditate together, and for about one week out of every month we run and participate in a meditation retreat. The rest of the time we work together on everything from caring for the land, maintaining the buildings, cooking, cleaning, planning, fundraising, and for the past few years developing software together.

httptap is a process-scoped http tracer that you can run without root priveleges. You can run `httptap <command>` where <command> is a linux program and you get a trace of http/https requests and responses in standard output:

    httptap -- python -c "import requests; requests.get('https://monasticacademy.org')"
    ---> GET https://monasticacademy.org/
    <--- 308 https://monasticacademy.org/ (15 bytes)
    ---> GET https://www.monasticacademy.org/
    <--- 200 https://www.monasticacademy.org/ (5796 bytes)

It's a genius idea to run the process in a isolated network namespace!

I'm more interested in the HTTPS part. I see that it sets some common environment variables [1] to instruct the program to use the CA bundle in the temporary directory. This seems to pose a similar issue like all the variants of `http_proxy`: the program may simply choose to ignore the variable.

I see it also mounts an overlay fs for `/etc/resolv.conf` [2]. Does it help if httptap mounts `/etc/ca-certificates` directory with the temporary CA bundle?

[1] https://github.com/monasticacademy/httptap/blob/cb92ee3acfb2...

[2] https://github.com/monasticacademy/httptap/blob/cb92ee3acfb2...

Another tool that can be used by an unprivileged user for analysing network traffic is rootless Podman with Pasta.

Just add the podman run option

--network=pasta:--pcap,myfile.pcap

Pasta then records the network traffic into a PCAP file that could later be analysed.

I wrote a simple example where I used tshark to analyse the recorded PCAP file https://github.com/eriksjolund/podman-networking-docs?tab=re...

Does this work with larger more complicated software like web browsers, skype, or discord?

I know I'd have to run firefox with --no-remote.

Very cool idea though, love tools with this sort of UX. I look forward to a V1 release in the future.

Using a TUN device for this is a really cool idea! And the "How it was made" section is one of the best things I've read in a Github README.

I'm building something called Subtrace [1] but it can intercept both incoming and outgoing requests automatically. Looks like we converged on the same interface for starting the program too lol [2]. Subtrace's purpose is kinda different from httptap's though (more observability / monitoring for cloud backend services, hence the emphasis on both incoming and outgoing). Also, it uses a different approach -- using Seccomp BPF to intercept the socket, connect, listen, accept, and ~10 other syscalls, all TCP connections get proxied through Subtrace. We then parse the HTTP requests out of the TCP stream and then show it to the user in the Chrome DevTools Network tab, which we repurposed to work in the browser like a regular webapp.

Any fun stories there from running programs under httptap? Who phones home the most?

[1] https://github.com/subtrace/subtrace

[2] https://docs.subtrace.dev/quickstart

Neat! This will immediately be used by me to debug nginx configs. Currently I use curl -v and have to manually skim the output to figure out what's wrong, but this would immediately make redirect loops and other things apparent. Cool tool!

Mitmproxy v11.1 can do a similar thing

Is this implementing TCP in userspace?

I really like their approach. other methods that might use something like LD_PRELOAD fail on statically linked ELF's, like golang binaries.

This looks great!

The GitHub profile points to https://www.monasticacademy.org/about which I have no particular opinion on but it did leave me wondering what the connection is between their monastic training retreat and their projects on GitHub.

Edit: Oh, I didn’t go to the bottom of the readme https://github.com/monasticacademy/httptap?tab=readme-ov-fil...

Why not use eBPF instead? Then you could see all http requests from all processes at once, including ones that are already running. Plus you wouldn't need to bother with TLS at all, just hook on e.g. write(2).