Show HN: Timep – A next-gen profiler and flamegraph-generator for bash code

3 comments

·August 25, 2025

Note: this is an update to [this](https://news.ycombinator.com/item?id=44568529) "Show HN" post.

timep is a state-of-the-art [debug-]trap-based bash profiler that is efficient and extremely accurate. Unlike other profilers, timep records:

1. per-command wall-clock time

2. per-command CPU time, and

3. the hierarchy of parent function calls /subshells for each command

the wall-clock + CPU time combination allows you to determine if a particular command is CPU-bound or IO-bound, and the hierarchical logging gives you a map of how the code actually executed.

The standout feature of timep is that it will take these records and automatically generate a bash-native flamegraph (that shows bash commands, not syscalls).

------------------------------------------------

USAGE

timep is extremely easy to use - just source the `timep.bash` file from the repo and add "timep" in front of whatever you want to profile. for example:

    . /path/to/timep.bash
    timep ./some_script
    echo "stdin" | timep some_function

ZERO changes need to be made to the code being profiled!

------------------------------------------------

EXAMPLES

[test code that will be profiled](https://github.com/jkool702/timep/blob/main/TESTS/timep.test...)

[output profile for that test code](https://github.com/jkool702/timep/blob/main/TESTS/OUTPUT/out...)

[flamegraph for that test code](https://github.com/jkool702/timep/blob/main/TESTS/OUTPUT/fla...)

[flamegraph from a "real world" test of "forkrun", a parallelization engine written in bash](https://github.com/jkool702/timep/blob/main/TESTS/FORKRUN/fl...)

In the "forkrun test", 13 different checksums were computed for ~670k small files on a ramdisk using 28 parallel workers. this was repeated twice. In total, this test ran around 67,000 individual bash commands. [This is its `perf stat` (without timep)](https://github.com/jkool702/timep/blob/main/TESTS/FORKRUN/pe...).

------------------------------------------------

EFFICIENCY AND ACCURACY

The forkrun test (see "examples" section above) was basically as demanding of a workload as one can have in bash. it fully utilized 24.5 cores on a 14c/28t i9-7940x CPU, racking up >840 seconds of CPU time in ~34.5 seconds of wall-clock time. When profiling this group of 67,000 commands with timep:

1. the time it took for the code to run with the debug-trap instrumentation was ~38 seconds, an increase of just slightly over 10%. CPU time had a similiar increase.

2. the time profile was ready at +2 minutes (1 minute + 15 seconds after the profiling run finished)

3. the flamegraphs were ready at +5 minutes (4 minute + 15 seconds after the profiling run finished)

Note that timep records both "start" and "stop" timestamps for every command, and the debug trap instrumentation runs between one commands "stop" timestamp and the next commands "start" timestamp, meaning the error in the profiles timings is far less than the 10% overhead. Comparing the total (sys+user) CPU time that perf stat gave (without using timep) and the CPU time timep gives (from summing together the CPU time of all 67,000-ish commands), the difference is virtually always less than 0.5%, and often less than 0.2%. Ive seen as low as 0.04%, which is 1/3 of a second on a run that took ~850 seconds of CPU time.

------------------------------------------------

MAJOR CHANGES SINCE THE LAST "SHOW HN" POST

1. CPU time is now recorded too (instead of just wall-clock time). This is done via a loadable builtin that calls `getrusage` and (if available) `clock_gettime` to efficiently and accurate determine the CPU time of the process and all its descendants.

2. the .so file required to use the loadable builtin mentioned in #1 is built directly into the script has an embedded compressed base64 sequence. I also developed the bash-native compression scheme that it uses. The .so files for x86_64, aarch64, ppc64le and i686 are all included. Im hoping to add arm7 soon as well. the flamegraph generator perl script is also embedded, making the script 100% fully self-contained. NOTE: these embedded base64 strings include both sha256 and md5 checksums of the resulting .so file that are verified on extraction.

3. the flamegraph generation has been completely overhauled. The flamegraphs now a) are colored based on runtime (hot colors = longer runtime), b) desaturate colors for commands where cpu time << wall-clock time (e.g., blocking reads, sleep, wait, ...), and c) use a runtime-weighted CDF color mapping that ensures, regardless of the distribution of the underlying data, that the resulting flamegraph has a roughly equal amount of each color in the colorspace (where "equal" means "the same number of pixels are showing each color"). timep also combines multiple flamegraphs (that show wallclock time vs cpu time and that us the full vs folded set of traces) by vertically stacking them into a single SVG image, giving "dual stack" and "quad stack" flamegraphs.

4. the post-processing workflow has been basically completely re-written, making it more robust, easier to understand/maintain, and much faster. The "forkrun" test linked above (that ran 67,000 commands) previously took ~20 minutes. With the new version, you can get a profile in 2 minutes or a profile + flamegraph in 5 minutes - a 4x to 10x speedup!

Visit

Pawamoy

I'll get a closer look at the code tomorrow, but maybe you can provide an answer in the meantime: how do you handle line numbers? Bash's LINENO is unreliable, contrary to Zsh's, see https://gist.github.com/pawamoy/cca35f0f5ccd56665d6421e9b2d2.... That's why I eventually gave up writing any profiling/tracing/debugging/coverage tool for Bash and moved to support these in Zsh instead. Unfortunately Zsh is missing a ZSH_XTRACEFD like Bash, but that's another story (never finished contributing one). Here's my own attempt at a shell profiler: https://github.com/shellm-org/profiler :)

imiric

I'm surprised there's been no discussion on this project.

A part of me thinks that if you have to profile a Bash script, it's a sign that you've surpassed its intended use case, and a proper programming language with all its modern tooling would be a saner way forward. But then again, that line is often blurry, and if the script has already grown in size, it might be a sunk cost to consider rewriting it, so I can see cases where this tool would come in handy.

I find it hard to believe that there is minimal overhead from the instrumentation, as the README claims. Surely the additional traps alone introduce an overhead, and tracking the elapsed time of each operation even more so. It would be interesting to know what the actual overhead is. Perhaps this is difficult to measure with a small script, whereas a larger one would show a greater difference.

The profile output is a bit difficult to parse at first glance, but I think I get it after a closer look.

In any case, this is some next-level wizardry. The amount of patience required to instrument Bash scripts must've been monumental. Kudos to you, Sir.

BTW, you're probably aware of it, but you might want to consider using Bats[1] for tests. I've found it helpful for small scripts.

EDIT: I took a look at `timep.bash`, and I may have nightmares tonight... Sweet, fancy Moses.

Also, I really don't like that it downloads and loads some random .so files, and base64-encoded blobs. I would personally not use this based on that alone, and you shouldn't assume that users will trust you. If this is required for correct functionality, have a separate well-documented step for users to download the required files manually. Or better yet: make their source visible as well, and have users compile them on their own.

[1]: https://bats-core.readthedocs.io/

jkool702

Yay, a comment!

>I find it hard to believe that there is minimal overhead from the instrumentation, as the README claims. Surely the additional traps alone introduce an overhead, and tracking the elapsed time of each operation even more so. It would be interesting to know what the actual overhead is.

note that timep does a 2 pass approach - it runs the code with instrumentation just logging the data it needs, and then after the code finishes it goes back and uses that data to generate the profile and flamegraphs. For "poverhead" im juyst talking abut in the initial profiling run...total time to getting results is a bit longer (but still pretty damn fast).

if you run timep with the `-t` flag it will run profiling run of the code (with the trap-based instrumentation enabled and recording) inside of a `time { ... }` block. You can then compare that timed to the running the code without using timep, giving you overhead. I used that method while running a highly parallelized test that, between 28 persistent workers, ran around 67000 individual bash commands (and computed 17.6 million checksums of a bunch of small files in a ramdisk) in 34.5 or so seconds (without timep). using `timep -t` to run the code this increased to 38 seconds. So +10%. And thats really a worst case scenario...it indicates the per command overhead is around 1 ms. what percent if the total run time that translates to depends on the average time-per-command-run for the code you are profiling.

side note: in this case, the total time to generate a profile was ~2.5 minutes and the total time to generate a profile and flamegraphs was ~5 minutes

timep manages to keep it so low because the debug trap instrumentation is 100% bash builtins and doesnt spawn and subshells or forks - so you just have a string of builtin commands with no context switching nor any "copying the environment to fork something" to slow you down.

> The amount of patience required to instrument Bash scripts must've been monumental.

It took literally months to get everything working correctly and all the edge cases worked out properly. Bash does some borderline bizarre things behind the scenes.

> EDIT: I took a look at `timep.bash`, and I may have nightmares tonight... Sweet, fancy Moses.

Its a little bit...involved.

> Also, I really don't like that it downloads and loads some random .so files, and base64-encoded blobs.

So by default it doesnt do this. It has the ability to, but you have to to explicitly tell it to....it wont do it automatically.

By default, it will get the .so file using the base64 blob (in ascii string representation and compressed) that is built into timep.bash. This has sha256 and md5 checksums incorporated into it that get checked when the .so file is re-created using that base64 blob (to ensure no corruption).

the .so file is needed to add a bash loadable builtin that outputs microsecond granularity CPU usage time. Without this it will try and use /proc/stat, which works but the measurement is 10000x more coarse (typically it displays cpu time in number of 10 ms intervals). to get microsecond accuracy you need the .so file.

> Or better yet: make their source visible as well, and have users compile them on their own.

the source is available in the repo at https://github.com/jkool702/timep/blob/main/LIB/LOADABLES/SR...

compile instructions are at the top in commented out lines.

the code is pretty straightforward - it uses getrusage and/or (if available) clock_gettime to get cpu usage for itself and its children.

the `_timep_SETUP` function has the logic included (but not used by default - you have to manually call it with a flag) to let allow you to use a .so file that is in your current directory instead of the one generated from the builtin base64 blob. So, you are able, should you wish, to compile and have timep use your own .so for the loadable.

However, realistically, most people who might be interested in using timep wont do that. So it defaults to fully automating this and having it all self contained in a single script file, while allowing for advanced users to manually override it. I thought that was the best overall way to do it.