Show HN: Uscope, a new Linux debugger written from scratch
132 comments
·January 31, 2025feelamee
> The available Linux debuggers are gdb and lldb. They both suck. They crash all the time, don't understand the data types I care about, and they don't make the data I need available at my fingertips as quickly as possible.
quote from https://calabro.io/uscope
Of course gdb, lldb have their problems (e.g. smashing tui with app output, what can be easily fixed, or very very very very long tab-tab completion, and crashing of course), but I dont know anything better. I am forced to use visual studio at work and its debugger really sucks - it can't even compare strings in conditional breakpoint, it cant even skip all std::function / std::bind garbage to let me step in callback, it can't watch evaluated expressions. Probably it can evaluate exprs (immediate window?), but there are very little guides about this.
So, gdb is winner for me now. rr (record-repeat)[0] also looks very nice, but it require hardware support(((
AlotOfReading
If you're willing to deal with enterprise pricing, Undo [0] implements something reasonably comparable to rr without the hardware timer requirements.
[0] https://undo.io/
matu3ba
Are you aware of solutions for multi-threaded and relative time-accurate recordings, for example by using user-selected synchronization points? Afaiu, rr and undo do simulation of threads on one CPU core, which rules out detecting timing issues.
gregthelaw
Co-founder of Undo here. This is a common misunderstanding, and just not true -- neither for Undo nor rr. Most races will reproduce at least as easily in Undo, especially if you use our "thread fuzzing" feature (rr has something similar, called chaos mode).
Sure, there will always be some races/timing issues that just won't repro under recording (Heisenberg principal and all that), but in fact most races are _more likely_ to occur under recording. Part of this is because you slow down the process being recorded, which is equivalent to speeding up the outside world.
And of course, when you do have your gnarly timing issue captured in a recording, it's usually trivial to root-cause exactly what happened. Our customers tell us that races and timing issues are a major use-case.
Veserv
Multiplexing onto a single thread is sufficient to observe and record concurrency errors. If that is not sufficient, then I am assuming you want to observe and record errors caused by true parallelism. If so, then you need a full memory trace. That restricts you to either hardware that supports full memory trace connected to a hardware trace probe or instrumented software full memory trace.
The former basically only exists for embedded boards and the latter does not exist (at say less than a 10x slowdown) for Linux or any other common desktop operating system as far as I am aware.
dzaima
While the single-threaded execution means that issues from thread interleaving on the scale of nanoseconds will effectively not happen, multiple threads are still allowed and will be context-switched between. rr also has a chaos mode to intentionally make the context switching unfair.
AlotOfReading
Antithesis is the only general purpose system I've seen for that. It takes the same single threaded approach, but can scale to N separate systems and fault injecting possible orderings.
feelamee
eh, thanks
> How can I persuade my boss to pay for Undo?
fun quote.. but my boss will never pay for it because I am the only one who use gdb in our company, unfortunately)
mark_undoio
For what it's worth we do offer a more printf-y interface for people who don't like a debugger - https://docs.undo.io/PostFailureLogging.html
And CLion / VS Code for people who prefer an IDE interface.
But a lot of people do really want to stick with their printf debugging.
If your boss won't buy you an Undo you can still use https://rr-project.org/ - or on Windows the built in time travel debug of WinDbg.
alcover
> gdb(..) smashing tui with app output
this got me losing my mind. How/why propose this tempting TUI mode when the result looks like a broken arcade game ??How do you people comfortably debug C in Linux ? I know VSCode looks nice but by principle I can't accept to use such a beast to basically edit code..
marssaxman
> How do you people comfortably debug C in Linux?
same way I debug everything, everywhere: logging.
debeloo
That not very environmentally friendly.
imron
> How do you people comfortably debug C in Linux ?
I just got comfortable using gdb/lldb from the terminal. Once you get used to it, it's fine (albeit not pretty).
baruch
The advantage for text based interfaces is that they work over ssh/tmate over trans atlantic connections whereas the GUI tools would suffer greatly. It just works everywhere and if you learn it, it will be just as good as a gui debugger. Potentially even more ergonomic without hunting for options with a mouse.
dan00
I‘m using cgdb, which is a minimal ncurses wrapper around gdb. It‘s a lot better than the TUI of gdb.
Icathian
Termdebug works great with gdb, and I get my usual editor features as well as the full functionality of gdb. Seems fine to me.
Before I switched from emacs I had an equivalently good setup with dap-mode.
billfruit
Use Emacs to drive gdb (through the GUD mode) gives a much more ergonomic interface. It highlights the line of code being traced in the code window.
feelamee
as I said - this easy to fix once and for ever
Also not all apps write to stdout/stderr by default.
> How do you people comfortably debug C in Linux ?
It depends on what comfortable is for you. Most of my pc experience is terminal and browser and this is comfortable for me. I just use gdb for debugging. Sometimes trying lldb
dima55
emacs. Worked great for decades and works great today. Learn it.
khuey
rr should work on any remotely modern Intel system, and generally on AMD's Zen CPUs too. Unless you're in a virtualized environment (some of which are supported) or a more esoteric architecture rr probably works on your silicon.
feelamee
I have AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx (8) @ 2.100GHz.
As I remember - it is should work according to documentation, but I couldn't launch it. Probably I'm not spend enough time to solve errors
bean-weevil
You probably need the fix from this page: https://github.com/rr-debugger/rr/wiki/Zen
imron
Which visual studio are you using?
It’s been a number of years since I’ve used it but Visual Studio PRO could do all these things - at least as long as I was using it (since visual c++ 5).
VS Code on the other hand is no where near as featured or powerful.
feelamee
I use VS2022 Enterprise
If you know solutions, I will be very thankful for any info.
P.S.
Note, though I meet all of this problems, probably I don't spent enough time to find a solution (maybe tried first links at google and so on). E.g. tried `strcmp` for breakpoints, tried to write .natstepfilter.
So, if VS really can do all of this, I'm sorry for my hurry.
Kuinox
I don't know for c but for C# you can write a custom expression that get evaluated for the condition.
caspper69
Man, if you read my comments lately you would think I'm a Microsoft fanboy, sheesh.
But honestly, in all my years, Visual Studio has been (by far) the best non-commercial (or should I say built-in?) debugger that I've used, and that includes gdb.
I am not a huge c++ on Windows guy though, so YMMV.
Here are a few guides that you may find helpful (and I am also going to include the beginner one, but please do not take that as an indictment of your skill level, I am including only for completeness).
These are all for VS2022:
C++ Debugging Tutorial: https://learn.microsoft.com/en-us/visualstudio/debugger/gett...
C++ Breakpoint Debugging: https://learn.microsoft.com/en-us/visualstudio/debugger/usin...
Breakpoint/Watch Expressions (pay attention to the debugger intrinsics): https://learn.microsoft.com/en-us/visualstudio/debugger/expr...
High Level Debugger Tour: https://learn.microsoft.com/en-us/visualstudio/debugger/debu...
VS2022 Debugging TOC: https://learn.microsoft.com/en-us/visualstudio/debugger/?vie...
My apologies if you've already found these references and they don't do you any good, but your issues just don't sound like the types of issues I've ever experienced with the debugger, and sometimes MS' documentation is just disorganized and incomplete.
mort96
I'm a UNIX guy through and through, literally everything I ever do is writing code on Linux or macOS which targets Linux and/or macOS. I loathe almost everything about the whole Windows ecosystem, from Microsoft as a company to the way they build their UIs to the technical foundations of Windows.
However I have to give them one thing: their developer tooling with Visual Studio and other first-party tools seem vastly superior than anything on macOS/Linux when it comes to debugging. I would never use it as a code editor, but it's clear that a lot of effort has been invested into the debugging experience.
feelamee
very thanks, this will be useful to simplify my debugging!
> and I am also going to include the beginner one, but please do not take that as an indictment of your skill level
Don't worry, I am really feel myself as newbie in windows
n4r9
Out of interest what are your main use cases for needing conditional breakpoints?
feelamee
For example for debugging gui apps. Let's say we have function `on_event(event_type)` and you want to examine execution if `event_type == mouse_up`.
In reality conditional breakpoint is the same as simple, but simple require support from code:
``` void on_event(event_type type) { if (type == mouse_up) { // set breakpoint here } } ```
Also useful to break depending on call stack[0]
[0] https://sourceware.org/gdb/current/onlinedocs/gdb.html/Conve...
n4r9
Yeah, I was specifically wondering about cases where you can't or don't want to change the source code. Sometimes that happens when I need to explore the state after a long process e.g. loading a large file. Generally though I avoid them like the plague as they make the code so slow. Curious to know what other people use it for.
godelski
> Build as a library so other people can build other interesting things as well
I LOVE this!I firmly believe so much tech has gone to shit because things are no longer hackable. We say "move fast and break things" but we try so hard to prevent that that when we do break things we just create bigger messes and debt, so no one cleans it up. It seems slower to write hackable code but that's in the moment. In the long run it is much faster. Not to mention you get added benefits of others being able to contribute more easily with a lower likelihood of actually breaking shit.
Analemma_
Is gdb another thing like gcc where the un-hackability and un-extendability was a deliberate choice by rms to ensure nobody would ever build proprietary toolchains on top of it?
debatem1
I don't know if it was deliberate, but writing code that interfaces with GDB is unpleasant enough that I opted to build our debugger-like tooling in eBPF + pyelftools instead.
dooglius
No, GDB has a pretty good Python extension framework
mort96
That's not comparable. That lets you extend GDB with your own additions. Having the debugger work as a library means being able to incorporate the debugger's functionality into other systems.
junon
Yes, this. I write a lot of my own GDB tooling for debugging my kernel.
godelski
gdb is a bit old and my comment is really more about building things in general.
You should always make things hackable, not just for others, but for you. One truth to coding is that the final thing will never end up where you think it will. So if you don't make your code flexible (i.e. hackable) then you're going to keep breaking it while fixing it. Things will continue to be patches and quick fixes. Nothing is more permanent than a temporary fix that works.
Truthfully, this is part of the unix philosophy. Make your programs small and integratable. The strategy is not to be finished, because there is no end, the strategy is to be adaptable, because there is no end.
tux1968
This is great to see. It'd be lovely if Linux gets a decent debugger. Another project to keep an eye on is https://github.com/EpicGamesExt/raddebugger Although, they haven't expanded much beyond Windows, yet.
mesbahamin
They fully plan to support Linux.
tux1968
Author just did a podcast about Uscope a few minutes ago, where they mention this HN post:
Mentioned at : https://youtu.be/stWBTv6grBc?t=456
lsllc
Looks promising -- it's about time, I haven't been satisfied with any debugger since the days of Periscope!
https://www.os2museum.com/files/docs/periscope/periscope-man...
malkia
Somehow this brought the memory of Numega's awesome SoftICE!
zwieback
Oh yea, we used both SoftICE and Periscope. I loved the pushbutton to fire off an NMI directly on the ISA bus. It was sad when your device driver crashed but it was a fun experience to push that button.
malkia
I've used only few times back then, but it was impressive (previous experience was with Borland's / Turbo debuggers which was nice also, but Numega was just another class)
IAmLiterallyAB
Yay this is awesome! GDB is a buggy (https://sourceware.org/bugzilla/show_bug.cgi?id=18772 https://sourceware.org/bugzilla/show_bug.cgi?id=9425) mess and rough code quality, I've wanted a do something like this for a while.
To be 100% clear, it's not using gdb/gdbserver under the hood right?
The bugs I linked above are over a decade old, and I have to patch them every time I compile GDB server. Ultimately (IIRC) GDB needs to rework how it handles signals (to their credit, ptrace is a horribly stupid API, especially before PTRACE_SEIZE, so I don't blame them for having issues)
mort96
Interesting, LLDB also has a serious bug related to ^c! In LLDB, if you run 'lldb -o run /path/to/executable', it won't pause on SIGINT. As a result, you can't have an ergonomic setup where you have e.g a make target to run your software under LLDB, you always need to manually type 'run' for ^c to work.
jcranmer
This is definitely an ambitious project, and I worry that you are biting off more than you can chew in doing so. (I've attempted my fair share of debugger projects in the past).
At a low level, one of the main problems is that Linux's kernel interfaces for debugging are just absolute trash. (I see you have multithreaded support mentioned as a future task item, and that's one of the areas where you discover just how bad it really is). And of course ptrace composes somewhat poorly with, well, anything else, so if you want to use perf_event or eBPF to help drive the low-level stuff, well, combining everything into a single event loop is just painful. (Oh, and the documentation for all of this stuff sucks.)
At the medium level, everything is hampered by the number of secret handshakes that go on between the compiler, the linker, the runtime loader, the debugger. And the chronic issue that, in the debugger, you have to be prepared for the possibility that everything is just catastrophically wrong: your stack might be garbage (or downright nonexistent), the thread-local storage register might be a garbage value, variables may have values that don't make sense. After all, you're resorting to a debugger when things don't work, and part of the reason why it might not be working is because you've accidentally corrupted everything.
And at the high level, there's the UI which, as someone who doesn't work on UI, I find terrifying in its own right.
Trying to solve not one of these issues, but all of them, at once, in a new project is ambitious. I'd personally prefer a lot more a project that only tried to tackle one slice of the stack, not all of it.
HexDecOctBin
Yup, Linux is nowhere the Unix successor we deserved. I have started multiple systems projects in Linux and gave always given up due to how shoddy the foundations are. It seems like the kernel just copies whatever cool feature they find in Solaris, BSD, Plan 9 or even NT without paying any attention to their composability, while the user space people are just a clique trying to force their way on everyone (why is the loader part of libc again?).
jcranmer
There are a few things where Linux has been the innovator (eBPF comes to mind).
But process control is not one of them, and almost any other operating system manages to have a more sane interface. Personally, if I were writing a debugger, I think the OS with the sanest kernel interface is probably Fuchsia, partially because everything is handle-based and partially because pretty much every syscall takes in a handle to the process to operate on, so you can do things like manipulate memory maps of another process without driving yourself insane.
AndyKelley
He's live on twitch right now demoing this on zig showtime: https://www.twitch.tv/kristoff_it
titzer
Nice project!
One killer feature would be the ability to connect to the debugger via a socket and control it. Gdb has this interface and for some use cases it's great.
As one of those long-tail "native" languages, Virgil might benefit from this. So far, I've had a student build a DWARF backend, and the experience from that is that DWARF is way too complicated and consequently implementations are broken and crappy in many ways. I think DWARF draws the wrong dividing line here. Control of the machine and customizing the source-level support to the language is probably better.
mindcrime
Speaking as a Java guy, remote debugging is SO useful (at times anyway). Thankfully the JVM has built in support for a remote debugging protocol[1] and there are good debuggers that can connect to a running Java system (if it was started with the correct command line flags) and do symbolic debugging over the network.
There can be certain situations where the network latency can make things difficult, but generally speaking I find it an incredibly useful facility to have.
[1]: https://docs.oracle.com/javase/8/docs/technotes/guides/jpda/...
funcDropShadow
Which debuggers do you like the most?
peterfirefly
> the experience from that is that DWARF is way too complicated
The representation and the compression are far too intertwined :(
peterfirefly
Oh, and there needs to be some sort of declarative, rules-based DWARF checker. Preferably one that doesn't only work on complete files (or collections of files) but also handles snippets.
If it's fast and can work on snippets before they are even written to disk, we can probably catch many compiler bugs.
sroussey
Have it use the WebKit debugger protocol and we can use browser dev tools as the UI.
:p
s3graham
Keep at it!
raddbg is one worth watching too (currently only Windows x64) https://github.com/EpicGamesExt/raddebugger
meitham
Very happy to see the increasing momentum in the zig community
koek67
Nice work! I remember meeting you at Systems Distributed in NYC where you mentioned this project, so cool to see the progress. Well done!
Hi! I've been building a debugger on my nights and weekends because it's fun, and I personally need a better debugger for my work. GDB and LLDB pain me greatly; we can and will do better!
As explained in the README, it's still very early-days and it's not ready for use yet, but check back often because it's improving all the time!
Check out https://calabro.io/uscope for a more detailed explanation.
Thanks for taking a look!