New horizons for Julia
73 comments
·February 20, 2025affinepplan
SatvikBeri
We've quietly been using Julia at our hedge fund for ~4 years. It has saved us a lot of money.
The syntax is easy for Math-heavy Researchers who might not have as much software engineering experience to use, but it's much faster than numpy, which we used for 5 years before.
The package management is sane – not as good as Rust, but way better than Python/Javascript.
We have one non-trivial macro which makes our codebase ~15% shorter, and would be hard to emulate in a lot of other languages.
The REPL is really good, second only to Clojure IMO.
The main thing I want to see more of is "static Julia", e.g. better support for static type-checking and binaries, and fortunately that seems to be where the language is going!
t_mann
I am curious, have you run into any of the issues with Julia discussed here [0] (also discussed on HN [1])? Those problems would seem particularly troubling at a hedge fund to me, if you have to question the accuracy of your results after chaining multiple complex models together. Curious about your stance on / experience with correctness issues in Julia.
SatvikBeri
We've run into issues with Arrow – it or its dependencies would frequently break on updates, and we stopped using it. But none of the other issues mentioned in the article have ever come up for us. In particular, a lot of the problems come from using OffsetArrays.jl, which is a library designed to let you use different starting indices for an array, which we just don't use.
It's worth noting that we initially migrated ~15kloc of numpy data/numeric pipelines to Julia, and mostly found the same values, with the exception of something that turned out to be a bug in a Python library.
I would say that the biggest Julia annoyance we've run into has to do with the way the Expr type is implemented, particularly the fact that it's mutable, which makes some of the metaprogramming we want to do substantially harder. But that's not a bug per se, just a design choice I don't like.
affinepplan
I won't pretend that Julia doesn't have its fair share of `wats`, but in my experience they tend to be in one of two categories
* usage of `OffsetArrays.jl` (really, just avoiding this package fixes most of the issues)
* hideously cursed syntax that should only pass a code review if your name is Lovecraft, but for some reason the parser allows it
if you don't do either of those things (which at least personally speaking, I don't) then I don't think the rate of "correctness issues" is any higher or lower than I experience in other ecosystems. In fact, it's probably lower
Don't forget that other languages are not immune... I love the `polars` library as well but in my two years of using it I've encountered organically two separate "correctness" bugs. It's just par for the course for any big code surface
_benj
> We've quietly been using Julia at our hedge fund…
This! I’ve found a lot of value in Julia for math specially in financial analysis.
Julia’s DataFrame and general development workflow are excellent for research and when it comes to deployment is not too shabby either.
I’m aware of pandas and the rest in Python, but for some reason Julia feels a lot more “ergonomic” for my use case.
dleather
Financial economist and Julia aficionado here looking to transition to academia to industry if you're ever hiring. If you have any feedback /tips it would be much appreciated. Email and website in bio.
cpfiffer
Also a financial economist and Julia lover -- it's a fantastic tool for finance IMO
Tarrosion
I can confirm this has been true since at least v0.4!
catgary
I’m kind of sad, because I think I have completed the journey from advocating for Julia at my first job to becoming a full-blown hater. I have complaints about the type system and the general failure of a good autograd library to emerge as the standard/default.
But my main complaint about Julia is its general approach to memory management. You are encouraged to think about how memory is being allocated e.g. by using StaticArrays for smaller, immutable arrays, or pre-allocating the arrays and operating on them in-place. But you don’t actually have control over allocations, and it’s easy for some type instability to cause unnecessary allocations - even after you stick a bunch of type annotations which should give the compiler sufficient information to force type stability or at least crash/fail to type check.
At this point I think people are better off investing their time/efforts into Rust for computationally heavy workloads on the CPU (polars for data frame stuff, ndarray for numpy functionality, Enzyme for autodiff) and using torch/jax for GPU-centric work (also, it’s significantly easier to write python bindings for rust libraries than C++ or Julia libs).
kbarros
Julia is my tool of choice for writing numerical code where performance is critical. I work in computational physics, and have found Julia and its ecosystem to be far nicer than Rust in this space.
It's true that accidental dynamic behaviors are a real concern and can be a performance killer. Fortunately, the language has nice tooling. In VSCode, I often use the visual profiling tool `@profview` to get a flame graph. Anything dynamic gets highlighted in red, and is quick to diagnose. There also exist nice static analysis packages like JET.jl. During development, one can use `report_opt` to statically rule out accidental dynamical behaviors. Such checks can also be incorporated into a project's unit tests. In practice, it's not much of an issue for me anymore. But to be fair, there is a big learning curve for new Julia users. See, e.g., https://docs.julialang.org/en/v1/manual/performance-tips/
catgary
Those tools didn’t actually exist the year or so I was writing Julia professionally (and that was only 3 years ago) so it’s nice to see the language coming along.
At the same time, I would expect the Rust ecosystem to overtake Julia’s in that domain in the next couple of years. Polars is already nicer than pandas, I’ve seen a some promising work on numpy-style tensor libraries, and I’m pretty impressed by the progress with getting enzyme integrated into Rust (I could never make it work with Julia). Here’s a nice example repo I saw recently:
thetwentyone
Another tool in this regard is https://github.com/JuliaLang/AllocCheck.jl, "a Julia package that statically checks if a function call may allocate by analyzing the generated LLVM IR of it and its callees using LLVM.jl and GPUCompiler.jl"
thunkingdeep
Not the person you’re respond to, but Rust is never going to have a REPL and is likely to never compile very quickly. For a lot of numerical and scientific use cases that’s a fundamentally restraining factor. WRT performance, you’re ofc correct but that’s not always as paramount as it may seem if you need to tweak the data dozens or hundreds or thousands of time. In that case, having to wait more than say 5 seconds or so is prohibitively annoying.
I think something in between Zig and Rust will emerge someday as a sort of optimal compromise between compile speed, safety, and programmability wrt memory and performance tradeoffs.
npalli
You were going through some good points until you hit Rust and then the entire argument seemed suspect. Rust has none of the interactivity of the REPL or dynamism. Complete pain for a normal scientist programmer. Given that security is not a burning need in this area, even C++ is superior as the entire computing infra is very much C++ based.
catgary
Oh, I would advocate for writing high quality libraries/components in Rust and then using Maturin to generate Python bindings for interactivity, [1] is an example of that workflow and it looks quite smooth.
DNF2
Then you are back to the "two language problem". I'm sure that's not a problem for you and for many others, but there is a reason it has its own, widely known name. It really is a problem for people who are mostly not software developers, but instead engineers or researchers.
Imustaskforhelp
https://github.com/evcxr/evcxr/blob/main/evcxr_repl/README.m...
There is this if we really want a rust repl
kjrfghslkdjfl
> But my main complaint about Julia is its general approach to memory management.
I'm not a full-blown hater, but I have problems with that as well. Specifically, you have no control about it whatsoever, you're just promised that "if you do things right, it'll be amazing". And it is! The problem is that any tiny minuscule mistake causes catastrophic failure of performance due to allocations. Since the good performance depends on type stability, and type stability propagates, any mistake anywhere will propagate everywhere. Think: if a variable becomes type unstable due to a programmer mistake, any function that consumes it generally might become type unstable as well, and any function that consumes the output of that function as well, etc. The upshot is that this forces you to think more carefully about your types and data structures. Programming in Julia extensively has made me a better programmer. I'm not a C++ expert, but I believe that in C++ these kind of mistakes always end up being localized.
DNF2
That is not really correct. Type instabilities tend to disappear at function boundaries, which is one of the reasons why using functions is so heavily promoted in Julia, it helps keep type instabilites 'localized'.
catgary
I was a Haskell programmer in grad school, and Julia was how I learned “oh, some times the programmer does know better than the type system/compiler”. I think the way they approached multiple dispatch in the language (and the resulting allocations due to type instability) is really the original sin of the language, and I just don’t think it can be fixed, so I can’t help but feel any effort to improve Julia is a waste of time.
cardanome
Small static binaries could be an killer-feature for Julia.
While I always had an eye on Julia and it is a lovely language, especially the multiply dispatch and general lispy-ness in spirit, I never really had a good use case for it.
Python always was the better choice due to the stronger ecosystem and faster launch time. Rarely did I need the extra performance Julia offered. But Python is annoying to deploy so Julia starts to seem tempting, at least for smaller projects where you don't want to do a complex docker setup.
Julia is for sure prioritizing the right things.
Imustaskforhelp
You can use uv with uv run --script main.py and give it some requirements like numpy as well with --with numpy
You can also upload it to pypi for example with name let's say for example heytest and then just do uvx heytest and it would just work.
Uv is great , uv is love.
dleather
I started using UV. My first thought was "Great, Python works more like Julia now"
Imustaskforhelp
I have also been able to install uv easily / even create a static binary of uv
currymj
The article describes Julia as "a general-purpose programming language aimed at science, engineering, and related arenas of technical computing".
I think this is about right. If you are going to solve differential equations or run some numerical weather codes or solve an integer program or simulate an electrical circuit, it would be a good idea to consider Julia rather than reaching for Fortran, C++, or MATLAB. It is a much nicer language and still very fast.
My experience is that people who want to use Julia for data science and especially for deep learning, usually find that you can get the job done, but often it's easier to use PyTorch or Jax and just accept the limitations of Python.
jarbus
Julia is still such an absolute joy to use. I have a dream that somehow, Torch.jl, which attempts to provide a julia interface to pytorch, will magically take off. Python is ok, but julia is just so fast, fun, and easy to scale.
systems
Julia is not a scripting language, or in other word, Julia scoping rule are not scripting friendly
currently if in a file i have
local s = 0
for i = 1:10
t = s + i
s = t
println("$s")
end
println("$s")
ERROR: LoadError: UndefVarError: s not definedeigenspace
Note if you're doing this in the global scope, the local declaration on `s` will cause it to functionally not exist.
You can write this as either
s = 0
for i ∈ 1:10
t = s + i
global s = t
println("$s")
end
println("$s")
let s = 0
for i ∈ 1:10
t = s + i
s = t
println("$s")
end
println("$s")
endforgotpwd16
Another alternative is putting the code in a function and calling that function.
function test()
s = 0
for i = 1:10
t = s + i
s = t
println("$s")
end
println("$s")
end
test()
affinepplan
while of course you're correct on a technical level, I do tend to agree with the statement that Julia is not particularly friendly to bash-scripting-heavy workflows
eigenspace
Meh. For my purposes, Julia strikes a quite nice balance of fussiness versus permissiveness.
Sure, there's things where I find it too fussy and I wish it'd go "yeah yeah I get what you mean" and there's also things where I find myself thinking "man, I really wish you had complained to me about that, it was clearly a bug / bad style!"
But all-in-all, I find it's a language that mostly gets out of my way and lets me code, which I find very useful and nice for scripting.
I think if one finds things like scoping rules annoying for scripting, that might just come down to just knowing and being more familiar with another language that has different rules.
sgt101
sorry - why would you expect a local declaration to be in scope for other code?
s is defined in the statement and then is undefined out of that statement because there is no other scope for it - it's at top level. just write s =0 and all is well as s exist in the global scope where your code lives as well...
forgotpwd16
OP should've written just `s=0`, yes. Because showcases their complaint better since, although works in REPL, also fails when in file. (It's also the example given in docs[1].) Now, it looks like they tried to make it fail.
[1]: https://docs.julialang.org/en/v1/manual/variables-and-scopin...
akkad33
Shouldn't the language flag that using local in that scope is an error then?
adgjlsfhk1
https://github.com/JuliaLang/julia/issues/57483 yes, yes it should.
sgt101
Yes - the error reporting has always been one of my complaints about Julia (although it's hugely improved). I think that the Julia dev community are really really interested in cool language features and optimisations but much less empathic with numpties like me that can't understand why their code crashes. More work on this would be much better.
Although it's a bit of an challenge to get the interpreter to understand which thing that's out of scope should be in scope for the code to work (because it's all out of scope..)
leephillips
Right, but did anyone say the Julia was a scripting language?
dleather
I prefer to code in Julia almost always. It really is such a wonderful experience. Now if only I could land an industry job with my Julia skills...
blindseer
I'm happy to see static compilation in the works! If static compilation takes off, and people are able to build Python packages that are secretly just compiled Julia code, I can see a world where more people opt to using Julia over C or C++. Though writing Rust libraries that work as Python packages is a joy and Julia would still have competition.
Julia biggest problem at the moment is growth. Julia has suffered from not having exponential growth, and has either maintained a small linear growth or has fallen in popularity. Search online on YouTube for tutorials, on Twitch for WatchPeopleCode, or on GitHub for benchmarks; and Julia is not even in the room where the conversation is happening - there just isn't any mindshare.
And for good reason. There are so many ergonomic challenges when using Julia in a large codebase and in a large team. Julia has no formal interfaces, LSP suggestions that are often just wrong, and no option types. This just makes writing Julia code a drag. And it makes it quite difficult to advocate to developers experienced with languages that offer these features.
Additionally, the core conceit pushed by Julia advocates is that the language is fast. This is true in controlled benchmarks but in real-world scenarios and in practice it is a real pain to write and maintain code that is fast for high velocity teams because it requires a lot of discipline and a strong understanding of memory allocation and assumptions the Julia can and cannot make. You can write code that is blazingly fast, and then you make a change somewhere else in your program and suddenly your code crawls to a halt. We've had test code that goes from taking 10 minutes to run to over 2 hours because of type instability in a single line of code. Finding this was non-trivial. For reference, if this were uncaught our production version would have gone from 8 hours to 4 days.
The lack of growth really hurts the language. Search for pretty much any topic under the sun and you'll find a Python package and possibly even a Rust crate. In Julia you are usually writing one from scratch. Packages are essential to data processing are contributor strained. If you have a somewhat unpopular open source code code you rely on that doesn't work quite work the way you want it to, you might think I'll just submit a PR but it can languish for months to a year.
The Julia community needs to look at what programming languages are offering that Julia developers want and will benefit from. The software world changing very quickly and Julia needs to change too to keep up.
kbarros
> We've had test code that goes from taking 10 minutes to run to over 2 hours because of type instability in a single line of code.
For those who might not be familiar, tooling can sometimes help a lot here. The ProfileView.jl package (or just calling the @profview macro in VSCode) will show an interactive flamegraph. Type instabilities are highlighted in red and memory allocations are highlighted in yellow. This will help to identify the exact line where the type instability or allocation occurs. Also, to prevent regressions, I really like using the JET.jl static analysis package in my unit tests.
patrick451
If they are so easy to identify, why not just make it a JIT error. Manually inspecting all of this sounds awful. I'd rather my compiler just do it for me.
kbarros
Dynamic behaviors can be a nice default for all the spots where performance is not critical. So Julia lets you code like Python in places where performance doesn't matter, and then code like C++ or Fortran in places where it does.
pjmlp
For almost 25 years, Python also didn't had any exponential growth it has been slow and steady.
jshen
I really like the direction of Julia and its managing of trade-offs. I hope it gains more traction.
LeroyRaz
How does Julia compare to Python with Jax?
I just love pythons ease of use, and the few times I've tried Julia it has felt clanky in comparison, and then Jax just makes everything so amazingly fast.
currymj
for problems where Jax works well, in theory there could be a great Julia equivalent, but in practice one has not emerged. instead every 3 or so years, a brilliant solo researcher writes an even better autograd library which is ultimately abandoned.
the language is at its best in other areas. for a few things (differential equations, mathematical optimization) it is actually state of the art.
catgary
Even as a full-blown Julia hater, it exists at a really nice level of abstraction for working in differential equations and Chris Rackaukas has written some fantastic software in that area. It can be a great tool for some domains (ODEs, statistical modeling/visualization) and I can see why Pumas uses the language.
I don’t think that Julia is a great tool to replace the complicated physics simulation code that would have traditionally been written in C++/Fortran, I think for that you’ll generally be better off using JAX/Rust/C++.
DNF2
"Clanky"? That is a word I would use when comparing Julia and Python, but I would reverse the roles. I mean, python works well, and has almost everything, but it really feels, well, clanky.
thetwentyone
I'm currently working on a re-write of a small model from JAX to Julia and finding the Julia code so much easier to write, it's more concise, and find the debugging tools easier to work with in Julia.
patagurbon
Check out Lux.jl and Reactant.jl for a modern attempt to create a Julia library with Jax’s XLA compiler backend!
acmj
> The next major Julia release, 1.12, is likely to appear in mid-2025. It will finally include the ability to generate static binaries of a reasonable size, appropriate for distribution.
I would be thrilled if Julia had this in early days, but now it is a little too late. I have jumped the boat.
akkad33
What do you use instead
kruxigt
[dead]
null
I know I know I know that this same refrain has probably been said nonstop since Julia's first day after launch, but the language really is getting noticeably better every single release. Each of the releases I've been around for (so 1.8-1.11, and now on nightly, 1.12) is adding meaningful new features, performance improvements, and addressing many of the oldest and sharpest pain points
the `--trim` feature coming in 1.12 will still be considered experimental, and if I had to take a guess won't be considered "stable" until 1.14 or so, but I'm really excited to see it picked up around the ecosystem!