Finding Signal in the Noise: Machine Learning and the Markets (Jane Street)

JS absolutely does filter on school though.

Might be different with a SUNY PhD or another trading firm on the resume.

I don't think they'd ever call someone like me back, not that I'd ever be able to pass a coding round with them.

[dead]

ever meet any such engineers from the bottom 20% of the world population? just some food for thought :)

I genuinely don't think people that went to Exeter and Harvard (or MIT or Stanford or Penn or other JS feeder schools) see the rest of us as human unless there's another common factor like being employed by Jane Street and having those quirky hobbies the rest of them do - and then they rationalize it away by saying that the exceptions didn't apply to top schools, or were admitted but didn't attend for "financial reasons".

People like me are "NPCs" in their parlance.

I went to an "elite" school, but I worked at one HFT where the local team of 5 people had only me with a background from a similar school, and another HFT where I was one of three on my immediate team of ~12 engineers. At one point, one of my colleagues at the second firm asked me why I was there as opposed to the alternatives of (1) a company with more of a focus on the elite and cultivating power or (2) being an artist supported by my well-off parents.

Obviously Michigan, UCB, UDub aren’t in the same category, especially with prior trading firm experience.

I’m intrigued as well.. My experience is notebooks struggle as a format for production code. We encourage people who work heavily in notebooks to use them for exploratory work, but choose other tools when it comes time to ship.

When you are exploring something, experimenting, showing.. it’s great; train-of-thought structure, APIs like Pandas optimised for writing and terseness etc.

But when you have a piece of code that will lose a million dollars a minute if someone ships a bug, and which will be maintained by many engineers over many years, then you really want a format that’s optimised for long-term maintenance, incremental change, testability, and APIs optimised for readers.

My book, Effective Pandas 2, has many of them. There's a few conference talks of mine floating around on YouTube that also mention some.

Writing clean data code is one aspect. Filling in knowledge gaps is another. Covertly teaching software engineering best practices to folks who "aren't programmers" yet sit down and write code in Jupyter all day is another.

I should probably write a blog post or record a short video. (I just taught a week long course on this for a client last week.)

Why are you trying to test and compose notebooks? Notebooks don't really fit into that lifecycle of work, they're implicitly script-like code.

Any time a notebook needed to be tested, refactoring it into a module and testing that was the better choice.

Notebooks in my opinion are by far the best tool for interactive and exploratory data work. I've been using Jupyter/IPython for 10 years and it takes very little discipline to keep them clean and clear. I've never bothered trying to deploy them in any meaningful way.

The quirks with version control are annoying, that's one reason I've switchex to marimo in the last few months.

Tyler Cowen, master of the ‘cum hoc ergo propter hoc’ fallacy. He frequently mistakes the occurrence of phenomena for causative proof of that phenomena. He particularly exhibits this inclination when his confidence rises amid scant data (like the rest of us).

This error seems to be a particularly common (and often lauded!) trait among those who work in high-conjecture low-evidence fields (eg, economics). The prominent thinkers become skillful at deploying this fallacy: “see, it’s there, therefore <insert-personal-belief> is certainly the cause!”, using their credentials and esteem to mask the error. Listeners think, “well he’s a smart, respected guy,” and nod along despite the missing logical link.

I greatly appreciate Cowen’s podcast, and I definitely respect him as a thinker and inquisitor – so I don’t mean to discard his work or opinions (in fact, I appreciate his occasional brashness because it exposes the underlying thought/principle). However, many of his aggressive-yet-speculative statements (like the one you roughly quoted) are best received with an understanding of the error.

> He replied with, roughly, "Those of you who work here probably couldn't do anything else other than perhaps math research. Arguably, working here is the economically efficient use of your time."

Complete garbage. The same way that Jane Street hires smart people that don't know anything about trading and those people contribute, the same would be true if there was money in curing cancer.

To build on / extend on this - quants / finance folks need to cultivate an image of only taking the very brightest, to justify the shit working conditions (even if pay is often decent) but honestly the brightest tend not to apply. Working in those environments is neither rewarding nor stimulating.

General intelligence is overstated sometimes, but it is a thing. Someone who is smart enough to work for Jane Street probably could at least be an intelligence analyst or software developer at the NSA contributing to national security. (Jim Simmons literally was a code breaker during the Vietnam War)

I don't think there's a gene for playing esoteric minigames on the options market while you literally suck at everything else

I plus-one the last paragraph, well explained.

Do you have a link to this talk by any chance?

Oof. Interesting take.

There is an extreme case of diminishing returns at play here, and unfortunately the amount of money that flows through these markets is so incredible that it becomes worth it (more than worth it) to commission the top minds of society to push the limit as close to 1 as absolutely possible.

Derivatives have great value to industry. Derivatives that require the fuel of 15 math Phds to lock in fractions of a percentage pricing inefficiencies so their firm can pocket the difference do not have much value at all.

It's like employing Harvard Med surgeons to remove gold dust from gold market sidewalks, and calling it "gold market efficiency".

Everyone understands that there's a pricing utility to markets and market speculation. But consider that, like in all things, the free market has its 'biases': wouldn't you expect that something like trading, which directly produces profit with very little overhead and maximal ability to hedge against rush, would draw an inordinate amount of capital -- in the same way that things like infrastructure, sans government intervention, naturally repel it?

We have far, far too much of our economy sunk into a sector that fundamentally produces nothing of value, one that only shuffles value around and chooses to whom it will be allocated (coincidentally, often the people doing the allocating!).

Compare our economy and its woes to China. Do they spend nearly as much of their human capital figuring out how to get an extra bp on some statistical arb? Thethe origins of DeepSeek are illustrative in this regard.

Sure, some derivatives have some utility.

But, ok, Spread Networks spent $300m around 2010 to build a somewhat straighter glass fibre data connection between NY and Chicago. Other companies spent more to build a micro wave connection, since microwaves move faster through air than light through fibre.

Is that money really well spent? And I'm afraid a lot of what happens in finance is similarly private gain, without much social welfare increase.

> I like apples and I like being able to buy them any day of year, even out of season. The only reason I can do so is because there is an entire industry of people who manage the inventory, distribution, try to predict supply and demand, and take on a risk doing that.

This has been the case for well over 200 years and is not done by the same kind of people that modern finance/fintech employs.

Imagine the world 150 years ago and imagine were you'd allocate the smart people that's the highest paid these days. They'd be doing science and developing technology instead of doing finance. Why is finance that much more important now? (is it just because it makes rich people richer?)

> I like apples and I like being able to buy them any day of year, even out of season. The only reason I can do so is because there is an entire industry of people who manage the inventory, distribution, try to predict supply and demand, and take on a risk doing that.

In practice, if the availability of the apples you're eating is dependent on finance people, they're commodity grade, not specialty varieties.

I don't eat apples, but I'd rather have a good heirloom tomato from the farmer's market than the commodity grade stuff that's at the grocery store.

While it’s true that trading and financial markets provide value, the conversation around “wasted talent” isn’t necessarily about denying the value of these industries. It’s more about the broader context - are all forms of financial activity contributing to the greater good, or are some activities just enriching a small group of people without providing societal benefits?

Amazing link. Including the car mechanic joke.

Thank you for the link, it was very insightful.

Lots of things provide no value that people pay for.

The inverse is true too: lots of things provide value that no one pays for.

I find it shocking that anyone that programs would think this way considering how widespread and common open source tooling is. I don't know how you can get through your day without using OSS or even free websites like stack overflow.

My actual gripe is with where that value comes from, and what it's true cost to society is.

I'd argue that pulling these people out of moving society forward is just another form of externalized cost. The stanford guy who figured out how to halve the cost of solar in 2012 never got to realize it because $750k/yr to do stochastic modeling of cattle feed to (secretly) overcharge farmers for futures contracts was just too enticing.

>The alternative to highly technical, agile quantitative trading is fat middle men, wide spreads

No, this would be the alternative to securities and derivatives markets being electronic.

>If it didn’t provide value, nobody would pay for it.

The parent was remarking on the value provided to society, not the value provided to the firms.

they make the mistake of paying for it unlike those who know what to do https://www.investopedia.com/articles/investing/030916/buffe...

This is also bad. Huge amounts of our human capital are invested in fundamentally anti social enterprises. This sort of market failure seems to be endemic to our system as it stands today, and the only answers I've seen are "well, but that other bunch are even less useful!"

Well. Read about him all the way to the end, then.

The problem is that smart people are the ones being employed, not the ones getting rich.

I would prefer good people get rich and bad people don't.

I’m not sure that’s true either.

See for example “effective altruism”, which turns out to have been neither, but more of a self-deluded justification for insane greed, coupled with a god complex.