"We're building a new static type checker for Python"
142 comments
·January 29, 2025theLiminator
IshKebab
I agree, Mypy is awful. Pyright is very good but also quite slow, and its Node requirement is awkward in some cases (e.g. pre-commit).
A fast Rust based type checker would be amazing!
Noumenon72
I'd appreciate knowing what's wrong with MyPy that this project will fix. Check time? Bad rules? Lack of features?
IshKebab
I found performance not that different to Pyright. The major difference is quality and correctness. Mypy is full of weird bugs and insane typing holes.
I found cases where it would even treat `foo: SomeType` and `foo # type: SomeType` differently!
I tried to fix that one but looking into the code lowered my impression of it even further. It's a complete mess. It's not at all a surprise that it gets so many things wrong.
Overall Mypy is like kind of like a type checker written by people who've only ever seen linters before. It checks some types but it's kind of wooly and heuristic and optional.
Pyright is a type checker written by someone who knows what they are doing. It is mostly sound, doesn't just say "eh we won't check that" half the time, and has barely any bugs.
Seriously check the closed/open issues on Github - there's a touch of "I disagree so I'm closing that" but only a touch. It mostly has so few open issues because the main author is a machine.
The only real problems with it are performance (it's ok but definitely could be better), and the slightly annoying dependence on Node.
harrall
This is a good summary: https://github.com/microsoft/pyright/blob/main/docs/mypy-com...
But without that, I always felt like I was actively fighting mypy. It seemed like it was written for a totally different language than Python.
Compared to another more modern type system like TypeScript, sometimes you don't explicitly type something and yet TypeScript usually does exactly what you expect.
arthur-st
MyPy's rules are reference-grade, being as close to an official spec as we get until the Typing Council is done establishing their moat.
To understand shortcomings of MyPy, I strongly suggest reading pyright's documentation for how they compare: https://github.com/microsoft/pyright/blob/main/docs/mypy-com...
Quoting the pertinent part:
> Pyright was designed with performance in mind. It is not unusual for pyright to be 3x to 5x faster than mypy when type checking large code bases. Some of its design decisions were motivated by this goal.
> Pyright was also designed to be used as the foundation for a Python language server. Language servers provide interactive programming features such as completion suggestions, function signature help, type information on hover, semantic-aware search, semantic-aware renaming, semantic token coloring, refactoring tools, etc. For a good user experience, these features require highly responsive type evaluation performance during interactive code modification. They also require type evaluation to work on code that is incomplete and contains syntax errors.
> To achieve these design goals, pyright is implemented as a “lazy” or “just-in-time” type evaluator. Rather than analyzing all code in a module from top to bottom, it is able to evaluate the type of an arbitrary identifier anywhere within a module. If the type of that identifier depends on the types of other expressions or symbols, pyright recursively evaluates those in turn until it has enough information to determine the type of the target identifier. By comparison, mypy uses a more traditional multi-pass architecture where semantic analysis is performed multiple times on a module from the top to the bottom until all types converge.
> Pyright implements its own parser, which recovers gracefully from syntax errors and continues parsing the remainder of the source file. By comparison, mypy uses the parser built in to the Python interpreter, and it does not support recovery after a syntax error. This also means that when you run mypy on an older version of Python, it cannot support newer language features that require grammar changes.
Astral's type checker seems to an exercise in speeding up Pyright's approach to designing a type checker, and removing the Node dependency from it.
tcdent
yeah, 'awful' is just the OP being dramatic, though I am looking forward to Astral's contribution
zeotroph
There is third, pytype (by google), which I found pretty good but it rarely gets mentioned. However, like the others it is slow, so I hope this one is fast and supports all the pytype features (especially being able to type-check un-annotated code).
https://github.com/google/pytype?tab=readme-ov-file#pytype--...
dcreager
> it uses salsa from rust for incremental type checking
This is true! We're also contributing salsa features upstream where we can, e.g. https://github.com/salsa-rs/salsa/pull/603
sgarland
I've avoided Pyright explicitly for that reason. I have a severe dislike of Node, and don't want it installed on my computer for any reason. I'm aware that this is a self-limiting position.
Anyway, agreed that this is very exciting news. Poetry was great, and then I found uv. It's... wow. It's really good.
pinoy420
Node is fine. This is very much old man shouts at cloud. Pyright is really quick. Node is fast enough for a majority of uses.
sgarland
I don’t dislike it because it’s not fast enough; I dislike it because I associate with the rise of tech influencers, who are at best charlatans, and who at worst have helped to usher in an Eternal September far worse than Web2.0 could ever have dreamt of doing.
rednafi
Not old and not yelling at anything. I don’t have any qualms about installing tools that depend on Node, as I use VSCode and therefore Pyright all the time.
That said, Node is awful as a backend language, and speed has nothing to do with it. I will write Go, Python, Rust, or anything else because of how those languages have been designed. JS doesn’t belong in my backend—YMMV.
insane_dreamer
I've found pyright to be limited in its ability to infer types, especially compared to Pycharm, which can resolve class inherence across multiple modules quite well. I noticed this when trying to use Zed instead of Pycharm.
jasonpeacock
It's hard to find details...apparently it's code named "red knot" (or "red_knot").
Here's the github issues filter linked in the screenshot:
https://github.com/astral-sh/ruff/labels/red-knot
And the best answer/description of what the type checker will be:
SandmanDP
> apparently it's code named "red knot" (or "red_knot")
Missed opportunity to call it typy in my opinion
smpretzer
Another useful issue filter: https://github.com/astral-sh/ruff/milestone/20
shlomo_z
I am really excited.
Ruff is amazing, uv literally fixed all the python packaging issues we have, and now a type checker!!
Can't wait to see it!
kittikitti
I started learning Python when it was barely at version 1 precisely because there was no real static type checking. If I wanted that, I would just write the program in C or its variants. I wasn't excited about type hints because developers became super judgy if there weren't any. I just want to know why you're using Python if you want stronger typing and feel the need to change the entire language instead of using another one yourself.
cobertos
Progressive typing gives you the option to type check when and if you want or need it. You don't _have_ to type. If things get really hairy but you know it's right, you can focus on the code instead of the typing and efficiently complete your task.
There's also something fundamental about taking a dynamic language like Python or JavaScript and adding typing versus taking a static language and adding dynamic typing (e.g.`auto` in C#). The dynamic language allows modifications that are _really_ convenient if not a little hacky that you just can't easily express without big refactors in static codebases. Things like tagging objects with properties, making quick anonymous structs (before that became a thing in modern languages) so you can return tuples of values, other constructs like dynamic functions or whatever. Progressive typing is just so much more expressive
agluszak
> taking a static language and adding dynamic typing (e.g.`auto` in C#)
Assuming you meant C++ - no, auto is not a even a tiny bit dynamic. It's still fully static typing, but with type inference.
The compiler will still prevent you from calling a method which doesn't exist.
cobertos
Ah yes, my mistake. `auto` is C++. I was thinking of `dynamic` and `var` in C#.
jaredklewis
Probably because a lot of programmers don't get to choose the language they are using. They have a job developing some existing app that is already written in Python, but they wish it wasn't Python. They don't have an option to rewrite the whole codebase in Rust or whatever, so the best they can do is advocate for Python to evolve to be more like languages they do like.
gt0
I think this is exactly it. My gut instinct is just "Why the obsession with adding static types to Python, just use a language that already has good static types".
As you say though, not everybody gets to choose their language, and I certainly wouldn't choose Python if I could possibly help it, so if we can add good static types to Python, it's still not a great language, but it's much improved by static types.
baq
I’m a Pythonista of two decades who is working with typescript (on the backend) for the past two years and absolutely loving it. I don’t like the js part, but it doesn’t matter too much except when you’re working with numbers, which are a bit insane.
If anyone built a typescript layer on top of Python, I’d cheer that effort. If that new thing from the article will be that, I’m stoked.
harel
But most people in our profession get to choose their job, don't they? Why would a Python project hire a rust programmer? Why would someone who hates Python work on a Python project? I feel the same way. Sporadic use of type hints is nice, and I use it sometimes for reference, like a very succinct comment. But if I want a typed language, Go, Dart, C or even Java are right there.
mbo
Because sometimes people choose their jobs prioritizing pay, location, title, company, project, role, coworkers, etc. and the language they will primarily use (or even use on a secondary basis) is a lower priority concern.
In my case, I have, in the pursuit of my engineering goals, had to dig into a Python component that I did not normally work on or intend to work on. I am thankful it had type hints.
kaashif
> Why would someone who hates Python work on a Python project?
I hate every language! For different reasons. But I still like programming.
jaredklewis
Even if we have some choice of jobs, your model only works if programmers have language as their only or most important requirement when job searching. There are lots of factors that might make a job more or less desirable aside from programming language, like location, salary, team, business domain, and so on.
Seems pretty plausible that someone can dislike python and still like a job because it checks other boxes.
Kerrick
> Why would a Python project hire a rust programmer?
> But most people in our profession get to choose their job, don't they?
Let's say someone has only worked in Python professionally but dislikes it. They spent nights & weekends learning Rust. Now they want a Rust job. Because a Rust project would not hire a Python programmer, this person who hates Python is stuck working on Python projects.
It's attitudes exactly like "Why would a Python project hire a Rust programmer?" that keep people stuck in languages they don't hate. Those attitudes are the reason a senior software engineer with over a decade in the field can't switch tech stacks. Without that attitude, job postings would be written for programmers, not python programmers, and the job market would be better for it.
rexpop
> most people in our profession get to choose their job
Choose, yes, but from a finite set.
> Men make their own history, but they do not make it as they please; they do not make it under self-selected circumstances, but under circumstances existing already, given and transmitted from the past. The tradition of all dead generations weighs like a nightmare on the brains of the living.
— Karl Marx, "The 18th Brumaire of Louis Bonaparte"
baq
C is basically untyped but makes you type out the types. This is not what modern type checking is; it’s usually the other way around - you specify types where you want them checked because it gives you confidence that the program is correct.
lelanthran
> C is basically untyped but makes you type out the types. T
This is grossly incorrect.
taco_emoji
I feel like if you think about it for 20 seconds you can come up with a lot of good reasons, so I question your motivation in posting this.
falcor84
I personally just love Python type hinting for testing purposes. So I can avoid types while prototyping or playing with architecture, but once I have something I'm happy with, I find that adding types allows me to gain confidence in the program while having fewer tests, and generally making it easier to refactor further.
And full typing is also a great first step for rewriting in another language.
mixmastamyk
“Never give up, and good luck will find you.”
ijustlovemath
Python's got a pretty incredible library ecosystem, and it's easy to get a project up and running quickly. The big problems with it start to appear as you cross the 10k+ sloc / 3+ dev thresholds; if things are changing quickly, the iteration velocity the duck typing provides can quickly become a big problem.
Type hints fix this by allowing you to ensure your contracts are upheld while your team continues to move. If you don't find yourself needing them, you probably don't have a team that's trying to ship and iterate super quickly, or your project is still at the one-off script scale.
mtalantikite
The entire ML/AI ecosystem is in Python for the most part these days, so if anyone wants to get involved it means needing to touch Python to some degree. So, many developers might get handed projects that require working in Python, but would really prefer not to give up all the nice things that come from a modern statically typed language.
Personally, I'm one of those people that got converted over to statically typed languages, as I spent a large part of my early career in startups using Ruby. I don't think I could ever go back after using Go, TypeScript, and Rust over the past decade or so.
tcdent
typed python is amazing for building understandable applications. I will never go back.
you can obviously still use it untyped to hack together scripts, but I still find myself leaving annotations for clarity.
I've been using it since the mid 2.x days and find the style we were writing back then to be incredibly dated.
dcreager
Bluesky cross-post for those who don't want to read the thread on X: https://bsky.app/profile/crmarsh.com/post/3lgvhzdfrps26
timeon
As someone without X or Bluesky, this link shows whole thread while X does not.
uneekname
Yeah, this is a better link. Can we swap it out?
StackTopherFlow
Doing the lord's work
mixmastamyk
I don't care about politics, but don't load javascript from just anyone. Not to read hundreds of characters, surely. Neither site works without it.
zzzeek
you care extremely deeply about politics such that they align so closely with your personal comfort that you need not have to notice it
esafak
This is all very well but how are going to make money? They have taken VC investments.
simonw
There's a bit of an answer to that from Charlie Marsh here: https://hachyderm.io/@charliermarsh/113103579845931079
> What I want to do is build software that vertically integrates with our open source tools, and sell that software to companies that are already using Ruff, uv, etc. Alternatives to things that companies already pay for today.
> An example of what this might look like (we may not do this, but it's helpful to have a concrete example of the strategy) would be something like an enterprise-focused private package registry. A lot of big companies use uv. We spend time talking to them. They all spend money on private package registries, and have issues with them. We could build a private registry that integrates well with uv, and sell it to those companies. [...]
TheTaytay
Ah, that makes a lot of sense. Sounds like they have patient investors. They are doing amazing work.
claytonjy
the packaging one is an odd example because it’s hard to imagine a company that both needs an enterprise packaging solution external to their cloud provider, and is python only
i have a similar concern for the Pydantic folks commercializing via observability platform; how big is the market for python-only observability?
i think both cos will either look for money elsewhere, or find themselves having to deal with at least a few other language ecosystems. Even python+typescript would 10x the TAM or more
giancarlostoro
Honestly, they should do that, it would force their competitors to actually improve their own, and if they don't well, there you go, really great funding your way.
_diyar
If they open source it, who cares? Obviously I would like them to continue maintaining their stuff, and I am sure they are imaginative enough to find some source of income.
esafak
It must be nice to receive VC funding without knowing how you are going to repay it!
maccard
Honest question - have you worked with VC’s much? That’s pretty much the MO. Get a snippet of an idea that has the potential to scale to unicorn levels, make sure you have a super coarse idea of how to monetise it, and then shoot and see how far you get.
barnabee
VC funded companies (almost[0]) never repay it.
They only try to create enough value that the VC can one day sell their stake for more than they paid for it. The key is convincing the VC not that that is particularly likely, but that if it happens there's a chance it'll be a lot more than they paid for it.
[0] occasionally a company gives money back to investors when it fails
joshdavham
These questions are often hard to answer for people unfamiliar with how VC works, but mostly because VC is not intuitive from a 'normal business' perspective.
Essentially, VC's know very that what they're funding is risky and may go completely caput. However the VC model explicitly aims to fund companies that have the potential of becoming huge and becoming very profitable.
Both the founders and VC's behind Astral are not stupid, they know that it could fail or only have moderate success... However, if they succeed in building the next generation go-to ecosystem for python development (which is arguably the world's most popular programming language), we could imagine a future where they spin off some product and it does incredibly well!
benatkin
A Microsoft acquisition? Could help them to keep their open source trickery going. https://ghuntley.com/fracture/
ehutch79
My big question is django support.
Django has a lot of magic that makes type checking more difficult than it should be.
I'm not really expecting anything, even just the speed bump will be nice.
aitchnyu
Will django-stubs work with other typecheckers?
Tangential, I'm waiting for Edgedb to become next SQL so all ORM moats like Django's are drained.
aleksanb
Agree with this.
All type checkers other than mypy (e.g. pyright, intellij) have ignored the level of plugin support necessary to make django work well, and so they are DOA for any large existing django codebase. Unless ruff decides to support such a dynamic interface as mypy's, it'll fare no better.
We use mypy with [django-stubs](https://github.com/typeddjango/django-stubs) which works ok nowadays.
There was an effort to create a typechecking plugin interface for dataclass-style transforms for python type checkers, but what was merged was so lacking that one couldn't even make something close to django-stubs with it.
claytonjy
is Django working towards using native types instead of all they stuff they invented before type hints existed?
rtpg
I am not part of the DSF or anything but I can say with a good amount of confidence that the answer to this is “No”.
Moving to a Pydantic-style model would be a whole different ballgame. Hell of a lot of work, and meanwhile we can’t even really get to “let’s have type hints in some of the Django infernals”.
But! I do think someone who wants to have fun could try to build a type hint based Model subclass as a third party lib! There’s a looooot of stuff to resolve in that world but for any proposal like that the general vibe is “prove the feasibility with a third party package first”.
c-fe
I think there is potential for a new python web framework with batteries included (like django) but using newer paradigms (typing, …). Fastapi itself is a nice concept but its a bit of a mess (the main author just keeps closing bug issues and adds somewhat useless things (just check the git commit log)), and also it requires db tooling (sqlalchemy) and other things
adsharma
It should be much easier to type check this:
https://adsharma.github.io/fquery-meets-sqlmodel/
https://github.com/adsharma/fastapi-shopping/blob/main/model...
agumonkey
Applaud the effort, but it's weird seeing python (and es6) living more and more on top of native static analyzers/transpilers .. at which point will they become a frontend syntax for ocaml or julia ? :p
zem
as someone working in this space, the idea is that most python code is implicitly statically typed anyway, in that your functions are only ever meant to be called with specific types for their arguments, and only return objects of a specific type. all that static type analysis does is to make that explicit via type annotations, and then verify that you have indeed used types correctly in your code. static compilers then say "well, if we know the static types of everything in this function, we can compile it to more efficient code, skipping all the dynamic lookups and runtime type checks and dispatching". typically there is an escape hatch that will fall back to regular python behaviour if your code is indeed dynamic (e.g. reflecting on a database schema to generate models at runtime), but losing the benefits of static analysis and optimisation.
if you would like to read more on the topic the keyword is "gradual typing"; in particular you might be interested in looking up stanza, a language designed around gradual types from the outset rather than retrofitting them on top of a dynamically typed language.
agumonkey
this https://lbstanza.org/index.html ?
what are places to discuss gradual typing these days ? I stopped looking a few years ago (after clojure and scheme started projects on that topic)
zem
yep, that's the one! I don't know if there's any one place to discuss gradual typing. typed racket is probably the language with the largest academic community behind it, so the racket discourse would likely have some good discussions, though I haven't been keeping up with it recently.
snovymgodym
> at which point will they become a frontend syntax for ocaml or julia ?
At no point.
What probably will happen is that both the JS and the Python reference implementations will have first class support for type systems which will allow them to generate optimized native code for the typed sections of your code while keeping the untyped sections as bytecode or as higher level compiled code (i.e. a bunch of runtime function calls dealing with tagged union objects).
intalentive
I would use a statically-typed Python that you can compile into a binary for better performance. Step 1: Develop fast in regular Python. Step 2: Add type annotations and build.
alias301
You're in luck!
wk_end
AFAICT this is dead (no updates in four years) and "alpha software...only recommended for production use cases with careful testing, and if you are willing to contribute fixes or to work around issues you will encounter." Unfortunate, because it's pretty cool.
iso8859-1
the repo says the code is in the mypy repo and it looks like it is maintained in there. so I think it has had updates the last four years?
Havoc
Wow those are pretty chunky improvements over vanilla
fuzztester
Mojo from Modular?
I've not checked it out enough yet, and it's early days for it anyway.
Just saying it may be worth keeping an eye on.
I've seen some videos of Chris Lattner talking about it.
pbronez
Looks interesting. Their “magic” package manager scares me though. Not excited to curl | bash something with strong opinions about my local python setup
curl -ssL https://magic.modular.com/deb11637-c8c3-4f76-9195-85af25ef2e... | bash
nimish
astral doing what the endless layers of admin/people in the PSF can't: solve actual problems python devs face
screye
At the risk of sounding like a fanatic, I bet it will be great.
Everything Astral have put out has been a step improvement to my python dev flow. Until they disappoint me, I'm staying on this hype train.
Very excited for this, looks like it's built on top of pretty solid technical foundations (iirc, it uses salsa from rust for incremental type checking). I've found mypy pretty terrible. Pyright is okay (but requires node).
Ruff truly can become one tool to rule them all.