Ovld – Efficient and featureful multiple dispatch for Python

It's actually more like "just use Haskell".

Having written one of these[1] a decade ago and inflicting it (with the best of intentions) upon production code in anger, I can tell you this often leads to completely unmaintainable code. It is impossible to predict the effect of changing a method, tracing a method, debugging a method (where do I put the breakpoint??).

The code reads beautifully though. Pray you never have to change it.

The reason I say "just use haskell" instead of lisp is bc lisp generics suffer from this same problem.

Btw if anyone has a solution to this "generic/multidispatch maintainability in a dynamically typed language" problem I would love to hear it.

[1]: https://github.com/jjtolton/naga/blob/master/naga/tools.py

I am thinking more about Julia here - which I would use if Python was not that common in several communities.

I will forever think of this talk by Peter Siebel (author of Practical Common Lisp) whenever I hear about multiple dispatch: https://youtu.be/VeAdryYZ7ak?si=wY3RmcRnW96jxQQm

Mainly codegen. Most other libraries do something like `return dispatch_dictionary[tuple(map(type, args))](*args)`. Whereas ovld generates a specialized function to the set of actual signatures of the overloads (still with a dictionary, but you remove a surprising amount of overhead just from removing varargs). The code is registered in the linecache, so you can step into it with pdb if you want to look at it. After that I became a bit obsessive about pushing it further and further (because it's fun). So... when dependent types are involved, it will actually generate custom dispatch code, e.g. if you define an @ovld with x:Literal[0], for example, it will generate an `if x == 0` in the int dispatch path (and you can define custom codegens for new types like Regexp).

Regarding PyPy, I did run some benchmarks recently (I use pytest-benchmark). The advantage over other libraries remains and the magnitude is similar, but when I compare with custom if/isinstance code, that code is optimized a lot more aggressively and it gains a significant edge (let's say ~5x). Now, since I'm into codegen, part of me feels like meeting the challenge and figuring out a way to generate optimal code from a set of signatures, but I think I've spent enough time as it is haha.

My company has used Julia for about 5 years now. 90% of the time in application code you only need single dispatch, same as OOP.

One case where I actually rely on multiple dispatch is conversion to more or less structured data. Inspired by Clojure, I have a function `conform(OutputDataType, input_data::SomeType)` and specific implementations depend on both types involved.

Multiple dispatch is also really helpful when two libraries don't quite do the same thing. In python pandas, numpy, and pytorch all have slightly different implementations of x.std() (standard deviation) with slightly different APIs. This means you can't write code that's generic across a numpy array or a pytorch tensor. In Python this could only easily be fixed if library authors coordinate, but with multiple dispatch you can just fix this in your own code.

I use it quite a lot (having made it), most often when I need to recursively process complex heterogeneous data structures. Merging two data structures, treemap, processing a Python AST, etc. Also in any situation where you want to be able to register new behavior.

One case where I'm finding it extremely useful is that I'm currently using ovld to implement a serialization library (https://github.com/breuleux/serieux, but I haven't documented anything yet). The arguments for deserialization are (declared_type, value, context). With multimethods, I can define

* deserialize(type[int], str, Any) to deserialize a string into an int

* deserialize(type[str], Regexp[r"^\$"], Environment) to deserialize $XYZ as an environment variable lookup but only if the context has the Environment type

* deserialize(type[Any], Path, WorkingDirectory) to deserialize from a file

That's one situation where ovld's performance and codegen capabilities come in handy: the overhead is low enough to remain in the same performance ballpark as Pydantic v2.

A use case is when a sub or method needs to do different things based on the data type of the argument.

Example: turning a data structure into a JSON string, solved with multi dispatch: https://github.com/moritz/json/blob/master/lib/JSON/Tiny.pm#...

Another common useful example is constructors. A Date class might have constructors that accept

   Date(Int, Int where 1..12, Int where 1..31) # Y, M, D
   Date(Str where /^\d4-\d2-\d2$/) # YYYY-MM-DD
   Date(DateTime) # extract the date of a DateTime object

Also, dependent types, IE value-based dispatch. Which can be incredibly useful when dealing with enums. That alone is enough to make me curious to try it!

Makes naming functions easier, eg multiply(mat3,vec3), multiply(mat3,mat3), etc

If the language is dynamically typed, you can use it for polymorphism.

AI has no soul

Probably because typing in Python is optional and kinda bolted-on after the fact, and you really need a good type system to get the most out of multimethods.

In particular, type declarations, as they were first introduced in python, did not have a run-time effect, and they very much have a run-time effect with multi methods.

(Dataclasses are kind of an exception to this rule, but one that feels far less fundamental than multi dispatch would).

Many of them do, e.g. Java. Why don't _some_ mainstream languages like Python not support it? Entirely design preference, usually because they want to have less emphasis on the type system.

It also looks familiar to multi dispatch in Raku, formerly Perl 6.

See for example https://docs.raku.org/language/functions#Multi-dispatch

Not just a prominent perl hacker, but one who's been working on a major new OO system for the language.

Or maybe it does: https://github.com/breuleux/ovld/blob/f254dcc6215d50d5f9c3d1...