The Little Book of Linear Algebra
74 comments
·September 2, 2025andrewla
srean
> Even the "why does matrix multiplication look that way" is incredibly deep but practically impossible to motivate from other considerations. You just start with "well that's the way it is" and grind away
In my experience it need not be like that at all.
One can start by defining and demonstrating linear transformations. Perhaps from graphics -- translation, rotation, reflection etc. Show the students that these follow the definition of a linear transformation. That rotating a sum is same as summing the rotated(s).
[One may also mention that all differentiable functions (from vector to vector) are locally linear.]
Then you define adding two linear transformations using vector addition. Next you can define scaling a linear transformation. The point being that the combination can be expressed as linear transformations themself. No need to represent the vectors as R^d, geometric arrows and parallelogram rule would suffice.
Finally, one demonstrates composition of linear transformations and the fact that the result itself is a linear transformation.
The beautiful reveal is that this addition and composition of linear transformations behave almost the same as addition and multiplication of real numbers.
The addition asociates and commutes. The multiplication associates but doesn't necessarily commute. Most strikingly, the operations distributes. It's almost like algebra of real numbers !
Now, when you impose a coordinate system or choose a basis, the students can discover that matrix multiplication rule for themselves over a couple of days of playing with it -- Look, rather than maintaining this long list of linear transformations, I can store it as a single linear transformation in the chosen basis.
andrewla
> Perhaps from graphics -- translation, rotation, reflection
Maybe ... but the fact that you included translation in the list of linear operations seems like a big red flag. Translation feels very linear but it is emphatically not [1]. This is not intended to be a personal jab; just that the intuitions of linear algebra are not easy to internalize.
Adding linear transformations is similarly scary territory. You can multiply rotations to your heart's content but adding two rotations gives you a pretty funky object that does not have any obvious intuition in graphics.
[1] I wouldn't jump into projective or affine spaces until you have the linear algebra tools to deal with them in a sane way, so this strikes me as a bit scary to approach it this way.
srean
Mea culpa about translation.
For a moment I was thinking in homogeneous coordinates - that's not the right thing to do in the introductory phase.
Thanks for catching the error and making an important point. I am letting my original comment stand unedited so that your point stands.
About rotations though, one need not let the cat out of the bag and explain what addition of rotation is *.
One simply defines addition of two linear operators as the addition of the vectors that each would have individually produced. This can be demonstrated geometrically with arrows, without fixing coordinates.
* In 2D it's a scaled rotation.
cosmic_cheese
If anybody is aware of materials that teach linear algebra via graphics as suggested here, I would be interested to hear about them. As someone who learns best through practical application, maths have been by far among my greatest weak points, despite having written software for upwards of a decade. It’s limiting in some scenarios and pure imposter syndrome fuel.
viewtransform
3Blue1Brown [Essense of linear algebra](https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2x...)
te
codingthematrix.com
tptacek
I didn't think any part of linear algebra was boring. I was hooked from the moment I saw Ax=b => x = b/A. Gaussian elimination is a blast, like an actually-productive Sudoku puzzle, and once you have it down you can blaze through the first 2/3rds of an undergrad linear algebra course. I don't consciously try to gain automaticity with math subjects, but matrix-column multiplication I got pretty quickly and now I just have it.
I learned from Strang, for what it's worth, which is basically LU, spaces, QR, then spectral.
I am really bad at math, for what it's worth; this is just the one advanced math subject that intuitively clicked for me.
dpflan
MIT OCW is an amazing resource -- anyone can learn from Strang, which is a goldmine.
He also created a course on using Linear Algebra for machine learning:
> Linear algebra concepts are key for understanding and creating machine learning algorithms, especially as applied to deep learning and neural networks. This course reviews linear algebra with applications to probability and statistics and optimization–and above all a full explanation of deep learning.
- MIT OCW Course: Matrix Methods in Data Analysis, Signal Processing, and Machine Learning (https://ocw.mit.edu/courses/18-065-matrix-methods-in-data-an...)
- The text book website: Linear Algebra and Learning from Data (2019) https://math.mit.edu/~gs/learningfromdata/
- The Classic Linear Algebra Course: https://ocw.mit.edu/courses/18-06-linear-algebra-spring-2010...
tenacious_tuna
> I didn't think any part of linear algebra was boring.
My formal linear algebra course was boring as hell, to me. The ~4 lectures my security prof dedicated to explaining just enough to do some RSA was absolutely incredible. I would pay lots of money for a hands-on what-linalg-is-useful-for course with practical examples like that.
sureglymop
There's a book called ILA (Interactive Linear Algebra) that I found really good: https://textbooks.math.gatech.edu/ila/
tptacek
Try this. :)
(If you work through the prerequisites and use "understanding this post" as a sort of roadmap of what you actually need to know, this gets you about 2/3rds through undergraduate linear algebra, and you can skim through nullspaces --- all in the service of learning a generally useful tool for attacking cryptosystems).
jdshaffer
Didn't know about Gaussian elimination until today. Really cool, and really useful! Thanks for sharing!
andrewla
I haven't looked at Strang's approach.
The "x = b / A" is a bit of a gut-punch on first look because my mind immediately tells me all the ways that that does not work. It makes a some sense once I take a second to think about it, and I can see why it would make you want to jump in a little deeper, but matrices being non-commutative makes me cringe at the idea of a division operator which does not very very clearly spell out where it appears in the chain.
Ax = b is all well and good, but AxA^-1 = bA^-1 is not meaningful; the application/composition order is very important.
null
jameshart
What I find amazing is, given how important linear algebra is to actual practical applications, high school math still goes so deep on calculus at the expense of really covering even basic vectors and matrices.
Where vectors do come up it’s usually only Cartesian vectors for mechanics, and only basic addition, scalar multiplication and component decomposition are talked about - even dot products are likely ignored.
JadeNB
I think that, to be frank, it's a combination of (1) a curriculum developed before it was clear how ubiquitous linear algebra would become, and (2) the fact that it's a lot easier to come up with a standardized assessment for algorithmic calculus than for linear algebra, precisely because linear algebra is both conceptual and proof-based in a way that has been squeezed out of algorithmic calculus.
(I use algorithmic calculus to describe the high-school subject, and distinguish it from what in American universities is usually called "analysis," where one finally has the chance to make the acquaintance of the conceptual and proof-based aspects squeezed out of algorithmic calculus.)
Sharlin
For anyone who’s interested in graphics programming and/or is a visual learner/thinker, there’s an incredibly motivating and rewarding way to learn the basics of linear algebra. (And affine algebra, which tends to be handwaved away, unfortunately. I’m writing a MSc thesis about this and related topics.)
andrewla
To a degree I think this is true, but it requires (at least in my experience) that you have an intrinsic grasp of trigonometry for it to make sense. If you have some complex function analysis and e^itheta then you can skirt the problem for a bit, but if you're like me and have to break out soh-cah-toa whenever you break down a triangle then this method ends up being pretty tedious too.
Sharlin
I’m not sure what you mean. Beyond rotation matrices, there’s really only trig involved in graphics if you actively want it.
bmacho
There is no such thing as affine algebra: https://en.wikipedia.org/wiki/Affine_algebra
Sharlin
There are affine spaces, and there is an algebra of the elements of affine spaces. That is, rules that describe how the elements can be manipulated. There are affine transforms, affine combinations, affine bases, and so on, all of them analogous to the corresponding concepts in linear algebra.
(The term "algebra" can also refer to a particular type of algebraic structure in math, but that’s not what I meant.)
greymalik
> there’s an incredibly motivating and rewarding way to learn the basics of linear algebra
What is it?
tptacek
They mean graphics programming; learning graphics programming will give you intuitions for a lot of linear algebra stuff.
cassepipe
...
What is this incredible motivating way ? Please do tell
bmacho
Linear algebra has motivations and applications everywhere, since its main defining concepts, 'addition' and 'transformation that keeps sums' are everywhere. So a linear algebra curse is a huge pile of disjointed facts. It is not such a set of material that can have motivations behind it.
But the good news is that if you are only interested in for example geometry, game theory, systems of linear equations, polynomials, statistics, etc, then you can skip 80% of the content of linear algebra books. You don't have to read them, understand them, memorize them. You'll interact with a tiny part of linear algebra anyway, and you don't have to do that upfront.
Sharlin
Well, graphics programming itself. Learning while doing, preferably from some good resource written with graphics in mind. 2D is fine for the basics, 3D is more challenging and potentially confusing but also more rewarding.
joshmarlow
The older I get the more convinced I am that "math is not hard; teaching math is hard".
dapper_bison17
> This "little book" seems to take a fairly standard approach, defining all the boring stuff and leading to Gaussian elimination. The other approach I've seen is to try to lead into it by talking about multi-linear functions and then deriving the notion of bases and matrices at the end. Or trying to start from an application like rotation or Markov chains.
Which books or “non-standard” resources would you recommend then, that do a better job?
andrewla
I have yet to encounter an approach that is not boring. You just have to power through it. This approach seems as good as any.
Once you get to eigenvalues (in my opinion) things start to pick up in terms of seeing that linear spaces are actually interesting.
This approach sort of betrays itself when the very first section about scalars has this line:
> Vectors are often written vertically in column form, which emphasizes their role in matrix multiplication:
This is a big "what?" moment because we don't know why we should care about anything in that sentence. Just call it a convention and later on we can see its utility.
jcranmer
> Even the "why does matrix multiplication look that way" is incredibly deep but practically impossible to motivate from other considerations.
It's only difficult if you are wedded to a description of matrices and vectors as seas of numbers that you grind your way through without trying to instill a fuller understanding of what those numbers actually mean. The definition makes a lot more sense when you see a matrix as a description of how to convert one sense of basis vectors to another set of basis vectors, and for that, you first need to understand how vectors are described in terms of basis vectors.
nh23423fefe
I dont agree with this. Matrices don't convert sets of basis vectors to sets of basis vectors. What would you say about singular matrices for example?
The natural motivation of matrices is as representing systems of equations.
jcranmer
If I write a vector v = [1, 3, 2], what I am actually saying is that v is equal to 1 * e₁ + 3 * e₂ + 2 * e₃ for three vectors I have previously decided on ahead of time that form an orthonormal basis of the corresponding vector space.
If I write a matrix, say, this:
[[1 2]
[3 4]
[5 6]]
> The natural motivation of matrices is as representing systems of equations.
That is very useful for only very few things about matrices, primarily Gaussian elimination and related topics. Matrix multiplication--which is what the original poster was talking about, after all--is something that doesn't make sense if you're only looking at it as a system of equations; you have to understand a matrix as a linear transformation to have it make sense, and that generally means you have to start talking about vector spaces.
griffzhowl
There's no single best way to understand any of this, but the action of a matrix on the standard basis vectors is a totally reasonable place to start because of its simplicity, and then the action on any vector can be built out of that because they're linear combinations of basis vectors.
null
photon_lines
If anyone is interested in a more visual or intuitive over-view, I made a mini-book on it as well a few years ago which you can find here: https://github.com/photonlines/Intuitive-Overview-of-Linear-...
WillAdams
Nice pairing the text with 3Blue1Brown's lectures on linear algebra!
https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2x...
phforms
As an autodidact who never learned this stuff at school/uni, his lectures are what made linear algebra really click for me. I can only recommend them to anyone who wants to get a visual intuition on the fundamentals of LA.
What also helped me as a visual learner was to program/setup tiny experiments in Processing[1] and GeoGebra Classic[2].
- [1] https://processing.org - [2] https://www.geogebra.org/classic
csunoser
At around 7.4 orthonormal basis and there after, the tex rendering stops working on the github readme preview page.
Instead, it is replaced with a red error box saying: [ Unable to render expression. ]
I wonder if there is an artificial limit for the amount of latex expression that can rendered per page.
asplake
I switched to the epub at that point. Still, credit I think to github that the page renders as well as it does.
eliaskickbush
Highly recommend to anyone struggling with linear algebra to check out Linear Algebra Done Right, by Sheldon Axler. Do always keep in mind that some concepts are very verbose, but truly out of necessity. If you're talking about an N by N matrix, you're naturally going to have to distinguish N^2 different elements.
You can go very far without touching matrices, and actually find motivation on this abstract base before learning how it interops with matrices.
dkga
By the way, 3Blue1Brown’s videos on linear algebra are nothing my short of amazing. And I use linear algebra every day (I’m an economist).
rossant
A linear algebra course without graphics? When I learnt it at school almost 25 years ago, the teacher made schematics all the time to explain the visual intuition behind each concept. I was totally confused when he introduced the abstract definition of a vector space with the addition and scalar multiplication. Then he drew some arrows. Then it all suddenly made sense.
barrenko
Tried to pick a book to get into linear algebra recently, the experience was fairly hellish. First course this, second course that, done right, done wrong... I'd to the LADR4e route, but I don't have the proof-it chops yet...
griffzhowl
I like Serge Lang's books for clarity of explanations. He has an Introduction to Linear Algebra which concisely covers the basics (265 pages in the main text), and grounds the matrix computations in the geometric interpretation.
Be aware that Lang has another book, called just "Linear Algebra", which is more theoretical.
cybrox
I found the book "Linear Algebra" and accompanying lecture recordings by Jim Hefferon very approachable and solid.
It's free, including exercises and an (also free) solutions book.
GabriDaFirenze
I've found the "No bullshit Guide to Linear Algebra" pretty good. Could be worth checking it out. It's the one resource that has things click more for me.
ddavis
The organization and formatting of the single .tex file is such that one could almost read the source alone. Really nice. Also, I had no idea that GitHub did such a good job rendering the LaTeX math in markdown, it's imperfect but definitely good.
defanor
Always nice to see CC-licensed textbooks. This one looks fairly minimal, not including much of explanation, illustrations, or proofs; I think those are generally useful for the initial study, but this should still work as a cheat sheet, at least.
ivan_ah
Wow very nice. Lots of content in here, with no lengthy explanations but useful point-form intuition.
The .epub has very clean math done in HTML (no images), which is a cool way to do things. I've never seen this before. I wonder what the author used to produce the .epub from the .tex?
ivan_ah
Update: using Sigil to look inside the .epub, I saw it was produced by `pandoc` and the math is rendered as MathML.
m3047
I'd encourage the author to put a pointer to the repo in the actual doc. Maybe I should send a PR...
It's crazy that Linear Algebra is one of the deepest and most interesting areas of mathematics, with applications in almost every field of mathematics itself plus having practical applications in almost every quantitative field that uses math.
But it is SOOO boring to learn the basic mechanics. There's almost no way to sugar coat it either; you have to learn the basics of vectors and scalars and dot products and matrices and Gaussian elimination, all the while bored out of your skull, until you have the tools to really start to approach the interesting areas.
Even the "why does matrix multiplication look that way" is incredibly deep but practically impossible to motivate from other considerations. You just start with "well that's the way it is" and grind away until one day when you're looking at a chain of linear transformations you realize that everything clicks.
This "little book" seems to take a fairly standard approach, defining all the boring stuff and leading to Gaussian elimination. The other approach I've seen is to try to lead into it by talking about multi-linear functions and then deriving the notion of bases and matrices at the end. Or trying to start from an application like rotation or Markov chains.
It's funny because it's just a pedagogical nightmare to get students to care about any of this until one day two years later it all just makes sense.