Build Your Own Database
47 comments
·October 21, 2025skeptrune
I love the design and examples in this post. Easy to read for sure.
Exercises like this also seem fun in general. It's a real test of how much you know to start anything from scratch.
ashleyn
I was tempted to knee-jerk dismiss this as "don't write your own database, don't even use a KV database, just use SQL". And then I remembered the only reason I'd say this is because I went through designing my own DB or using KV databases just to avoid SQL...only to realise i was badly reinventing SQL. It could be worth the lesson.
kevinqi
my only minor critique is using lorem ipsum examples. It tends to make me want to gloss over instead of reading; I prefer seeing realistic data. other than that, it's a really cool post
cube2222
I clicked through a couple of the articles in the OP, and I must say, the design and animations are extremely pretty!
Kudos for that!
myth_drannon
I also recommend this free online book to build a database https://build-your-own.org/database/
keybored
Part of the reason why I'm not a "maker" is because my mind gets ahead of me with all the things that I would need to do in order to do things properly. So the article starts out interesting and then gets more and more, well, not exactly stressful but I get a bit weary by it.
Not that I would aspire to implement a general-purpose database. But even smaller tasks can make my mind spin too much.
browningstreet
I don't disagree with your take in general, but I do think it's different reading about minutiae than being invested in it. If you actually are curing these requirements it's probably quite engaging. If not, the eyes and mind start to gloss over them.
As a different example: I'm moving this week. I've known I'm moving for a while. Thinking about moving -- and all the little things I have to do -- is way more painful than doing them. Thinking about them keeps me up at night, getting through my list today is only fractionally painful.
I'm also leveling up a few aspects of my "way of living" in the midst of all this, and it'd be terribly boring to tell others about it, but when next Monday comes.. it'll be quite sweet indeed.
keybored
> As a different example: I'm moving this week. I've known I'm moving for a while. Thinking about moving -- and all the little things I have to do -- is way more painful than doing them. Thinking about them keeps me up at night, getting through my list today is only fractionally painful.
this sounds familiar... :)
FpUser
>Problem. How do we store data persistently and then efficiently look it up later?"
I would say without transactions it is not a database yet from a practical standpoint.
dangoodmanUT
I think a lot of databases would disagree
4ndrewl
> Databases were made to solve one problem:
>
> "How do we store data persistently and then efficiently look it up later?"
Isn't that two problems?
nonethewiser
It's amusing to me that this is really quite a pedantic observation yet it's driving very earnest engagement from hackernews. Myself included. Absolutely nothing in this article is riding on if its 1 or 2 problems - it's an aside at best. Yet I'm still trying to think through if it's 1 or 2. I mean, the "and" is right there - that clearly suggests two. It's almost comical even, to say "Here is one problem: X and Y." Yet in another way it seems like 2 sides of the same coin.
I guess there is a rather fine line between philosophy and pedantry.
Maybe we can think about it from another angle. If they are 2 problems databases were designed to solve, then that means this is a problem databases were designed to solve: storing data persistently.
Is that really a problem database were designed to solve? Not really. We had that long before databases. It was already solved. It's a pretty fundamental computer operation. Isn't it fair to say this is one thing? "Storing data so it can be retrieved efficiently."
mrighele
It is a single problem that contains two smaller problems, but the actual hard part (a third problem, if you wish) is putting them together. If you limit yourself to solve those two problems independently you won't have a (useful) database.
i_k_k
I always wanted to ship a write-only database. Lightning fast.
null
pcdevils
Pretty much how eventstoredb works. Deleting data fully only happens at scavenge which rewrites the data files.
hxtk
I think it was a joke. It sounds like you read it as append-only, like most LSM tree databases (not rewriting files in the course of write operations), but I think GP meant it as write-only to the exclusion of reads, roughly equivalent to `echo $data > /dev/null`
elygre
Back in the 80s a professor at our college got a presentation on the concept of «write-only memory» accepted for some symposium.
Good times.
archerx
That would be useful for logging.
Etheryte
Also useful for backups, so long as you don't need to restore.
warkdarrior
If it's write-only, and no reads ever happen, one can write to /dev/null without loss of utility.
grokgrok
How do we reconstruct past memory states? That's the fundamental problem.
Efficiency of storage or retrieval, reliability against loss or corruption, security against unwanted disclosure or modification are all common concerns, and the relative values assigned to these features and others motivate database design.
kiitos
> How do we reconstruct past memory states? That's the fundamental problem.
reconstructing past memory states is rarely, if ever, a requirement that needs to be accommodated in the database layer
nonethewiser
Can you elaborate? That certainly seems to be what happens in a typical crud app. You have some model for your data which you persist so that it can be loaded later. Perhaps partially at times.
In another context perhaps you're ingesting data to be used in analytics. Which seems to fit the "reconstruct past memory stat" less.
gingersnap
You're thinking of regex
pratik661
This is analogous to an elevator that’s unidirectional
rzzzt
One that lets people enter. We will figure out exiting later, with exiting on a different floor as a stretch goal.
dayjaby
Store data persistently so it can be looked up efficiently* sounds like a single problem.
SirFatty
Definitely two.
SahAssar
"Store data persistently" implies "it can be looked up" since if you cannot look it up it is impossible to know if it is stored persistently.
The "efficiently" part can be considered a separate problem though.
prerok
Well, if you just want to store data, you can use files. Lookup is a bit tedious and inefficient.
So, if we consider that persistent storage is a solved problem, then we can say that the reason for databases was how to look up data efficiently. In fact, that is why they were invented, even if persistent storage is a prerequisite.
nonethewiser
How about "store data in certain way." That sounds more like 1 problem and encompasses an even larger problem space.
cjbgkagh
It’s not persistent if it can’t be recovered later
codezero
[dead]
null
235ylkj
Here's a simple key-value store inspired by D.B. Cooper:
~/bin/cooper-db-set
===================
#! /bin/bash
key="$1"
value="$2"
echo "${key}:${value}" >> /dev/null
~/bin/cooper-db-get
===================
#! /bin/bash
key="$1"
</dev/null awk -F: -v key="$key" '$1 == key {result = $2} END {print result}'MathMonkeyMan
/dev/null is persistent across restarts and cache friendly, so it's got you covered.
if author is reading, can you add an rss feed to your site? i want to add to feedly.