Litestream: Revamped

Looks like the code is here: https://github.com/benbjohnson/litestream/tree/v0.5

Really nice to see this, I wrote this comment almost 2 years ago when I was a little miffed about trying to use litestream and litefs: https://news.ycombinator.com/item?id=37614193

I think this solves most of the issues? You can now freely run litestream on your DB and not worry about issues with multiple writers? I wonder how the handoff is handled.

The read replica FUSE layer sounds like a real nice thing to have.

edit: Ah, it works like this: https://github.com/benbjohnson/litestream/pull/617

> When another Litestream process starts up and sees an existing lease, it will continually retry the lease acquisition every second until it succeeds. This low retry interval allows for rolling restarts to come online quickly.

Sounds workable!

This post is like they read my mind and implemented everything I wanted from a new Litestream. So exciting.

Just a heads-up, the link in the "Litestream is fully open source" callout is malformed and leads to:

https://http//litestream.io/

Is Litestream on a path to subsume LiteFS's capabilities? Re: PITR, would this be used to facilitate automated A/B testing of AI-generated code changes against live data subsets? I can imagine a lot of cool stuff in that direction. This is really cool Ben!

If you wanted to use litestream to replicate many databases (ideally, one or more per user), which is one of the use cases described here (and elsewhere), how do you tell litestream to add new databases dynamically? The configuration file is static and I haven't found an API to tell it to track a new db at runtime.

What a coincidence, I was just researching Litestream today! I use Sqlite on my VPS and was thinking about adding this.

Am I understanding correctly that I will be able to restore a database to any point-in-time that is while the litestream process is running? Because auto-checkpointing could consume the WAL while it isn't running?

So for an extreme example if the process crashed for an hour between 2:00 and 3:00, I could restore to 1:55 or 3:05 but the information required to restore between 2:00 and 3:00 is lost?

This looks great! A few years ago I wrote a sqlite vfs for using dynamodb as a backing store[0] called DonutDB. With the recent addition of CAS to S3, I was thinking about making a new version of DonutDB backed by S3. I'm really glad lightstream supports this so I don't have to!

I can't wait to try this out.

[0]: https://github.com/psanford/donutdb

I wish Fly would polish the developer experience on top of SQLite. They're close, but it's missing:

1. A built-in UI and CLI that manages SQLite from a volume. Getting the initial database on a Fly Machine requires more work than it should.

2. `fly console` doesn't work with SQLite because it spins up a separate machine, which isn't connected to the same volume where the SQLite data resides. Instead you have to know to run `fly ssh console —pty`, which effectively SSH's into the machine with the database.

The problem in general with SQLite web apps is they tend to be small apps, so you need a lot of them to make a decent amount of money hosting them.

Fantastic to see it's getting updated! I am a big fan of litestream, have been using it for a while together with pocketbase. It's like a cheat code for a cheap, reliable and safe backend.

This is exciting! Especially glad that Litestream is still maintained. Is there a use-case for Litestream for more than backup? I am a fan of offline-first but it would be cool to have a way to synchronize on-device SQLite instances to a single central instance.

Amazing to see and hear about the progress. Always a pleasure when Ben works on something and shares it. Keep it up!

ben, thanks for litestream!

we're using it on production for a write-heavy interal use-case (~12GB compressed) for more than a year now; and it's costing us a couple hundred pennies per month (azure).

excited to try the new changes when they land.