Scalable OLTP in the Cloud: What's the Big Deal?

> I'm also curious how much of this extra complication comes from having a central database that tries to be a source of truth for everything?

You can reverse this as 'how much extra complication comes from having many disparate databases, so now you push all ACID complications at the company level instead of the software(rdbms)'

It is a tricky problem!

Having a single source of truth is a tremendous simplification, especially for the people.

It should be ideal that the other databases are pure materializations on top of it, but then not matter what you have the need to input more stuff that is not part of the central one, and now you have both problems.

> From my experience, the vast majority of complications in systems is people not realizing they are asking an OLAP question while wanting parts of OLTP semantics.

If you could elaborate on this further, I and others are probably very interested in reading more about it.

I agree that analytical queries cause a significant portion of the complication in transactional systems.

However, at global/twitter scale, just basic CRUD operations become pretty hard for OLTP, so scalable OLTP in the cloud is a pretty big deal, especially when you get fun things like phantom read/writes as writes may take time to get to distributed read replicas, so a user may send a tweet and then not see it on their profile for a while.

Spanner really can. It has its own quirks, but I've enjoyed working with it.

As others have said, this is a solved problem in a lot of companies. Basic answers are: 1. Queuing 2. Asynchronous APIs (don't wait for the 'real' response, just submit the transaction) 3. Call-backs to the Client

A good async setup can easily handle 100k+ TPS

If you want to go the synchronous route, it's more complicated but amounts to partitioning and creating separate swim-lanes (copies of the system, both at the compute and data layers)

Add a rate limit in nginx so that a single client has bounded work on your backend (e.g. 1/1-10s), and batch your requests into ~100/transaction using an in-memory queue. 1 server has been able to deal with ~1M connections for many years. No fires needed.

On a business side, they drastically lower the request load and make scalping unprofitable by holding a Dutch auction.

You build systems designed for this workload, you don't foolishy try to use the same patterns as a CRUD app.

Sharding, pre-allocating leases of blocks of tickets across available resources, and eventual consistency. You don't need to keep the UX transactionally correct; you are able to say "0 tickets remaining" and then a minute or hour or day later say "100 tickets remaining". For something as popular as Taylor Swift, the fans will keep checking.

> ticketmaster has 50,000 General Admission Taylor Swift tickets and 1M fans eager to hoover them up.

I think something like trivial postgres setup can handle this thing..

You, as a business entity, have to figure out how you want to allocate them. They aren't going to get sold the normal way, so you get to choose whether you want to acknowledge that and design the system, or not design it and let nature take its course.

I've been to events that used a lottery. You order a ticket any time in, say, a month window. At the end, they tell you if you actually got a ticket. They have a process for linking orders together so you can choose to get a ticket if and only if your friends do.

I've also been to C3, which (in the second phase) knowingly used a first-come-first-serve three times, putting some kind of lightweight proxy in front of the actual shop that would only allow a certain number of people to access it at a time (this is important because you don't know how many tickets each user is going to order). In the first phase, they use a system of "self-replicating vouchers" for various trusted C3-affiliated groups: one voucher is given to each group, allowing an order; at the end of each day until this portion of the ticket pool runs out, a new voucher is given to whoever made an order the previous day. I don't know the reasons why self-replicating vouchers are designed exactly that way, but it means each group gets to run down their own self-determined priority order and gets punished for inefficiency.

The capitalist approach is, of course, raise the price to twenty thousand dollars or whatever level it takes for only 50,000 people to want to buy a ticket.

Citus still has a centralized coordinator that all requests go through to be routed to their shard, or at least it did several years ago when I last ran it in production. It definitely scales further than single instances, but it's not a magic bullet.

Others can do it too: Spanner, FoundationDB. SI just has generally better performance because there is less to check, and you can always opt-in to row locking if you need (because of write-write conflicts)