Scalable OLTP in the Cloud: What's the Big Deal?

> I'm also curious how much of this extra complication comes from having a central database that tries to be a source of truth for everything?

You can reverse this as 'how much extra complication comes from having many disparate databases, so now you push all ACID complications at the company level instead of the software(rdbms)'

It is a tricky problem!

Having a single source of truth is a tremendous simplification, especially for the people.

It should be ideal that the other databases are pure materializations on top of it, but then not matter what you have the need to input more stuff that is not part of the central one, and now you have both problems.

I agree that analytical queries cause a significant portion of the complication in transactional systems.

However, at global/twitter scale, just basic CRUD operations become pretty hard for OLTP, so scalable OLTP in the cloud is a pretty big deal, especially when you get fun things like phantom read/writes as writes may take time to get to distributed read replicas, so a user may send a tweet and then not see it on their profile for a while.

> From my experience, the vast majority of complications in systems is people not realizing they are asking an OLAP question while wanting parts of OLTP semantics.

If you could elaborate on this further, I and others are probably very interested in reading more about it.

Add a rate limit in nginx so that a single client has bounded work on your backend (e.g. 1/1-10s), and batch your requests into ~100/transaction using an in-memory queue. 1 server has been able to deal with ~1M connections for many years. No fires needed.

On a business side, they drastically lower the request load and make scalping unprofitable by holding a Dutch auction.

You build systems designed for this workload, you don't foolishy try to use the same patterns as a CRUD app.

> This is a crushing load on a shared resource.

Taylor Swift tickets cost $500 each [1]

An AWS u7in-32tb.224xlarge with 896 vCPUs, 32 TB of memory and 200 Gbps network bandwidth costs $407/hour even at list price [2].

The question is whether they feel like going to the expense and engineering effort.

[1] https://www.businessinsider.com/guides/streaming/how-to-buy-... [2] https://instances.vantage.sh/aws/ec2/u7in-32tb.224xlarge

You, as a business entity, have to figure out how you want to allocate them. They aren't going to get sold the normal way, so you get to choose whether you want to acknowledge that and design the system, or not design it and let nature take its course.

I've been to events that used a lottery. You order a ticket any time in, say, a month window. At the end, they tell you if you actually got a ticket. They have a process for linking orders together so you can choose to get a ticket if and only if your friends do.

I've also been to C3, which (in the second phase) knowingly used a first-come-first-serve three times, putting some kind of lightweight proxy in front of the actual shop that would only allow a certain number of people to access it at a time (this is important because you don't know how many tickets each user is going to order). In the first phase, they use a system of "self-replicating vouchers" for various trusted C3-affiliated groups: one voucher is given to each group, allowing an order; at the end of each day until this portion of the ticket pool runs out, a new voucher is given to whoever made an order the previous day. I don't know the reasons why self-replicating vouchers are designed exactly that way, but it means each group gets to run down their own self-determined priority order and gets punished for inefficiency.

The capitalist approach is, of course, raise the price to twenty thousand dollars or whatever level it takes for only 50,000 people to want to buy a ticket.

As others have said, this is a solved problem in a lot of companies. Basic answers are: 1. Queuing 2. Asynchronous APIs (don't wait for the 'real' response, just submit the transaction) 3. Call-backs to the Client

A good async setup can easily handle 100k+ TPS

If you want to go the synchronous route, it's more complicated but amounts to partitioning and creating separate swim-lanes (copies of the system, both at the compute and data layers)

Sharding, pre-allocating leases of blocks of tickets across available resources, and eventual consistency. You don't need to keep the UX transactionally correct; you are able to say "0 tickets remaining" and then a minute or hour or day later say "100 tickets remaining". For something as popular as Taylor Swift, the fans will keep checking.

while all of the above engineering solutions work, I wish they adopt a human centric solution. When there is too much demand and too little inventory, they could introduce some kind of lottery system where they accept partial payments through out the day and then randomly pick tickets in a lottery (individuals and group bookings as well). In this way, fans can avoid the mad rush, ticketing systems don't have to handle the peak mad rush for 10 minutes.

> ticketmaster has 50,000 General Admission Taylor Swift tickets and 1M fans eager to hoover them up.

I think something like trivial postgres setup can handle this thing..

I think this is really talking about an orthogonal sort of scaling to what you're looking at. Yes, given sufficiently balanced traffic patterns, spanner can serve essentially arbitrary load. If I understand correctly, this article is proposing that most of the work of transaction processing can be done in a layer that's not affinitized, and only do the bare minimum in the key order affinitized server that is responsible for enforcing consistent ordering. This way, single key ranges can handle significantly greater load than they otherwise could, and load from read only calls can be entirely offloaded.

Spanner really can. It has its own quirks, but I've enjoyed working with it.

They seem to be missing a pricing page (or pricing information of any kind) as well as how to actually purchase the product.

Citus still has a centralized coordinator that all requests go through to be routed to their shard, or at least it did several years ago when I last ran it in production. It definitely scales further than single instances, but it's not a magic bullet.

Others can do it too: Spanner, FoundationDB. SI just has generally better performance because there is less to check, and you can always opt-in to row locking if you need (because of write-write conflicts)