Tolerating full cloud outages with Monzo Stand-in
33 comments
·February 13, 2025QuinnyPig
What I wonder is “have they isolated third party dependencies?” If AWS is hard down, those may well be impacted—in some cases, by their own third party dependencies. You can test turning off your AWS environment, but you can’t really test turning off S3 for everyone…
sleepgou
From what I understand of payment systems this is so that payments through card machines, contactless payments for public transport, cash withdrawals from ATMs, etc. all continue to work. A lot of those systems are surprisingly insulated from AWS simply by virtue of being extremely archaic
fujinghg
I wouldn’t assume that is the case. The failure modes are different that is all.
I saw a whole corp POS platform a couple of decades ago that was hanging off a TFTP server on a machine that no one dared turn off in case the world ended. One day the DC UPS failed, it didn’t come back up and they had no retail operations for several hours while they sent a bunch of cash to a guy who had left to help them fix it.
There’s stuff like that everywhere lurking in the archaic.
I know of a modem in a DC which is used to talk to a branch office running AS400 hardware that is so old they have to buy spares off eBay.
paulbjensen
My only conclusion is that Monzo would rather embrace the apocalypse than rely on Microsoft Azure to provide a tertiary fallback.
tikkabhuna
These blog posts are why I continue to support Monzo. Their openness is really appreciated.
noodlesUK
This seems especially relevant given the massive outage that Barclays, another major UK bank just suffered. Barclays was down for around two days with customers unable to spend money at all.
I suppose had they implemented a similar system, they would have degraded into a minimum viable banking system rather than the total outage that impacted so many brits.
mmikeff
On the last day that tax payments were due
4ndrewl
Really interesting. Would love to understand how they came to the decision to build this,and whether there's any precedent for it.
quesera
Payment card networks have delegated authorization plans, where if a major processor goes down, they will still route transactions and use a simplified secondary network for making approval decisions.
It's called "stand-in processing", and I assume it's the inspiration here.
4ndrewl
The Monzo example feels different though, as they're explicitly not looking to replicate all functionality, just something minimal to get by whilst they fix the primary cloud services.
theginger
A decent setup which allows you to prove you are not dependent on 1 cloud provider will probably pay for itself when it's time to negotiate discounts.
cbg0
I doubt the sales folks you'll be talking to will care about your multi cloud deployment, as they don't have the skills to verify something like that.
joshstrange
Completely unrelated to this blog post but I really dislike Fintech saying "Get paid early" in their promos.
It's clearly marketing at someone too stupid to be able to see right through how utterly useless that is. If you are celebrating getting your paycheck 1 day earlier (every time) then your financial literally and financial health are probably in the toilet. They _must_ know they are preying on people with statements like that.
Then again, 90% of Fintech seems to be just a heavy layer of lipstick over an archaic system. Often with very little care of if any of the tools actually help people and more of a focus on how flashy or how much people think they are being helped.
jkingsman
Though, in some cases (like when it's your bank saying it), it's usually just them frontrunning reliable (coming from a payroll provider) and predictable (getting paid the same time each month) ACH transactions with a near-zero likelihood of not settling, then crediting you the money before the ACH is totally settled, so not ALL cases are fintech gimmicks.
But most are, and unfortunately, as the proliferation of payday loans shows us, there is no shortage of desperate people and organizations willing to take advantage of that.
andrewaylett
Remember that Monzo is a UK institution -- ACH isn't relevant, and they can see the payment in flight if it's using BACS.
https://monzo.com/blog/2019/08/20/monzo-now-lets-you-get-pai...
quesera
Right, some banks will not post a deposit to your account until after a holding period. I deal with a lot of ACH payments, and despite a very strict schedule in the network, the retail customer-facing side is surprisingly unpredictable.
So the "post credit early" promise is not a gimmick, but the whole idea of being paid early is a gimmick. The next pay period is still a full period away, so any benefit to being credited early is literally a one-time, and probably just one-day thing.
blibble
as a banker, when I first heard about that I did I wonder if they've modeled that risk correctly
it's the sort of thing that could probably wipe out their capital completely in a black swan event
quesera
There's an ocean of historical data to predict reversal or settlement failure of ACH transactions.
I would guess that payroll credits are the second most-reliable category in the ocean of ACH transactions, right after US Treasury payments.
How black would this swan need to be to blow up this stability?
simonvc
It's a risk that was very much understood and it's fully covered.
joshstrange
> not ALL cases are fintech gimmicks.
Fair and that's all well and good. I'm just saying if 1-3 days delay of getting your paycheck is going to have a big impact on one's life then I encourage one to reexamine their decisions, something else is the problem.
null
Unrelated tangent: I was reading the article and suddenly realised that I could not identify the font. After a quick search:
> Our functional typeface is Monzo Sans, a custom cut of Universal Sans, meaning it’s unique to Monzo. We chose it for maximum readability, with generous dots and curled ends.
Intersting choice, but I dig it :)