The Newark airport crisis
76 comments
·May 25, 2025dehrmann
> its implementation of a “NextGen” air traffic control system to replace the current version may not be completed until 2034, even though the project was started in 2003.
Governments (and a lot of businesses) like to look at software as a one-time purchase, but it's really better too look at it as a liability and an ongoing cost. It'd be better to have a team make continuous, incremental improvements to the system than have "NextGen" last-gen replacement vaporware.
throwaway48476
There are examples of both. Firmware and embedded software usually is a one time purchase. Tax software requires ongoing updates. FAA software would fall somewhere in between.
pphysch
It really is a binary though. Either you need to think about and deal with deploying (system-wide) updates, or you don't.
Even infrequent ~5y update lifecycles tend to be extremely painful unless there is substantial investment in treating it as an essential business process. This leads to a "kick the can" mentality that translates to show-stopping amounts of tech debt.
yoaviram
Software is a liability, a product is an asset.
bombcar
This is a classic system at 99.9% capacity - there’s no slack to take up issues or do anything but run at, well, 99.9%.
And without something like a major disaster, it’ll likely continue to get worse and worse.
coderatlarge
i flew delta out of ewr last week. the pilot announced over speaker that air traffic control was serializing take offs on one runway to avoid problems. we sat on the tarmac for two hours waiting our turn. that was on top of another hour delay for our flight arriving due to this policy.
delta doesn’t reimburse for missed connections claiming air traffic control policies are outside of their control.
it reminds me of a dev team i worked with once which used single threaded memcache as a way to serialize inbound requests to a server with improper locking logic inside.
hbsbsbsndk
> delta doesn’t reimburse for missed connections claiming air traffic control policies are outside of their control.
AFAIK if it's one booking on the same airline or a codeshare they are required to rebook you. If you planned a "connection" which is two single flights with different airlines you don't get any legal protections. This isn't just Delta, no airline will reimburse you for missing a flight you didn't book through them
elijaht
What do you mean delta doesn’t reimburse connections? I believe they would still be responsible for getting you to your final destination
pixl97
Sounds like it's time to drop Delta
ctoth
> To save money, the FAA elected not to build a new STARS server in Philadelphia to support the move. A new server alone would require tens of millions of dollars, as well as installation of new internet and power infrastructure.
> Instead, it elected to send a “mirror feed” of telemetry from the STARS servers at N90, traveling over 130 miles of commercial copper telecom lines, with fiber optics to follow by 2030.
> The annoyances of traditional cable internet — frequent lag, dropped sessions — are probably familiar to those who stream video or play games online. But for air traffic controllers, even the smallest service disruptions can become dangerous.
So LOL what, they just ... piped it over the Internet? Also can someone make sense of this "new server" costing millions of dollars? Presumably it's not the cost of a server, which is orders and orders of magnitude less than that?
amluto
I doubt that 130 miles of copper is the public Internet. It’s presumably some legacy telecom system that depends on a bunch of generally fairly well made but thoroughly obsolete hardware.
Keep in mind that Ethernet over copper is only specified to ~100 meters. Long distance copper networks have been obsolete for a few decades.
izacus
Here in my EU state, air traffic control still used leased copper twisted pair for server link over ~150km as of at least 2015 or so. It's pretty reliable and not "public internet" at all - just two modems with known performance and known latency taking to each other.
Might be something like that in this case as well.
amluto
Sure, this kind of technology works fine. Except that it might be difficult to find people who how maintain or repair the fancy (pressurized? oil-filled?) copper cables or the modems, and finding parts might be interesting.
I did some enterprise network work ~20 years ago, including fiddling with some inter-building links, rummaging through closets, and visiting the inside of a legacy campus-scale analog phone exchange, and I’ve never even seen the kind of equipment that can send data at an appreciable speed (even by 1990s standards, and even with repeaters) over copper at a range like 100km.
In contrast, single-mode fiber has improved over time, but it’s not obsolete, and it has maintained a remarkable degree of compatibility over the years. New transceivers largely work on old fiber, old transceivers work on new fiber, etc.
SoftTalker
Why not pipe it over the internet? A few redundant ISPs at the endpoints should be as reliable as a private run of fiber or T1 lines or whatever bespoke solution they think they need?
mschuster91
As soon as it's on the general Internet, any ill-meaning enemy can just force the VPN endpoints offline by blasting them with junk traffic.
stevenhuang
That can be mitigated effectively in this case using an IP whitelist. All unrecognized traffic can simply be dropped.
yusyusyus
Nah, probably on leased lines like t1s/t3s. Airport telecom infra isnt always the best.
wkat4242
A "Server" is just layman's speak for that big room with all the noise and blinky lights. Including all the hardware, networking and software that goes with it. Of course you're not going to run critical infra on a single server :)
Though I have to admit I have seen government operations where the "server" was an old dell optiplex desktop lying on its side in a broom closet without ventilation, a post-it with "IT SERVER DON'T TURN OFF", a spiderweb of cables running through the closet and the "server" fans screaming for air trying to keep everything cool in the enclosed space. I'm not kidding.
I mean, I know, government. Small local welfare-related org. Shoestring budget. Sure, that sucks. But at least you can make sure it's tidy and the cabling doesn't look like shit. Jeez. I didn't imagine I'd still see that in this century. Do people no longer take pride in their job? They hadn't even activated the "AC Power on" in the BIOS so after electrical maintenance they had to wait for the "engineer" to press the on button again.
gopher_space
There's no budget to fix whatever you messed up when redoing the cabling, which we also didn't really have the budget for. Everyone who knows what the box actually does has retired or died, so nobody wants you in the broom closet shifting dust motes near a working system we can't replace.
edoceo
When they say "server" it means the (cheap) hardware and the (very expensive) software to drive these complex and critical systems.
mschuster91
If it's mainframe, and there is no reason to believe it's not given the strict requirements, the hardware is expensive as fuck and the people who can keep these beasts alive are just as expensive.
WillPostForFood
I think they are running on old Sun Ultra workstations, not mainframes. Hardware easily replaced and upgraded with much faster, cheaper options. Difficulty is much more on the software side.
edoceo
Agree, I didn't mean 10k servers, I mean 100k servers (eg); and I also meant $10M persons. Trying to (poorly) state a two orders spread.
khazhoux
> pp. lag
> hold all traffic
> rdy?
> gogogo
alephnerd
Aerospace is hard.
An ATCS like STARS needs to feed from multiple different OT and IT sources like radars, weather stations, other TRACONs, etc and is implemented in it's own airgapped environment.
It can get very pricy very quick. On top of that, the FAA's budget has been sclerotic for decades now after the 1980s era union action and the 1990s era national cost cutting.
And finally, it is a political organization, and NATCA is a fairly prominent union within the AFL-CIO, and could make the lives of NJ representatives hell for pushing reassignment out of Newark.
amluto
> Aerospace is hard.
I would believe it was hard. And maybe it still is if you’re unwilling or unable to take advantage of modern technology.
Current low-cost equipment can easily send 10 or 100Gbps over long distance fiber links. Depending on how quickly you want to fail over when a link or an entire switch, router or rack fails, there are plenty of options that make various tradeoffs between failover latency and bandwidth, all the way up to completely duplicating all the traffic on redundant routes. I would bet that the entire aggregate traffic needed for air traffic control in a region is well under 10Gbps. And 10Gbps dedicated links or leases or (effective) purchases of dark fiber are not expensive on the scale of the FAA. Air traffic should use a network with a lot of redundancy, so maybe multiple those low costs by something like 5.
pixl97
Heh, have fun hooking legacy systems to high speed networks without significant testing.
If seen plenty of old stuff crash because you'll have some ancient serial device with a limited buffer and someone jams a faster link in-between. All of a sudden you have a much larger amount of bandwidth delay product and the system doesn't handle a few megabytes of data getting lost on the line when it bursts for some unexpected reason. On the old fixed line that just couldn't happen.
moomin
It's amazing, really. You hear about "government overspending" all the time. You actually look into something in any detail and what you discover is a consistent pattern of underspending. Call it mismanagement if you want, but it is consistently what we, as votes, and the executive, ask them to do.
RyJones
Meanwhile, back at the ranch, I was a contractor for HHS for a year or so. When I started, there was a server farm built for a contract that was powered up and maintained; never used, it was broken up and sold for surplus. Just before I left, HHS thought they would like that program again, so us US taxpayers bought another tranche of computers to power up. I don't know the end of the story.
I could type more, but it would be a long and boring story.
newsclues
There is overspending. It’s a scam.
There is mismanagement.
There is also a misallocation and underfunding of essential services and infrastructure. This is the excuse for ever more funding.
ars
[delayed]
jjallen
The US overspends on military and underspends on most other things. Not sure when this will ever change. Only when it has to I guess.
cperciva
The general pattern I see is that operating spending is too high while capital spending is too low.
anon7000
So true. The state of Washington has a great example: the outgoing Secretary of Transportation was very clear that if the state doesn’t change the budget to be primarily about maintenance, and much less about new highways or lane expansion, then our infrastructure will quickly begin crumbling.
Even the people in charge of our highways want us to switch to operations & maintenance oriented projects, but the representatives have not done so.
The incentives in government are really fucked — you get visibility and wins through cool projects, not by keeping the lights on and things running smoothly. Honestly true in big companies as well.
jagger27
Newark’s overworked controllers might argue differently. Of course in this forum the general suggestion will be to replace tired controllers with sleepless machines, and the technologists here have strong incentives to advocate for such solutions.
cperciva
My understanding is that ATC controllers would be far less overworked if they had modern (and properly functioning!) equipment.
CamperBob2
(Shrug) ATC is no job for humans, and I'm tired of pretending it is.
If we were building our aviation infrastructure from scratch starting today, you would get some really strange looks if you suggested employing humans to manage air traffic.
Modified3019
Came across this post, which perfectly encapsulates a sort of pathological inversion of sunk-cost fallacy, where some people I encounter would happily let infrastructure rot and countless people suffer if it saves them pennies on their taxes, and are incapable of re-evaluating their position because “taxes bad” has somehow become such an ingrained/religious value that anything else is unconsciously and immediately rejected no matter the consequences.
https://old.reddit.com/r/goodnews/comments/1kuaasx/i_voted_f...
>>I posted this somewhere else ages ago, feel it's relevant
>I remember having a conversation with my ex's sister and their mum a few years ago, around election time. I try to not talk politics with people because it's a fast way to lose friends, but the topic came up between my ex and them over dinner and I just listened in.
>I remember them saying that the only thing they're interested in is tax cuts. More money for them. I had to chime in and ask what about the NHS, what about funding schools? They said they didn't care because they had private healthcare through their jobs (finance), so they don't need the NHS. The mum said her kids are through school so she doesn't care about funding schools, and the sister said she'll be sending her kids to private school one day. I was pretty gobsmacked at the brazen selfishness of it, and asked what if they lose their jobs - and therefore their private health care - or become unable to work, what if when you have kids you can't afford private school? Neither of them could grasp this hypothetical... it was as if I was speaking another language to them. They were just like 'but we do have jobs.' And what if you didn't? 'But we do.' It was just circular and they couldn't see themselves in any situation other than the one they were currently in.
>I can quite see why empathy is a hard concept for right wingers to grasp, it was like they just simply couldn't understand the concept. They weren't stupid either, and nor were they rich - the mum worked in admin for a finance company in the city, and the sister was being paid by the same company (mum got her in the door) to train as an accountant.
>I think about those two every now and then when I can't understand how the other side thinks. Because it seems we do literally think very differently.
J37T3R
It's like tech debt. It's an ongoing cost in a one and done environment, it's hard to see problems from the outside until there's catastrophic failure, and if there's a slow niggling annoyance of things getting worse over time the point where people notice enough to care is usually past the point of needing a refactor. So we get underspending where it matters, overspending where it doesn't, and the solution is always a redo.
alephnerd
Most people see a number $6 Trillion and something breaks inside them.
What they don't realize is maintaining infrastructure is expensive.
Sure there are a lot of inefficiencies with need to be fixed, but you can't throw the baby out with the bathwater.
Sadly, such is society, and this is a problem that happens everywhere - be they democracies or authoritarian states.
pj_mukh
"Most people see a number $6 Trillion and something breaks inside them. What they don't realize is maintaining infrastructure is expensive."
It breaks all our brains too, because that $6 Trillion has very little to do with maintaining infrastructure. The bulk of it is just direct payouts (Social Security, Medicare and defense contracts).
Pretty frustrating that this big number means the government is politically forced to do drastic austerity for things like keeping planes flying safely.
Also, makes DOGE starting at USAID (<1% of budget) look especially incompetent.
juujian
When people think wastage, they think bureaucrats wasting away in offices. Social security admin, air traffic control, and teachers are more like Amazon warehouses at this time in terms of utilization. Less than 1% of social security admin funding is overhead at this time. Any charity would be exhilarated to hit numbers like that.
imadierich
[dead]
bombcar
With 3 million employees and about one cent a flush, the US government spends 100,000 or so a day just to flush. Assuming three bathroom breaks a working day.
Barrin92
>With 3 million employees
that is, for context, about as high as it was in 1960 when the US population was half as large as it is today.
rad_gruchalski
Close down the toilets. Poof, problem gone. Think about applying to DOGE. They may like your ideas.
readthenotes1
Have been many billions of dollars spent over the years to "modernize" air traffic control. If my recollection is correct, most of it ended up being wasted.
The current administration is asking for a lot of money to try to fix it again
autobodie
Need more trains.
KennyBlanken
Air travel is far too subsidized by the public and I can think of few worse applications of public funds save maybe sports stadiums. It's a huge waste of energy, pollution, human labor...
Rail is incredibly efficient, and there's a reason China has been building high speed rail as fast as it can.
To all the "it would never work here" people: we used to be a nation of rail travel, where you could walk or bike or take a taxi to the local trolley/train/bus station, take a train to where you needed to go.
All that was systematically ripped apart by the auto industry either directly or indirectly. There is no reason whatsoever we can't work our way back, especially given how much faster and easier construction of a railway line is now.
zamadatix
I'd LOVE some proper high speed rail across the US but EWR is also huge for international flights. A good rail network would at least help getting to/from their for it but that still leaves a good amount of flights that make no sense to dump.
everybodyknows
> I can think of few worse applications of public funds save maybe sports stadiums.
Californians can.
Subsidies to the movie industry, plumbing run to waterless urinals, bullet trains between farm towns, ...
sofixa
> bullet trains between farm towns,
This is disingenuous and you know it.
I think it was the wrong choice for a number of reasons, but the farm towns in question are just first lot of the whole network, starting with the (supposedly) easiest part. Instead of building in the densest parts which would be even more complicated and expensive.
dehrmann
> Air travel is far too subsidized by the public
Don't most of FAA's funds come from taxes on air travel? And around half of Americans travel by air every year, so it's not a niche service.
alephnerd
> Rail is incredibly efficient
If you have the density to justify it.
There is a case to be made for enhancing rail transit in the eastern seaboard and maybe parts of the Midwest, but America is too large and sparse to justify rail transit at scale.
It makes more sense to concentrate on rail infra for freight transit and work on revamping our existing rail freight infra.
> there's a reason China has been building high speed rail as fast as it can
China stopped subsidizing HSR during the COVID recession. It costs the exact same as a flight ticket now [0] due to high debt [1] (excluding the Beijing-Shanghai track, which actually can justify usage).
Most Chinese use normal rail for intercity transit, but this is easier to justify given the density and ease of land acquisition.
But even then, China began slowing down railway investment and construction since 2018 [2][3], and started calibrating towards air transit [4] as part of a commercial aviation push [5]
[0] - https://www.nytimes.com/2024/05/13/business/china-bullet-tra...
[1] - https://www.caixinglobal.com/2019-01-29/zhao-jian-whats-not-...
[2] - https://www.caixinglobal.com/2018-01-02/china-railway-corp-s...
[3] - https://www.caixinglobal.com/2021-03-30/china-looks-to-slow-...
[4] - https://www.caixinglobal.com/2021-03-25/smaller-cities-reach...
[5] - https://www.reuters.com/business/aerospace-defense/comac-jet...
tuna74
Going from Qingdao to Beijing is much cheaper on train vs airplane. Takes roughly the same time as well.
Also, China does both new airports and new rail lines.
ck2
We had a 9/11's worth of death EVERY DAY for the first TWO YEARS of covid
Then a 9/11's worth of death EVERY WEEK for the next TWO YEARS of covid
We still have a 9/11's worth of death EVERY MONTH in 2025
Personally I don't want anyone to die in a plane crash
But apparently a hundred million other people do not care anymore about others dying needlessly
So factor that into your next flight if you are taking your life into your own hands?
wnevets
It turns out trying to run an airport like twitter is a bad idea, who knew
bzmrgonz
What's stopping us from implementing holo style 3d displays like in the movies? Star-Trek etc. Are we not there yet?? alternatively, what about VR? we could virtually project the traffic controllers out into space like silver-surfer or ironman right(pov of course)? I'm not an expert, but it seems we need a better UX/UI right?
plorkyeran
Are you aware that many things depicted in movies do not actually exist?
dehrmann
> I'm not an expert, but it seems we need a better UX/UI right?
This isn't where the problem is. It's system reliability, increased air traffic, and increased controller workload.
alephnerd
> but it seems we need a better UX/UI right?
Wrong.
It's the fact that the FAA has $5.2B in outstanding repairs but only $1.7B allocated for repair.
On top of that, the GS pay scale penalizes federal employees in high CoL areas.
Both are very difficult problems to solve, as the former means dramatically increasing the FAA's budget (which is tiny for the scope of responsibility it has across North America), and the latter means completely reforming the General Schedule.
On top of that, Congress constantly meddles with the FAA and DoT in general because it's the easiest way to get some quick wins for constituents.
The FAA has been working on modernizing air traffic control, but that project won't be completed til 2030 at the earliest.
Furthermore, the Northeast is a uniquely congested airspace with the massive number of airports and passengers.
mschuster91
The budgetary questions are unrelated to UX/UI stagnation though.
It's annoying how many modern web sites change their entire design framework once every two years, yes. But ATC? Aeronautics in general? Most of maritime? Once it's certified, it's practically ossified - and for good reason. Bad UI/UX can literally kill [1].
Nevertheless, I think it's worth having the debate - and that led by actual air traffic controllers, please - if and if yes, how, UX/UI can be improved.
[1] https://uxmovement.com/buttons/how-an-interface-mode-killed-...
alephnerd
On the hierarchy of needs, it's a much lower priority than actually investing in solving maintenance related problems that are the primary cause of the Newark ATC related issues.
I'm in no way qualified to comment on the details of the issue outlined here (though I did get stuck babysitting a friend's kids for many extra hours due to a 6hr delay on said friend's flight into EWR a few weeks ago). For anyone who lives far enough north and west of EWR though, I highly recommend trying either the Allentown or Scranton/Wilkes-Barre airports. I've moved my business travel to Scranton 100% and have been loving it. There are more restrictions on flight times, and basically everything needs a connection, but it more than makes up for any time I would have lost due to sinkhole traffic on I-80 and the EWR parking shuttles. It's amazing to park and be at the security line basically 3 minutes later.