Skip to content(if available)orjump to list(if available)

All BART trains were stopped due to ‘computer networking problem’

scarmig

I don't know the last time the entire BART system got shut down. There was a loud explosion earlier this morning, which may or may not be related, but there aren't any power outages.

No information as to the actual cause right now. Easy to speculate that it's a cyberattack, but I'm going to go for the free square and wildly guess it is a DNS issue.

Best wishes and godspeed to the folks who are working on fixing the issue, whatever it is.

throwup238

My free square is one of their control computers’ parallel ports frying out and now they’re unable to find a replacement because they buy them all as old stock off of ebay [1].

The ports are used as hard realtime GPIO so if some of the electrical isolation failed downstream, it could take out the motherboard. Back before Vista’s security model change, drivers could fill DMA buffers to the parallel ports controller and get hard realtime time outputs on Windows so there’s a quite lot of old industrial control systems running on a thread.

[1] https://www.mercurynews.com/2022/09/17/how-clever-mechanics-...

mrguyorama

Forgive my arrogance, but nothing presented in that article should be such a big blocker. Spare parts for the traction gear sure, but none of the computing "problems" make any sense.

It's not like hard real time systems aren't available. More concretely, they talk about running a DOS virtual machine on a laptop to download logs from the cars, but there's no way that protocol is so complicated it couldn't be re-implemented reasonably.

This sounds more like "It's cheaper to just buy old stuff off ebay than it is to actually care about this system"

jkingsman

I'd wager it is technically a "cheaper" problem, but I'd also consider the axis that this is public transit, an industry known for being neglected budget-wise. Reverse engineering/reimplementation with competent parties may be laughably out of budget.

culopatin

I thought I heard that explosion and then just attributed it to a dream. Thanks for confirming

AStonesThrow

> wildly guess it is a DNS issue

Riding around in Waymos here, in the Jaguar models there is a button on the console labeled "DNS" and I have no desire to grab the steering wheel or adjust any other control, except every time I climb in, I am sorely, sorely tempted to press this "DNS" button because (1) I do not know what it does and (2) I have always had a soft spot in my sysadmin's heart for DNS in particular.

Please do not reply to tell me what "DNS" means in a motor vehicle, because you will ruin the mystique.

dmoy

Sorry I will ruin the mystique:

It's Do Not Schedule, carry over from when there was a human behind the wheel.

scarmig

Oh, that just updates the AAAA record for the Waymo. Totally harmless.

ToucanLoucan

Do Not Slide. So just hit that if you lose traction, or if you're having a hard time getting your kid out of the park.

jeffbee

I don't see any reason other than sheer speculation to suspect a cyberattack. It can just as easily have been some car crashed into a fiber optic thingy.

The last time in recent memory there was a large BART disruption it had been caused by a motorcyclist who somehow flung himself over a fence into the trackway and died. That stopped service in and north of Oakland, which is more than half of the system by riders.

scarmig

All sheer speculation--the reference to cyberattack was from a now dead comment suggesting it was ransomware.

It doesn't seem likely to be a physical obstruction on the tracks, though, as the entire system is down and trains aren't running anywhere. I don't know if that's happened before.

fhkatari

I used to live in Oakland, and am really sad to see continued failures of BART. They had started significant expansions right before the pandemic, allowing a long (but no traffic) ride possible from Oakland to San Jose. A key challenge for BART is that they depend a lot more on ticket sales than subsidies, and as a result, have been hit much harder with lower ridership.

schoen

> They had started significant expansions right before the pandemic, allowing a long (but no traffic) ride possible from Oakland to San Jose.

They're still working on this, with four more stations planned beyond Berryessa (Little Portugal, Downtown San José, Diridon, and Santa Clara), plus an additional infill station on the Berryessa line. I think that would be really cool. Unfortunately it looks like this new extension won't be that competitive with Caltrain as a way to get to San Jose from San Francisco. Maybe at non-express times.

Also, it looks like it won't be complete until 2040!?

jerlam

As an occasional BART rider, the changes they've made since the pandemic have been in the right direction. I'm mostly indifferent to the new trains and payment cards, but they've increased the frequency so that missing a train doesn't mean you can be waiting for over 30 minutes, which can be longer than your entire trip.

The main problem which BART cannot fix is that the trains usually don't go to where you want to go.

sagarm

BART can fix that by building dense mixed use -- office, commercial, and residential -- around their stops.

xlbuttplug2

> sad to see continued failures of BART

I think this is overstated, at least from an operations point of view. My mom has been using BART to commute to work for over a year and I can't recall many incidents like this.

andbberger

BART's doing great. good headways, same schedule all day everyday. name another american transit agency doing a takt

yawnxyz

ever since I paid $8 for jumping in and out of a BART station b/c I meant to go into a Muni station... I lost all respect for them.

Lammy

Fun fact: the reason it's like that is because both levels were envisioned for BART usage before the Peninsula lines got cut. In the original design both levels would have been the same fare area and you would have been able to walk between them instead of having to take the big escalators down to BART caged off from the Muni level. It's comical to watch one of the Muni trains crawl to one end of the giant platform that was sized for 10-car BART trains.

Johnny555

This is pretty common among transit agencies, I got on the wrong platform in Japan once and couldn't get back out until I talked to a station agent, the fare gate gave an error and wouldn't open the gates. Not sure if that's better or worse behavior than charging a fare.

It's called an "excursion fare", which is meant for those that just ride the train without getting off and come back to the same station. You can talk to a BART station agent (assuming you can find one) and they'll let you out, or call customer service and they'll reverse the charge.

Modern fare systems should be able to figure out when you've exited right after entering and not charge you. BART is supposed to be adding a 30 minute grace period so if you go in and out of the station within 30 minutes, you won't be charged.

https://www.bart.gov/guide/faq#3

xlbuttplug2

I too learned that the hard way when dropping a family member off. I naively assumed it wouldn't charge me if I tapped out at the same station 10 minutes later.

ta1243

London Underground has nice clear policies around this, both about what you'll be charged

https://tfl.gov.uk/fares/how-to-pay-and-where-to-buy-tickets...

And about refunds (typically you'd get an automatic refund for a one-off event)

https://tfl.gov.uk/fares/refunds-and-replacements/touched-in...

I always have high levels of respect for companies with such clarity

crote

In The Netherlands you get a full refund if you tap out at the same station within 20 minutes. If you travel with NS (National Railways), you even have 60 minutes to tap out.

Having someone pay just to wave off someone is incredibly customer-hostile. Besides, how many people are even committing fraud like that?

mogdance

Oh! I thought they were going to get rid of that fee: https://sfist.com/2022/12/08/rejoice-bart-is-dumping-the-6-4...

a_t48

You can talk with the clerk to get a refund. But I thought they were doing away with it entirely…

impute

You can also call them and get it refunded after the fact

dyauspitr

It’s not frequent though.

ryanmcbride

I'd say the BART administrators overpaying themselves is the bigger moneysink. Not to mention the spending 73M dollars to overhaul the gates to stop fare evaders, when it SHOULD just be free for bay area residents.

darth_avocado

The problem is that the Bay Area needs a single transit agency. All the agencies fighting each other for funding doesn’t help. It’s a much easier task to levy a minuscule tax on the entire region that pays for free public transit. If BART wants to do that at the moment, they can’t.

trollbridge

I live in an area with a county-wide transit sales tax, and it just seems to result in them running a lot of empty buses around and building (and eventually abandoning) transit centres. Full price is $1.50 (which if paying cash means fiddling around with coins) except for the 50+ mile express routes which are $2.50.

jeffbee

The MTC is that organization for the Bay Area.

jedberg

> when it SHOULD just be free for bay area residents.

There was a study done on this. It turns out that the median income of someone riding under the Bay Bridge in BART is higher than someone driving a car across it.

In other words the wealthier people are using BART. So if you made it free, you'd be subsidizing the wealthy.

darth_avocado

That’s a very wrong conclusion to draw from the study. If median income of someone riding the bart is higher than someone driving, it could also mean that they drive because it is cheaper to do so. If you make it free, they would probably ride BART too, making it more accessible and equitable.

tekno45

bart doesn't get to decide its free.

SllX

> when it SHOULD just be free for bay area residents.

Absolutely not.

scarmig

I think there should be options for low income residents to ride for free or heavily discounted rates (which exists now). But it's all about implementation: simply letting anyone jump on turns the system into a madhouse. Making sure everyone pays (even if through a government issued pass) and works with the system helps balance equity considerations with maintaining safety and cleanliness.

satiated_grue

Not quite but nearly totally unrelated, but I love that Debian ships a package of a PDP-8 PAL-like cross-assembler written at BART:

https://tracker.debian.org/pkg/palbart

http://www.pdp8online.com/ftp/software/palbart/

talldatethrow

Most people I know wouldn't take Bart if it was free. Dirty, noisy, and unsafe. I can't imagine anything can be done to make Americans take trains at scale unless atleast the unsafe part is handled, and then the dirty part atleast.

neilv

> Due to a computer networking problem BART service is suspended system wide until further notice. Seek alternate means of transportation.

That's worth a screenshot.

Any idea whether the political and technical will is there, to post-mortem this, and make the system more robust and resilient?

a_t48

Once the initial shock of “how the hell do I get to work” wore off, it was nice taking the ferry and F to SOMA. Took an extra 20 minutes, but better than the time it would have taken waiting in bridge traffic.

throwaway81523

Some system wouldn't power up this morning. System is back in service now, with major delays in all directions.

https://www.sfgate.com/bayarea/article/no-bart-trains-runnin...

CoffeeOnWrite

I’m worried about BART. The new gate tap scanners work so poorly, somebody really screwed those up, and they haven’t publicly explained what went wrong. I don’t understand why the media doesn’t cover this apparent bad screw up. There were a number of articles on the selection of the design and the planned rollout and the demonstrated susceptibility to ongoing fare evasion, but silence on the adverse impact on paying passengers.

Every day the last week at my station there are piles of commuters held up by the semi-broken scanners. Kudos to the front line staff down there apologizing. I am not holding my breath it’ll be fixed anytime soon.

kaladin-jasnah

It's interesting this is such a large problem there. In NYC, the tap-to-pay OMNY system is literally excellent. I swear it processes cards in a second or less and you save a lot of time by not buying a MetroCard. Are the BART scanners for some sort of RFID/NFC cards that are local to the train system, or do they accept credit/debit card payment? I'm curious how OMNY is so fast (compared to even going to a store and paying with tap to pay) and why BART is slow.

trollbridge

And OMNY will either accept a tap to pay credit card, a phone with tap to pay (including Apple's express card option, where it works even if your phone is dead), or a prepaid value dedicated OMNY card.

It does not actually run the transaction through the entire way - if a transaction fails, the card info gets placed on a blacklist and that particular NFC device won't receive an instant authorisation next time. Generally speaking, people don't have an easy way to generate lots and lots of fake NFC devices, so this hasn't been a problem for widespread fraud (vs just jumping over the turnstile).

plorkyeran

The Clipper cards used by BART are 90s technology that took so long to fully deploy that it was outdated by the time most people started using it. There was a brief window where if you were an early adopter it was better than what was available in many places (or not-so-brief compared to OMNY, which came along a decade later), but unsurprisingly the things designed later have mostly improved on it.

The killer feature that also causes most of the quirks is that it can be used to make payments fully offline without allowing double-spending of balance. This is of course mostly a killer feature for transit operators rather than users. OMNY solves the same problem by just accepting that it'll occasionally permit free rides.

null

[deleted]

beezlebroxxxxxx

I don't know if this is the case with BART, but I've seen in my own local public transit network an incredible inability or lack of will to use systems proven in other areas. Instead of going with quick, proven, and reliable, systems, they'll default to going for the cheap option, which is usually slow, re-inventing the wheel, and unreliable. I visit a city like NYC or London or Tokyo and see a transit system with decades of accrued understanding, technical experience, and optimization. Then I come home and ride something shiny but slow, janky, and bug ridden.

0_____0

It works so well that I was severely confused the first time I used them, coming from the Bay Area. I spent ages trying to figure out how to instantiate and load a transit card into Google Wallet before realizing I could just tap any card I had.

abeppu

Yesterday I went through Glen Park Bart, and every one of the scanners had a taped-on hand-written note in sharpie that said "hold 4 seconds". I doubt that the taller gates will ever pay for themselves (I thought in general fare-enforcement costs meaningfully more than systems lose in evaded fares), but I definitely don't understand why we needed to "update" gates with scanners that are clearly worse.

scarmig

The gates are less about direct fare recovery and more about limiting the externalities that gate jumpers impose on other passengers. BART is definitely calmer and cleaner than it was a year ago.

joshuamorton

> (I thought in general fare-enforcement costs meaningfully more than systems lose in evaded fares)

This is generally true for active fare enforcement, since you have to pay employees to do enforcement, there's appeals processes, and some people just don't pay the fine.

If the new faregates result in some more people (lets say 1%!) paying for trips (less freeriding), that's directly 1-2 million per year. If they also increase real-ridership, that's additional income. To make the cost back quickly you do need a significant increase (10-15%), that's not totally out of the question, though it's probably not only due to faregates.

jeffbee

The point of the new scanners is you can pay to ride with any payment instrument, not just a Clipper card.

a_t48

Source on this?

jeffbee

BART isn't the only agency suffering from those new readers. The problem is the vendor, Cubic. They supply this junk to many agencies.

a_t48

IIRC it’s the same sensors as the old ones

CoffeeOnWrite

I have no idea and didn’t comment on what the sensors are. Before the new gates, they worked. Now they only half work.

whalesalad

I lived in DC for a few years before I moved to SF, spending a few years there as well. DC Metro is not perfect by any means, but I was really surprised at how much better transit was in the district than the bay area considering they have zero weather issues to contend with and are - ahem - home to a tremendous amount of technological talent.

My personal observations are really that California are just truly fucking terrible at this sort of thing. Ironic considering they are such a huge economy and so wealthy. In northern California PCH (pacific coast highway) has been closed for over 15 months due to a rockslide. In southern california, a huge segment of ACH (angeles crest highway, one of my favorite places on earth) has been closed since 2023! You cannot drive from one end of the range to the other at this time.

China would have fixed these issues in weeks. For all the cash and people they have, Cali really manages to drop the ball on these things constantly. Don't even get me started on high speed rail that was built out in the middle of buttfuck farmland from Madera to Shafter. Like a stairway to nowhere.

jbverschoor

Due to human failure when operating computer systems*

noitpmeder

Their HQ lost power yesterday, maybe related?

quickthrowman

Right now I am a subcontractor on a project to replace the Liebert unit in the server room that runs my metro area’s transit system, my worst nightmare is my tech calling and telling me the servers are down, glad this isn’t me!!