Car companies are in a billion-dollar software war

For american cars at least, I read that one of the reasons this process exists is because car companies want to work around union rules for manufacturing by outsourcing components of the cars to subcontractors that they can make deals with.

Ultimately it's a price control strategy to pit these suppliers against each other to lower costs. But it means that designing these electronic sub-systems isn't just a question of the design itself, but also of managing all of these supplier relationships as well, they all have different contracts, you would have to coordinate all of them at once to make sure things are interoperable, etc.

> (every time a component needs to talk to another component a new protocol was invented).

i'm sure that every time this happens, it individually makes sense to do it at the time.

This is a microcosm of how large systems get developed in small pieces, by different people, over a long(-ish) period of time. It's the same in the software world too i think, but presumably has a lot more consolidation than cars (as software for cars might be less common, and thus employees moving between companies is unlikely to make any sort of cross-pollination like there would be for FAANG-like companies).

This makes it sound like the problem is that they either lack a person with architectural responsibility for the cars' electronics as a whole or that person lacks the skills necessary to do their job.

This aspect of the industry has seriously regressed. We started out trying to standardize and as vehicles have become more dependent on onboard networks manufacturers have gone completely proprietary and have put all information behind lawyers. The consumer is the real loser.

> every time a component needs to talk to another component a new protocol was invented

> We actually spent considerable amount of time just developing our own custom protocol

Not only is this unintentionally hilarious, it’s a real life example of an xkcd comic (https://xkcd.com/927/) that will never cease to be true.

> eventually all our data communication would use a single unified protocol from micro-controller to IoT Linux to cloud data-ingestion pipeline to database.

This, however, is remarkably impressive, that you were able to build a single protocol that fit this end to end use case.

I hate the penny-wise pound-foolish attitude both in embedded and Android phone development.

For years now, Samsung has used a 'virtual proximity sensor' in everything but their premium stuff. Sensors like that are a few cents. Degrading the entire experience on the phone for a few cents cost savings. Say you do that for 25 components, saving 4 cents each. You've now saved $1 on a BoM of $100-$200, whilst making the whole experience of your product feel a lot worse.

Your account sounds accurate, but how fitting then that their cost cutting focus is losing them customers and potentially their entire company. VW is losing the EV war. Most manufacturers have already lost. Tesla and BYD are going to eat everyone's lunch. They either need to revolutionise their approach, or they're toast. I suspect they'll attempt to milk their existing supply chains into bankruptcy.

If the OEM stayed with the same chip for several years, wouldn’t the price go down over time?

That’s always been the case with Tesla. I still have no idea how the yoke with no progressive steering and a tiny button for a horn ever passed any sanity check. Not to mention the NHTSA.

> The memory goes bad because they write useless logs to a chip, and it eventually fails.

I worked for a $ ~billions revenue software storage vendor who had the exact same issue (excessive logging wearing out under-spec'd flash drives).

You’re using a software fault which wore out the flash as evidence of poorly designed electronics?

I work in one of the big three - the culture here is more waterfall and less agile. They decided at some point ‘we don’t need to be experts in building systems, we should only be good at spec’cing them and putting them together’ This leads to a mindset of relying on suppliers for changing even one line of code and at their mercy. Talent leaves because they didn’t get to do any of the fun stuff. And you’re left with bunch of MBAs trying to wing it in what is available which is - no talent, bunch of admineers, and a long list of supplier bills. They go for cheapest component they can spec for a given feature cutting 4MB memory will save 5 cents per car, we sell half a million cars, that’s big savings! I can go on and on about this, but one of us even tried to be Tesla trying to build our own zonal architecture - and are currently struggling due to costs, tarrifs and turnover. Also you can’t overnight change this mindset - building vs assembling. But there has to be some way and I’m too about to walk out the door due to ~10yrs of frustrations.

There have been attempts at it. Unfortunately, they consistently botch the execution so badly that most of the executives in the business have PTSD from the experience. And these were very expensive failures that become lore inside the companies. When they do acquisitions of small companies entering this market those end up getting smothered by the culture of the automotive companies.

Everyone has spent a mountain of money on this problem but spent it all assiduously avoiding addressing the root causes.

The answer is that current car platforms were designed with flexibility as first goal.

Car companies realized early on they could outsource component development and production to 3rd parties and they could make them bid each other to further lower the prices.

So their platforms were optimized to be able to swap component vendors very easily (to achieve lowest costs).

Of course the vendors are not 100% interchangeable and building a platform to accommodate everyone has to make sacrifices.Aka target the least common denominator across all vendors.

How many issues due large companies run into thinking they can just throw money at it? Just look at google and stadia, or amazon and their failed game studio. They have immense money and knowledge and ended up with nothing.

Each car has dozens to 100+ ecus, written in different languages, by different teams, different requirements, and different companies. Some are proprietary. Ford can’t just tell Bosch, hey your abs module needs to now integrate with our api, multiplied by 100+ companies. The legacy car makers need to revisit everything, and move most of it in-house.

Because the auto companies outsource everything, lay the risk onto the outsourced companies and expect that some significant percentage of them will go bankrupt every year.

With that kind of adversarial relationship, you are never getting anything above the barest minimum of competence.

I worked at BMW. I knew there was a project in there, using a certain ECU that was being quite problematic (as in, project being slightly late because ECU was a bit buggy and sometimes crashed when it was supposed to have almost 100% of uptime for legal reasons).

You ask: Why BMW doesn't just buy the ECU manufacturer?

Well... the company that was selling the ECU to BMW, is BIGGER than BMW. Even if BMW sold 100% of its assets and stock, it wouldn't have enough money to buy the ECU manufacturer.

The talent might not exist. Software development has been seen as the preferable career over electrical engineering for a long time now.

They don't have a culture that values it, at any level. Historically hardware was important and software was a nice-to-have addon cost center. That's the mentality that the people at the top are still in, and it trickles down.

If you’re making very low end HW maybe this is true. Because HW is something that you put into the real world there are other constraints such as power, cooling, space, security of supply, ability to ramp, cost, reliability, etc. The calculus for HW selection is much more involved than simply SW. Good SW/FW can be performant on much less capable HW but it does mean that SW engineers need to understand more about the HW. This is a very rare skill in 2025. Most SW engineers I’ve encountered cannot explain stack vs heap. Furthermore even fewer understand how to use malloc correctly.

Blaming hardware people rubs me the wrong way.

People just use android and javascript front-end.

It's not crappy hardware by miles, crappy hardware as a category doesn't even exist these days.

It's hardware that can run everything necessary hundreds of times over, but shitty bloatland sloppy javascript it + android bloat it can not.

Motorola were a car radio company originally

Give it five years and it will be guaranteed garbage. Spotify will refuse to run on an unsupported older Android without the latest DRM API, while Google Maps will crash your system randomly, requiring you to disconnect the car battery to jumpstart it again. Volvo will offer you an upgrade of their proprietary device at the low price of $1899.

It's puzzling to see this push for general computing on devices that need to far outlast the typical release cycle of GC devices. There is nothing good that can come out of installing Android in your TV, fridge, let alone a - for fuck's sake! - a car.

If your consumer hardware needs to last for decades, then the core functionality and automation should be provided by sturdy embedded computers that are self-contained and do not require any kind of network access or regular updates, while the general computing functions functions should be provided by the user's own device or a replaceable/upgradable computer with a standardized interface.

Why did you think you'd "never again" use anything like Android Auto?

My own car is too old for Android Auto, but I sometimes drive a car that's from 2017 or so, and Android Auto works just fine on it, it's a pleasure to use (with the caveat that the phone has to be plugged in the USB port, wireless came later). So to me it seems like it always worked well.

Having your accounts, preferences, and history follow you into the car without juggling cables or switching UIs is exactly the kind of seamless experience SDVs should be delivering

My 2017 Mazda cx5 refuses to not play the radio. There is no "off" for the audio, you have to choose a source. I use my phone, via bluetooth. But sometimes, for unknown reasons, the car does not connect with the phone. It then falls back to the last source chosen before BT, which is radio. Okay, so I created a flash drive with an mp3 of 30 seconds of silence, played that, then went back to bluetooth. This failback strategy worked one time, then it also failed to recognize the flash drive, and failed back to radio, again.

I will never want to listen to the radio. I would love to remove radio as an option. I would love to have no fallback as an option. But no, the car just f-n loves the radio and will not stop trying to force it on me.

Oh yeah, and the radio is buggy and could get stuck if I tune into the wrong station. https://www.bbc.com/news/technology-60333765.

This car definitely tries too hard to be smarter than it is. There's all sorts of exceptions that keep the doors from auto-locking when I walk away, and I would turn all of them off, but I can't. Walk away too fast? doesn't lock. Open the rear? won't auto lock. Car just doesn't feel like it? doesn't auto-lock.

And god forbid you hit the unlock button when the passenger has already unlocked it. Anxious beeps from the car for several solid seconds. That is not an error condition!

Performance and reliability have been great though. They just need to stop trying to be smart. They're not.

Mazda also managed to squander a huge brand and structural advantage by falling into lockstep behind other Japanese automakers in underinvesting in EV manufacturing infrastructure. Now they have to rely on their JV partner Changan to lead the way in producing EVs, giving up the core structural strengths that Mazda previously had in designing and building their own components - including software and controls, which in the Changan-led models have no continuity at all with Mazda's domestic models. They just superficially copy the Mazda exterior design language while wholly dependent on Chinese supply chains (and some Android Auto for the software, it seems) for manufacturing the actual EV.

I bought a Mazda3 a few months ago and I love it. It is exactly what I want as a driver.

I even adore the scroll wheel and wish it could be in any car I own in future. Yeah it takes slightly longer to do certain actions in CarPlay, but I can do it so much more safely than I could in the Civic I had before. The infotainment boots basically instantly; as you mentioned CarPlay starts itself, and the patronising-but-mandated “don’t use this in motion” warning dismisses itself. In the Civic I would be half way down the road already by the time it booted, blindly prodding at the screen to try to dismiss that warning so I could pause the podcast that started playing itself because I plugged my phone in.

And, while my 2022 car predates the stupid auto-re-enabling ADAS requirement in Europe, the 2024+ models have single button deactivation. I dunno how, cause it’s supposed to require a minimum of two presses legally, but it sure makes me wanna stick with Mazda.

However that makes the upcoming 6E that much more disappointing. They’ve partnered with a Chinese manufacturer, I assume because they don’t have an EV platform of their own ready yet. Looks fantastic from the outside, but the inside is a sea of touch screens with barely a physical control in sight.

When I was doing my car shopping two years ago, I was initially considering another Mazda, specifically looking at the Mazda 3 AWD Hatchback. Their high tech features were significantly behind the other Japanese auto manufacturers. Some features like the ability for the car to automatically stay in a lane were not present.

When looking at who is doing it right, I wouldn’t put Mazda on a pedestal. They simply are behind the competition.

Generally agree but they are laying the path to enshitification. You see you can get turn by turn directions on the HUD, but only through their app where they want you to pay $10/mo for the privilege. Same for inputting addresses into their crappy nav system.

So I only use Google maps with Android Auto now, but cannot put the turn by turn display on. Also, who knows what telemetry Mazda is sending home on me without me knowing or wanting them to. Probably selling it to data brokers.

Oops somehow a switch has attached itself to the fuse of the LTE module in my vehicle.

Wouldn't be the first or the last time that a car has a different build out for different locales - as differences go, that's pretty minor.

> required

That’s…terrible

That’s interesting - what vehicles require you to do that? I know the usual suspect is the Tesla, which I have, but I never have to navigate through menus to change the temperature while driving.

As an aside a lot of people like to levy criticism on the infotainment screens which I think is very well deserved, but then people text and drive, watch YouTube videos, and do all sorts of crazy things too.

Instead of levying criticism on these distractions (let’s include billboard too) we should instead focus on just reducing car usage since we won’t stop people from being distracted.

The safest car is the one in your garage.

"Have done this" implies Slate has delivered even one vehicle. They have not. I hope Slate succeeds, but let's not get caught up in the preorder hype.

I was quite interested in this until I realized:

* Bed size is just five feet

* Towing capacity is just 1000 lbs

* Not AWD

None of these can be retrofitted after the sale.

Where I live, it'd struggle to be called a "truck" with these limitations.

The window winders I can do without. Not sure that even saves a noticeable amount of money at this point with electric windows such as commodity.

I mean, they did something, for sure, but they sure as hell didn't do "this" ;P. What they are doing is more in the line of not providing even hardware, much less software, which is an entirely different paradigm... like, they don't even provide speakers?!...

It'd be great if they make an engine swap package for existing trucks with optional battery sizes.

... but you can be a bro and make sure that hardware is close to the surface somewhere for easy access, its presence isn't required to start and operate the car (either firmware check or the immobilizer performing metrics), and its removal does not cause an obvious and annoying alert during operation (IE removal should not make the car appear to be in a 'degraded' state per its indicators).

You are complying by installing it, the customers are the ones [easily] removing it [because you were a bro].

Nissan tracks you and sells your data. Pretty much every manufacturer does, if your car has a modem rest assured that your car is collecting and selling data from you.

I would argue that the software quality of ADAS systems is very different from Infotainment.

Infotainment systems are a race to the bottom on BOM+SW price point. ADAS OEM's understand that there is a human cost, liability, and reputational cost for failure.

The real risk with these monoliths is when companies start to remove the distributed/redundant nature of safety critical systems, in order to reduce hardware costs.

There are multiple very good reasons for a distributed system in a car. However, irrespective of how clever your architecture is, there is only one good reason for centralized systems in a car and that is cost. It benefits no one but shareholders and C-suite.

OTA updates are sold as a key benefit but again it's marketing, they only reduce costs for the manufacturers and effectively remove a lot of the penalties of recalls. I would argue that difficult/costly recalls put pressure on manufacturers for 'first time right' design, OTA favours happy-go-lucky software.

Statistics work on generic population but mush away a lot.

People are careless and inattentive beast of animals in our modern societies. Things are done for them, expected this way, they do not need to pay attention that much, which has lot of merits and advantages for the advancement of humanity. Dumb solutions doing as told and need to be handled expertly can be dangerous for modern people. Developing automation right (emphasis is here, big emphasis!!) is very necessary.

But unfinished and sloppy developers are killing careful people. Not show in the statistics, saving more bad drivers than killing good ones overridden by shit software cars.

Need to do it right with no collateral casualties.

I believe the tone of the conversations are into this direction anyway: please, pretty please, do it right! Not the current sloppy way! This is a dangerous game not mobile messaging platform, needs different mindsets than average software development approaches.

I want the center console, but not hooked to the rest of the car. Instead, it’d have a standard screen, and a jog wheel that’s compatible with third party computers.

I’d settle for a bluetooth (call and music) capable fm radio though.

It is only as inevitable as consumers' alreadism-driven apathy. The moment they recognize that Car As A Service is something out of the sane world and having a means of transportation that can simply expire or be blocked remotely for a far-fetched TOS violation is against their interests, all inevitablism goes up in flames.

Interestingly, for me this worked very well. On my BMW M235i it was flawless. It had normal beams, and one could turn on the auto beams. One button to switch it on/off. I really liked it, as it was easy to activate, did its job, and when in doubt, I could deactivate it easily (button).

On my VW Golf GTD (mk7) it works also pretty good, only the activation/deactivation is strange. It uses the same switch, which is used to switch on the beams. Depending on the current state, it activates the beams, auto beam or turns it off. After more than a year of ownership, I still don’t know how to use it. Sometimes when I need to turn it off, it doesn’t turn off, but does something I don’t want it to do.

My car randomly braked today because it thought a car on a side road was pulling out. Not just sound the alarm but actually apply the brakes. Fortunately I didn't have a tailgater behind me.

I disable the "land assist" every time (which often tries to steer me into wildlife or other cars and was clearly not built for use on a single track country roads with hedges and random verges), but this was the first time in 3 years that the "front assist" caused problems.

If that's "high quality", I dread to think what low quality would be.

How do you square this with the article?

It states that consumer reports, (a for profit company providing independent reviews, and not a regulatory body) said the Model 3 stopping distance was not good. Allegedly due to a “bad ABS calibration”. Tesla released an OTA SW update.

Why wasn’t the bad calibration and degraded performance caught by regulators testing automobile safety standards?

The article also posits that this ability to make OTA updates expands the (IMO very very bad) SWE perspective that “it’s OK to ship unfinished and buggy products” into safety critical systems.

> ISO 26262.

That is a piece of paper.

> Software for steering or braking systems is of high quality.

There's literally no way for me to know that before I trust my life with it.

They also have a poor safety rating from NCAP (at least they did 2 years ago), because they don't fit their cares with electronic aids such as emergency automatic braking, which is just another reason to buy one.

A fender bender is lethal in a Lada

> I'm still appalled that Tesla got caught pulling footage of people having sex in their own vehicles

Anywhere I can read more about this? Sounds terrible.

[dead]

They only really became relevant ~ 10 years ago, I don't think they began selling lots of cars until ~2018 or later.

Having an engine problem on a back road is one thing, having a software-system-integration-what-the-hell problem on a Autobahn at 180 km/h +/- is a different story. And yes, I do not want my family in the car at that moment.

It was Boeing that intentionally hid the importance of a system much more relevant than flight characteristics of a plane. That is an intentional violation of the spirit of the safety checks.

That wasn't a malfunction but rather a flight control feature the pilots didn't know about. (Iirc)

I did this to my old car, but I skipped the CarPlay cause that's too glitchy. It's just aux or bluetooth.

Climate controls, including in-seat heating, as well as radio/media is exactly the stuff that needs actual hardware knobs that are always in exactly the same place and that I can use by knowing where in 3D space they are by muscle memory and feel without looking.

We have an EV9, and the user interface is so pathologically bad that we’re planning to get rid of it.

Everything makes it beep. Beeps for “you will die now” are similar to “you put me in gear”.

There’s one exception: For many reasons, it turns off one-pedal driving. When it does that and is unexpectedly accelerating into cross traffic, it does not beep (until the collision alarm sounds, presumably, ask me if it kills me…)

You can get tiny USB dongles that convert wired CarPlay to wireless. I bought one and adore it! https://www.geekbuying.com/item/AB019B-Wireless-CarPlay-Adap...

Hyundai is actual buttons, not touch.

I would love to have both. The scroll wheel is convenient when I’m driving, but the touchscreen would make entering a new address much easier, as it’s very annoying to do now by scrolling, and voice dictation doesn’t work well in my language.

Shoutout to the thing I got for my mom's Merc where it's a literal 10 second installation to have carplay in a 2011 vehicle. Swap the built-in navi box with a $75 Aliexpress plug - bam, CarPlay/AA all wirelessly on the main screen and controllable by the standard navigation knob.

Car companies have to worry about regulatory compliance, certification, approvals, as well as warranties; aftermarket manufacturers do not have such concerns (at least to the same degree).

What is an "MMI box?"

2010 Crown Vic is newest of that kind of car. I like it.

While I definitely hope Slate is a huge success and I really love its premise, it's a shame that, judging from their homepage images, the dashboard is an LCD screen, no dials.

Their vehicles remind me of the Datsun trucks I saw as a kid in the 80's. I believe back then their value proposition was that they were cheap too.

No need to drive a shitbox. These points are easily covered by 90s/early 00s decent Italian/German cars.

I do like me a good 90s car. Honestly, though, my above-listed criteria would be my ideal car. But, a good number of cars up through the 00s would be enjoyable for me, if not exactly perfect. For example, the 2007 Dodge Charger is pretty good. :)

It does. But that isn't what I want it to do. I want to manually lock the doors as soon as I close my driver side door.

The article is discussing moving safety critical functions like door handles and drivetrain into a centralized computer. This has been a disaster so far, and consumers hate it.

They cite tesla as an example of a “good” approach, and don’t understand that (in addition to Elon) a large percentage of the market won’t consider a car where the computer decides which way the vents point, and if you are allowed to open the doors after an accident.

It does mention that people hate touch screens, and probably will not like these new cars. Other than cost savings, and “the infotainment computer is slow” there zero discussion of how these new systems improve the car or the user experience. “Slow infotainment” should be fixable by throwing a better cpu/ram in.

Fortunately that is incorrect. I mentioned in another comment, but I’m well over $100k USD equivalent in salary alone as an embedded engineer, working in a relatively low cost of living area in Canada, graduated 3 years ago. Working for a “regular” company.

Maybe things just really suck for embedded in the states? But since my last year of university I’ve been inundated with recruiters for embedded positions, and I’ve never had a problem finding work. ~75th percentile in salary alone for software engineers in my area, ~55th-60th for Canada. I make more than every JS developer I know who graduated with me, except for the ones who moved to Seattle, Vancouver, or the Bay.

Admittedly I don’t know your salary or market, but it is possible to make decent money in embedded. Connections & market timing are both vital though, in my experience, as well as being actually good at your job. I’m in Canada so numbers are way different, but salary-wise I’m in the ~75th percentile software engineers in my area, my title is embedded engineer, and I’m fairly junior (3 years out of university, ~6 years full-time experience). I’m working with some other embedded people who are in the 95th percentile for software engineers in the country. The main problem is there are very few high-paying embedded jobs; conversely though, there seem to be even fewer highly-skilled embedded engineers looking for work. I recently interviewed at a company paying 50th-85th (based on experience) percentile trying to hire pretty much any competent embedded engineer, and their problem isn’t insufficient salary, it’s just a lack of applicants or any skill level. From what I’ve heard, the same seems to be true pretty much everywhere.

Now sure, if you’re looking for 500k+ jobs, embedded isn’t the area to be in, unfortunately. But I prefer low-stress, fun-environment embedded jobs, and don’t mind trading off salary for that. Different strokes.

Programming embedded devices is not the same thing as "embedded-systems programming". The latter means, first and foremost, that the software is not allowed to crash, ever, for any reason, else it is people's lives.

I did some initial requirements work on a system to monitor continuous-web papermaking machinery; the line had to be stopped, physically and completely, within 100ms if anything went wrong, because an uncontained web of paper can literally cut people in half. They wanted, in order to be able to hire, to use one of the embedded flavors of a well-known consumer-grade OS, and I had to prove to them that there was no way to make any of them safe, at any cost. And they knew their hardware, because they had built it themselves.

The absolute last resort is a watchdog timer that hits the reset button if N milliseconds go by without the software telling it it's okay. This is what you have to implement if you are dealing with buggy and undocumented hardware -- as, all too often, you are. Sometimes you can get some doco for $ and an NDA, but then in order to get the real doco it is much more $$$ and a much tighter NDA, and the existence of that option is not even divulged until after things have already gone very far south.

If it were only a matter of reading the top-level doco for this or that chip, there would be no issue.

RTOS-based development varies significantly from RTOS to RTOS, so I’m not sure how much it’d help to learn to use one. On the other hand, most fundamental OS knowledge is fully transferable to RTOS, so that would be helpful for embedded developers to understand.

Yes! But it's also obvious that the industry doesn't have a prayer to ever reform. Stuck between proprietary and NDA's chips everywhere, using proprietary and NDA's toolchains and development kits, to product proprietary DRM'ed products.

This is an industry that is about as far from the light of science & enlightenment as it is possible to get, ensnared as deeply in the entangling anti-human anti-science Intellectual Property qualgmire-hell as can be got. Oh sure plenty of science goes it! It's fantastically interesting & technical! But aside from some Application Notes write-ups trying desperately to help move the practice along, move it out of jank, knowledge goes in, but it doesn't ever come out! There's such a lack of peershios with which to practice science, to report your findings to, to replicate works on.

The software world talks about its patterns and practices. The biggest industries on the planet are building software like wild AND are mad into open source. But... computer engineering is the shadowland, where no talk nor victories that happen there are allowed to be shared, where nothing escapes confinement. What a fucking plagued awful land of people unable to ever do the right thing, unable to bring their work out of the dark & into real civilization.

It needs two! One to keep engine fires out of the passenger compartment[1] and one to keep unauthorized users or code out of your infotainment and control systems.

[1] https://wikipedia.org/wiki/Firewall_(engine)

It's not practical to produce a car that never needs updates. That would be a bug-free system, which is impossible. Since they're going to ship updates anyway, a lot of focus is on minimizing the cost and hence OTA.

For what it's worth, I work in this industry and the general rule of thumb is that every increase in validation from QM (standard quality) up to the various levels of safety critical code has up to 10x the cost per line of code of the previous level.

You can find router firmware sourcecode online and find pretty egregious vulnerabilities if you're really trying to learn.

Alot of embedded stuff is outsourced and doesn't want to waste the computing power for stuff like stack canaries. I recall the following from making a tool for a dlink? router?

//Reads a file name foo ReadFilePath() { // Get file name // TICKET 21321: Fixed crash by increasing buffer size char FilePath[100]; ReadFileName(&FilePath); }

It sticks out to me, since the crash was clearly from a buffer overflow, and they had this documented in the source code that increasing the buffer size fixes it. What they didn't realize was that the bug would still happen and you could get a buffer overflow from this and do whatever you wanted. This is the level of programmer you're dealing with who's writing embedded software in an overseas sweatshop. And the talent isn't even there domestically since they're severely underpaid compared to someone writing simple javascript.