A single line of code cost $8000

That's a good summary and explains many ills in the software engineering industry.

Because developers can suck.

I work with developers in SCA/SBOM and there are countless devs that seem to work by #include 'everything'. You see crap where they include a misspelled package name and then they fix it by including the right package but not removing the wrong one!.

>> their software also downloaded a 250MB update file every five minutes

> How on earth is a screen recording app 250 megabytes

How on earth is a screen recording app on a OS where the API to record the screen is built directly into the OS 250 megabytes?

It is extremely irresponsible to assume that your customers have infinite cheap bandwidth. In a previous life I worked with customers with remote sites (think mines or oil rigs in the middle of nowhere) where something like this would have cost them thousands of dollars per hour per computer per site.

Or.. Why on earth you need to check for updates 288x per day. It sounds and seems more like 'usage monitoring' rather than being sure that all users have the most recent bug fixes installed. What's wrong with checking for updates upon start once (and cache per day). What critical bugs or fixes could have been issued that warrant 288 update checks.

As I recall, it’s an Electron app. I just checked and the current version of Google Chrome is 635 MB, with its DMG being 224 MB.

So yes, it’s insane, but easy to see where the size comes from.

The app itself is probably much bigger than 250mb. If it is using Electron and React/other JS library like a million other UIs just the dependencies will be almost that big.

Just my hypothesis: some softwares includes video tutorial accessible offline. A short but not-compressed-high-res video can easily go big.

It was probably written by the type of programmers who criticise programmers like me for using "unsafe" languages.

I don’t use their software but if someone has they should be able to decompile it.

Even doable on very long range ADSL, guess there are still some dialup users.

Germany?

how? Do you investigate each blocked packet as separate alert?

murus+snail?

The big clouds love these people too. So much of the software industry is just an outrageous combination of inexperience and "YOLO". Every problem can be solved by just giving AWS another $100,000 this month because we don't have time (and don't know how) to make even basically optimized software. So just burn the gas and electricity and give more money to the YAML merchants at Amazon.

How about never? If I want to update my software, I'll update it. I don't need the application itself to hound me about it, at any frequency.

Why pay for your own runners when you can do CI/CD on your users’ machines?

That's stretching my definition of "good" quite a bit.

You can still do that with daily, weekly, or monthly checks.

Nobody under any circumstances needs usage stats with 5 minute resolution. And certainly not a screen recorder.

You definitely can, although it would be unlawful under the GDPR without user consent, so you could never release the figures publicly.

one time should be sufficient

Exactly. The AppStore already exists and does updates (either automatically or manually, configurable by the user). The developer didn't have to lift a finger to get this functionality. Imagine sitting down and spending time adding functionality to your application that is already provided for free by the operating system, and then after all that, doing it incorrectly!

> malice and/or other serious bugs should be assumed

Going back to the blog post and re-reading it with this possibility in mind is quite a trip.

> It turns out thousands of our users had the app running in the background, even though they were not using it or checking it for weeks (!). It meant thousands of users had auto-updater constantly running and downloading the new version file (250MB) over and over again every 5 minutes

This could easily have been data exfiltration from client computers instead, and few (besides the guy whose internet contract got cancelled for heavy traffic) would have noticed.

Yeah no, publishing to the App Store is a nightmare in cost and time. I can 100% guarantee they still saved money on 30% fees even after this $8000 snafu.

Screen Studio has 32k followers, lets say 6% are end users, 2000 users at $229, that is $137k in App Store fees.

I am going to say writing your own app update script is a wash time wise, as getting your app published is not trivial, especially for an app that requires as many permissions as screen studio.

That’s pretty much the floor for an Electron app.

If you’re a small shop or solo dev, it is real hard to justify going native on three platforms when electron gives it for (near) free. And outside of HN, no one seems to blink at a 250MB bundle.

There are alternatives like Tauri that use the system browser and allow substantially smaller bundles, but they’re not nearly as mature as Electron, and you will get cross platform UI bugs (some of which vary by user’s OS version!) from the lack of standardization.

Screen recorder under 100 bytes: ffmpeg -video_size 1024x768 -framerate 25 -f x11grab -i :0.0+100,200 output.mp4

And even when nothing changed?!? Fucking lazy developers aka "I have an idle ≥1Gb/s pipe to the download server". What happened to rsync/zsync/zstd (with dictionary)? There are so many good tools freely available to reduce wasted bandwidth when you insist on reinventing the wheel. sigh

yes obviously something as mature as the App store supports phased rollout. I believe it is even the default setting once you reach certain audience sizes. Updates are always spread over 7 days slowly increasing the numbers

Yes it does support this

You can? Don’t check for updates every 5 minutes. Daily or even weekly would be sufficient for an app like this (if auto-updater is even necessary at all.. just show a notification)

Don’t give them ideas!!

or they can send report usage statistics without you knowing or being able to disable it.

never attribute to malice what can be attributed to incompetence

Do they say that they don't do any usage reporting?

Eh, this one is probably ignorance over malice. It's super common to see people who need to make an arbitrary interval choice go with 300 out of habit.

> You have to understand, if user has 30 pieces of software, they have to update every day of the month. That is not a good overall user experience.

That's not an user issue tho, it's a "packaging and distribution of updates" issue which coincidentally has been solved for other OS:es using a package manager.

notepad++ works this way

550 megs?!?!?? On Apple’s unreasonably stingy SSD sizes?

Who would be foolish enough to download that?

Our 10MB was also for a "much easier dev experience" on Mac. The framework we shipped was basically 4x the size of the application.

> Code reviews kill velocity

This feels like a strange sense of priorities which would be satirised in a New Yorker/Far Side single-panel comic: “Sure, my mistake brought down the business and killed a dozen people, but I’m not sure you appreciate how fast I did it”.

Code should be correct and efficient. Monkeys banging their heads against a keyboard may produce code fast, but it will be brittle and you’ll have to pay the cost for it later. Of course, too many people view “later” as “when I’m no longer here and it’s no longer my problem”, which is why most of the world’s software feels like it’s held together with spit.

> Code reviews kill velocity

Yes, they kill your velocity. However, the velocity of a team can be massively increased by shipping small things a lot more often.

Stable branches that sit around for weeks are the real velocity killer, and make things a lot more risky on deployment.

The up-front cost of code review can be easily be tripled or quadrupled when it’s distributed over several weeks after the fact in the form of unplanned work, each instance of which incurs its own cost of context switching, as well as the cost of potential rework.

The purpose of such a review is a deliberate bottleneck in the earlier stage of development to stop it becoming a much larger bottleneck further down the line. Blocking one PR is a lot cheaper than blocking an entire release, and having a human in the loop there can ensure the change is in alignment in terms of architecture and engineering practices.

CI/CD isn’t the only way to do it but shifting left is generally beneficial even with the most archaic processes.

> Code reviews kill velocity - introduce context switching, and are make work

This is the same point three times, and I don't agree with it. This is like saying tests kill velocity, there's nothing high velocity about introducing bugs to your code base.

Everything introduces context switching, there's nothing special about code reviews that makes it worse than answering emails, but I'm not going to ignore an important email because of "context switching."

Everyone makes mistakes, code reviews are a way to catch those. They can also spread out the knowledge of the code base to multiple people. This is really important at small companies.

CI is great, but I have yet to see a good CI tool that catches the things I do.

If you demand accountability you need to grant authority.

When I work on my own code, at home, with no-one to assist or review, I write tests, and open a PR anyway, and review it myself, sometimes the next day with fresh eyes, or even 10 minutes later after a quick walk in and out of the room and a glass of water.

If you're not confident you can review a piece of code you wrote and spot a potentially disastrous bug like the one in OP, write more tests.

You’d be forced to use Apple’s App-Store, though? I don’t think there is an other package manager

I'm sure it cost a lot more than $8000. That was only the direct visible cost to them. There were likely users hit with costs for the additional downloads, who never even knew what was the issue. Users working on a mobile hotspot who had to pay for extra data etc etc.

> What did the CEO think of this?

I doubt there’s a CEO. Despite the use of “we”, pretty sure this is one guy building the app. All the copyright notices and social media go back to one person.

Imagine if that was Meta that had over 1B users with their messenger desktop app update functionality that did just that. The loss would be in the hundreds of millions.

> But agree that senior gatekeepers should know by heart some places where review needs to be extra careful. Like security pitfalls, exponential fallback of error handling, and yeah, probably this.

The lesson here is much better use of automated tests (The app likely has no tests at all) and proper use of basic testing principles like TDD would prevent such junior-level embarrassing bugs creeping up in production paid software.

That is the difference between a $100 problem vs a $200M problem.

See the case of Knight Capital [0] who lost $460M, due to a horrific deploy.

[0] https://www.henricodolfing.com/2019/06/project-failure-case-...

Yep, my first job was at a company like that. Huge Windows desktop app built in Delphi. No automated testing of any kind. No testing scripts either. Just a lot of clicking around.

I would mock/hook/monkey patch/whatever the functions to get the current time/elapsed time, simulate a period of time (a day/week/month/year/whatever), make sure the function to download the file is called the correct amount of times. That would probably catch this bug.

Although, after such a fuck up, I would be tempted to make a pre-release check that tests the compiled binary, not any unit test or whatever. Use LD_PRELOAD to hook the system timing functions(a quick google shows that libfaketime[0] exists, but I've never used it), launch the real program and speed up time to make sure it doesn't try to download more than once.

[0] https://github.com/wolfcw/libfaketime

I imagine they’d have to start testing first…

Similar to doix said, consider reading the time as IO and then rewrite the code in sans-IO style so you can inject the time.

Then it's a unit test that looks too obvious to exist until you read the ticket mentioned in the comment above it

No need for monkey patching or hooking or preload

But before that you add a couple checkmarks to the manual pre-release test list: "1 hour soak test" and "check network transfer meters before and after, expect under 50 MB used in 1 hour (see bug #6969)"

In Linux they're under /sys/class/net I think

> Or, given it's a Mac app, just have the Mac app store take care of updates. That's part of the value that using the app store service gives you,

And pay Apple their 30% cut on your revenue? No thanks.

> the other one being not spending thousands in accidental data transfer when you do auto updates wrong.

Or just actually write proper automated tests for basic features first, before a large refactor to prevent introducing issues like this from happening again?

While I respect the author's honesty in this mistake, the main takeaway here is not mentioned and that is just writing proper automated tests as their impression on this post is that there aren't any.

A quick warning "Hi User, your out of date, please update." Is fair.

What's really scary here is the lack of consent. If I want to record videos I don't necessarily have an extra 250mb to spend( many users effectively pay by the gig) everytime the developer feels like updating.