Ukrainian hackers destroyed the IT infrastructure of Russian drone manufacturer

Why did they raid you?

I use GitHub Issues for this. It works so well - any time I make a decision I drop a comment on the relevant issue (often formatted as "Decision: ..."). Now they are archived, searchable, accessible via API and easy to navigate to from my source code because my commits all reference the issue number that relates to the change.

Every project I work on has a technical-decisions.org file. Also a daily-notes.org file with every failed experiment, test, install command, etc. The top level headings are dates.

Technical decisions used to be in the daily-notes.org file, but keeping in a separate file makes it more accessible to LLMs. I actually started that practice before LLMs were in common use, I struggle to remember why.

You also have to document alternative worlflows for your business while you don't get everything back to normal.

Lots of things can keep going with pen and paper or some cloud software.

At the very least, you have to communicate with your clients.

We just recently started using ADRs (Architectural Decision Records). They are deliberately stored (in markdown) in the same repository as the source code for our SaaS business lives. If we can recover the source, chances are high that we can also recover the "why's". If we cannot do that, we are screwed anyways.

That's why I delete all my company's data stores every quarter too!

I was very supportive of the infrastructure IT team when they moved their datacenter. I also had popcorn when watching the switch being figuratively flipped on.

It went surprisingly well despite having stayed 15 years in the old DC without rebooting. They were super scared of exactly the case you described but except for some minor issues (and a lot of cussing) it was OK.

Actually I think this is hard to properly implement. If you're a small shop, really setting up backups with redundancies, writing the documentation, and testing disaster recovery, that's so much more work than people anticipate, and it has implications on all areas of the business, not just IT. So usually it's hard to justify to management why you would put in all that work and slow down operations—which leads to everyone postponing it.

Either that bites you sooner or later, or you're lucky and grow; suddenly, you're a larger organisation, and there are way too many moving parts to start from scratch. So you do a half-hearted attempt of creating a backup strategy held together by duct-tape and hope, that kinda-sorta should work in the worst case, write some LLM-assisted documentation that nobody ever reads, and carry on. You're understaffed and overworked anyway, people are engaging in shadow IT, your actual responsibilities demand attention, so that's the best you can do.

And then you've grown even bigger, you're a reputable company now, and then the consultants and auditors and customers with certification requirements come in. So that's when you actually have to put in the work, and it's going to be a long, gruesome, exhausting, and expensive project. Given, of course, that nobody fucks up in the mean time.

Just the other day one of my clients had a production critical server failing and we started restoring it from backups.

Turns out some of the software running on it had some weird licensing checks tied to the hardware so it refused to start on the new server.

It turns out that the company that made this important piece of software doesn't even exist anymore.

> Modern IT practices don’t really contemplate disaster recovery. Even organisations with strict backup procedures seldom test recovery (most never at all).

I think this is an outdated view. In modern enterprises DR is often one of the most crucial (and difficult) steps in building the whole infra. You select what is crucial for you, you allocate the budget, you test it, and you plan the date of the next test.

However, I'd say it's very rare to do DR of everything. It's terribly expensive and problematic. You need to choose what's really important to you based on defined budgets.

That's a choice that companies make. I've certainly worked at places which don't test DR, while at my current job we do annual DR runs, where we'll bring up a complete production ready environment from scratch to prove that the backups work, and the runbook for doing a restore actually works.

If you’re doing it right, the DR process is basically the deployment process, and gets tested every time you do a deployment. We used chef, docker, stored snapshot images, and every deploy basically spun up a new infrastructure from scratch, and once it had passed the automated tests, the load balancers would switch to the new instance. DBs were created from binary snapshots which would then slave off the live DB to catch up (never more than an hour of diff), which also ensured we had a continuously tested DB backup process. The previous instance would get torn down after 8 hours, which was long enough to allow any straggling processes to finish and to have somewhere to roll back to if needed.

This all got stored in the cloud, but also locally in our office, and also written onto a DVD-R, all automatically, all verified each time.

Our absolute worst case scenario would be less than an hour of downtime, less than an hour of data loss.

Similarly our dev environments were a watered down version of the live environment, and so if they were somehow lost, they could be restored in the same manner - and again, frequently tested, as any merge into the preprod branch would trigger a new dev environment to automatically spin up with that codebase.

It takes up-front engineering effort to get in place, but it ended up saving our bacon twice, and made our entire pipeline much easier and faster to manage.

I used to find it amusing how many people thought Backup was a requirement.

"No, Restore is" I would say to stunned faces...

The nature and place of my work helped to quickly clear this.

I volunteered to help because I knew that even broadly planning the recovery, evidence preservation etc. would be completely beyond the capabilities of the two IT folks (they were extremely nice and helpful, and glad that there was someone to help).

I was there to draw things on the board and ask the questions that will help to recover. I would not have (nor want, not have the need) to access patient information. This is something I warned them about early in the process, as the chaos was growing.

You need to imagine a large hospital completely blocked, with patients during an operation being stabilized and driven away.

I am used to crisis situations and having someone who will anticipate things you do not think about (how to communicate, how to reach prople having planned procedures, who does what and who talks with whom) is a useful person to have before the authorities kick in.

My wife had a planned operation that morning and I was on site when the ransomware hit, it is just this. Nothing James Bond like, just sheer luck to have been around.

The hospital made a recovery but it took about a year IIRC

wait, what games are in that genre?

There's lots of pushes to add software to more of the military, but I don't think these kinds of resilience questions are really taken seriously. A system intended for wartime use will be running in non-optimal conditions while under constant attack. But many of these "enterprise" systems barely work better than paper to start with.

When HP converted to SAP, I think their production basically stopped, for six months, and they lost $400 million.

Switching to a new system; even when it is for the better, is a painful, expensive process.

The company that I worked for, did a successful transition to SAP, but it took about two years, and a lot of butthurt.

Forklift-Proof ERP was not on my bingo card. Thanks for the laugh :D

I know a manufacturing plant that used an Excel spreadsheet to do all its production planning. There was only one person who understood the spreadsheet and could modify it, a consultant who made more than the plant manager.

"Understandable and fixable" depends more on the complexity of the application rather than the fact it's in Excel.

The fact that every office worker understands excel, does not mean that every office worker understands every excel sheet.

Most of the projects we did in consultancy dev, was turning that one critical excel sheet nobody but 'the excel guy' understands into a simple to use web application, so that everybody could use it and the business won't explode when mr. excel would leave the shop.

Agree. IT has forgotten that computing should enable more people to be producers instead of mere consumers. IT management cares about control, audit, permissions and expense - no focus on achieving productivity in the workplace and in many cases are anti-user.

Most of those office workers were not capable of fixing anything on the first day they used Excel. Many didn't understand it at all. The main difference isn't that Excel is super accessible and easy to use for non-technical people; it's its ubiquity, and especially that of training on its usage.

I would argue that excel being "fixable" by regular office workers is half the reason why these projects fail in the first place. I've worked on migrating people's reporting to BI platforms before and what looks like a simple spreadsheet produced monthly is often really 12 different sets of formulas, special cases, kluges, hard-coded data and long-gone sources etc. etc. Because instead of correcting the source of data used for the report, it's all "done in post" in the excel sheet itself by a regular office worker.

They can continue to run the same thing they had before.

As an old software engineer, I can say with certainty that software engineering is a very, VERY wasteful practice. We could all be running Windows 3 right now, DOS, or some old Unix. The overhead involved in making actual advancements shows our slow progress as a species, and that we’re in a thread discussing a drone manufacturing facility being blown up in a war and how much that matters.

I think the natives had it right to live off of the land peacefully, and if anything to devote full time on science to determining what we do to help life survive in the universe.

That and 'if you don't use SAP you're not compliant with EU regulation XYZ and we won't do business with you'

Russia has shown to be plenty resilient across the board. I find it hard to assume anything different here.

No, they’re right, manufacturing machines like these are independent. We’re so used to interconnected software systems for everything, but even though these things may run old versions of Windows in airgapped or isolated networks, that’s just to run the machines. You give it a thumbdrive, save a part file on it, and as long as it’s got power, materials, and whatever is necessary for basic safety like noble gas for sintering safety, you’re set.

Even accounting systems are able to usually run fairly independently.

It’s not that IT and business and manufacturing support software engineers don’t help, but they aren’t necessary, especially if they’re just making the same thing over and over.

It's fallacious to assume that defenses stop evolving after new weapons come to the fore. Some drones are deployed in anti-drone capacities; the war economics becomes balancing how advanced to make the attack drones vs. how cheap the countermeasures are. In Ukraine we've already seen small drones that are able to damage the wings of much larger and more "technically advanced" platforms.

War didn't end the first time man invented the longer spear; defenses adapt.

You're right about this being a one-sided story, but not to suspect ChatGPT - it has none of the hallmarks of AI slop, plus it brings up a couple of reasonable and relevant points. You're only addressing a tiny part of the comment, but the rest stands, in my opinion.

Seems like drone warfare is just democratizing what e.g the US has in capabilities with their precision munitions already, in a perhaps less capable but far cheaper manner. Put it in other words if this was the US directy engaging russia, it would probably be tomahawk missiles or something along those lines just like we’ve seen last few decades, vs a sort of Air Hogs with a bomb.

This is called "break-glass procedure" in enterprise IT (as in "break glass in case of emergency"), and often consists of independent, normally unused, admin accounts on key systems, access info for which is locked in some safe location, e.g. physical safe in a secure location.

Testing this reliably is difficult, though, and often these procedures and their documentation is outdated.

I agree that fully redundant & separate infrastructure is unrealistic. I'm also not saying you can be 100% prepared. My point is that you can improve your posture.

What you can do is to have a sandbox environment where you periodically do a full setup exercise from a prepper disk. Conceptually it's not that different from testing backup recovery (ok, most companies neglect this too, so maybe you have a point :) ).

It's a model of a realistic scenario. Hackers (like in the article), long running ransomware that managed to corrupt lots of data, maybe a natural disaster. So by "wiping all data storage units" I meant the dynamic ones used in production. You can assume a static backup exists and contains a sensible set of sources and binaries, although obviously creating such a backup is part of the recovery plan.

Wikipedia has an entry for "hacktivism".

The people who illegally obtained classified information to leak to WikiLeaks have made a political impact: https://www.washingtonpost.com/technology/2024/06/26/wikilea... as well as reprisals in the form of arrests and prosecutions.

We also call Greenpeace "activists", but they also employed violent direct-action in their efforts against whaling.

Carl Icahn calls himself a shareholder activist, and many people still consider him a vulture capitalist.

>residents of Russian Federation probably were not a target audience of Russian version of this website anyway

Deliberately blocking the supposed enemy from hearing you does strike me as irrational, though. The mere fact they're doing Russian censors' job should probably make them recheck if they got anything wrong in their decision process, just in case.

No, that page is always a sign that the person who configured Cloudflare for their website has chosen to deliberately block Russian IPs.

But once that happened a handful of times it would be corrected.

I suppose it could be used sparingly but Ukraine would have no way of knowing when to use it. Perhaps a Bluetooth or whatever else the drone has on board "keep away" beacon for vips.

The Shadow Brokers are a great example, very likely Russians.

https://www.youtube.com/watch?v=fxqcwK5OMag

They have been, in various ways, been attacking NATO countries for past 2 decades. Its simply puttin's modus operandi. Physical attacks on civilian and military infrastructure, murders, meddling with elections, cyber attacks, you name it.

I mean Putin will spin everything as "the West did it" anyway, so it really does not matter who pushes that button. I also seriously wonder why we don't have US/NATO air forces over Ukraine for this reason.

Which NATO countries would those be?

There are plenty of reasons to have plausibly deniability even this late into the war.

I think the point being made is that Western agencies (5-eyes) would give Ukranian intelligence the button to push (indirect action) and not push it themselves (direct action).

It's reasonable to read it as implied

> Tomato, tomato.

This is a silly expression for written text, since I always read both tomatoes as 'tomato', before realising the intention. :)

It's Ukraine. Are you aware of the "banks' debt collectors"? They had thugs knocking on your door (and your face) for an overdue loan payment; they would _of course_ use violence/torture to extract information.