Microsandbox: Virtual Machines that feel and perform like containers

The issue, at least with multitenant workloads, isn't "container vulnerabilities" as such; it's that standard containers are premised on sharing a kernel, which makes every kernel LPE a potential container escape --- there's a long history of those bugs, and they're only rarely flagged as "container escapes"; it's just sort of understood that a kernel LPE is going to break containers.

You cannot build a secure container runtime (against malicious containers) because underlying it is the Linux kernel.

The only way to make Linux containers a meaningful sandbox is to drastically restrict the syscall API surface available to the sandboxee, which quickly reduces its value. It's no longer a "generic platform that you can throw any workload onto" but instead a bespoke thing that needs to be tuned and reconfigured for every usecase.

This is why you need virtualization. Until we have a properly hardened and memory safe OS, it's the only way. And if we do build such an OS it's unclear to me whether it will be faster than running MicroVMs on a Linux host.

Importantly I'd like to see the configurations of the machines. There's a lot you can do to docker or systemd spawns that greatly vary the security levels. This would really help show what needs to be done and what configurations lead to what risks.

Basically I'd love to see a giant ablation

In a way, containers already run as honeypots with cash or crypto prizes, it's called production code and plenty of people are looking for holes day and night. While this setup sounds like a nice idea conceptually, the monetary incentives it could offer would surely be miniscule compared to real targets.

You can optimize a lot to start a Linux kernel in under a second, but if you're using a standard kernel, there are all manners of timeouts and poll attempts that make the kernel waste time booting. There's also a non-trivial amount of time the VM spends in the UEFI/CSM system preparing the virtual hardware and initializing the system environment for your bootloader. I'm pretty sure WSL2 uses a special kernel to avoid the unnecessary overhead.

You also need to start OS services, configure filesystems, prepare caches, configure networking, and so on. If you're not booting UKIs or similar tools, you'll also be loading a bootloader, then loading an initramfs into memory, then loading the main OS and starting the services you actually need, with eachsstep requiring certain daemons and hardware probes to work correctly.

There are tools to fix this problem. Amazon's Firecracker can start a Linux VM in a time similar to that of a container (milliseconds) by basically storing the initialized state of the VM and loading that into memory instead of actually performing a real boot. https://firecracker-microvm.github.io/

On Windows, I think it depends on the hypervisor you use. Hyper V has a pretty slow UEFI environment, its hard disk access always seems rather slow to me, and most Linux distro don't seem to package dedicated minimal kernels for it.

I think you need to provide more details on what VM software you’re using. On VirtualBox what you describe is very noticeable, and it didn’t have that delay in older versions. So it could be just an issue with that VM software and not a general “traditional VMs” issue.

The answer is that it doesn’t have to be like that.

In practice virtual machines are trying to emulate a lot of stuff that isn’t really needed but they’re doing it for compatibility.

If one builds a hypervisor which is optimized for startup speed and doesn’t need to support generalized legacy software then you can:

> Unlike traditional VMs that might take several seconds to start, Firecracker VMs can boot up in as little as 125ms.

In Linux, VM memory allocations can be slow if it tries to allocate GBs of RAM using 4K pages. There are ways to help it allocate 1GB at a time which vastly speeds it up.

Windows probably has an equivalent.

Creating the VM itself is fast. It depends on what you run in it. Unikernel VMs can start in a few milliseconds. For example, checkout OSv.

Try Windows Server Core on an SSD. I've seen VMs launch in low single-digit seconds. You can strip it down even further by removing non-64-bit support, Defender, etc...

Sounds like a VirtualBox problem.

I'm using Hyper-V and I can connect through XRDP to a GUI Ubuntu 22 in 10 seconds and I can SSH into a Ubuntu 22 server in 3 seconds after start.

I mean it is basically booting a computer from scratch, kind of makes sense. You have to allocate memory, start virtual CPUs, initialize devices, run BIOS/UEFI checks, perform hardware enumeration, all that jazz while emulating all of it, which tends to be slower than "real" implementations. I guess there is a bunch of processes for security as well, like wiping like zeroing pages and similar things that takes additional time.

If I let a VM use most of my hardware, it takes a few seconds from start to login prompt, which is the same time it takes for my Arch desktop to boot from pressing the button to seeing the login prompt.

I'm trying this out now and it's very promising. One problem I'm running into with the Python library is that I'd like to keep that sandbox running for several minutes while I do things like set variables in one call and then use them for stuff several calls later. I keep seeing this error intermittently:

    Error: Sandbox is not started. Call start() first

The documented code pattern is this:

    async def main():
        async with PythonSandbox.create(name="my-sandbox") as sb:
            exec = await sb.run("print('Hello, World!')")
            print(await exec.output())

Any recommendations?

Looks great! This might be extremely useful for a distributed/decentralized software testing network I'm building (called Valet Network)...

Question: How does networking work? Can I restrict/limit microvms so that they can only access public IP addresses? (or in other words... making sure the microvms can't access any local network IP addresses)

i'm on a mid-level laptop, at times with slow or expensive internet, running ubuntu. i want to be able to run nominally-isolated "copies" of my laptop at near-native speed

1. each one should have it's own network config, eg so i can use wireguard or a vpn

2. gui pass-through to the host, eg wayland, for trusted tools, eg firefox, zoom or citrix

3. needs to be lightweight. eg gnome-boxes is dead simple to setup and run and it works, but the resource usage was noticeably higher than native

4. optional - more security is better (ie, i might run semi-untrusted software in one of them, eg from a github repo or npm), but i'm not expecting miracles and accept that escape is possible

5. optional - sharing disk with the host via COW would be nice, so i'd only need to install the env-specific packages, not the full OS

i'm currently working on a podman solution, and i believe that it will work (but rebuilding seems to hammer the network - i'm hoping i can tweak the layers to reduce this). does microsandbox offer any advantages for this use case ?

This is very neat tech, but I think you might want to wait until you actually have Windows covered before making claims like https://github.com/microsandbox/microsandbox/blob/main/MSB_V...

How does the microvm architecture compare with firecracker?

Hi appcypher, very cool project! Does the underlying MicroVM feature provide an OCI runtime interface, so that it could be used as a replacement for runc/crun in Docker/Podman?

Only did a quick skim of the readme, but a few questions which I would like some elaboration.

How is it so fast? Is it making any trade offs vs a traditional VM? Is there potential the VM isolation is compromised?

Can I run a GUI inside of it?

Do you think of this as a new Vagrant?

How do I get data in/out?

Looks neat. If I understand correctly, I can use it to spin up backends on the fly? You have an ambitious list of languages to support: https://github.com/microsandbox/microsandbox/tree/main/sdk

edit: A fleshed out contributors guide to add support for a new language would help. https://github.com/microsandbox/microsandbox/blob/main/CONTR...

Your statement initially went over my head. Sorry lol. You can always download the installer script and audit yourself. I will set up proper distribution later.

You are right. We leverage libkrun. Libkrun uses virtio-mmio transport for block, vsock and virtio-fs to keep overhead minimal so we basically depend on any perf improvement made upstream.

Firecracker is no different btw and E2B uses that for agentic AI workloads. Anyway, I don't have any major plan except fix some issues with the filesystem rn.

On recent Linux, try:

    findmnt --real

Sorry I am lacking the context to understand this post. What does running mount inside a container do that's so egregious? Are host mounts exposed to the container somehow? I thought everything needed to be explicitly passed through to the container (e.g. using a volume)?

> can you share some thoughts on how you compare or future direction?

Microsandbox does not offer a cloud solution. It is self-hosted, designed to do what E2B does, to make it easier working with microVM-based sandboxes on your local machine whether that is Linux, macOS or Windows (planned) and to seamlessly transition to prod.

> Do you also use Firecracker under the hood?

It uses libkrun.

I can't tell if it uses firecracker but thats my main question too. I'm curious as to whether microsandbox will be maintained and proper auditing will be done.

I welcome alternatives. It's been tough wrestling with Firecracker and OCI images. Kata container is also tough.

I recommend trying Javy[0]. Javy allows you to build a WASM file that includes Javy's JS interpreter along with your JS source code. Note that Javy is a heavily sandboxed environment so it doesn't have access to the internet, or npm modules, a desirable feature for running user code.

We're building an IoT Cloud Platform, Fostrom[1] where we're using Javy to power our Actions infrastructure. But instead of compiling each Action's JS code to a Javy WASM module, I figured out a simpler way by creating a single WASM module with our wrapper code (which contains some further isolation and helpful functions), and we provide the user code as an input while executing the single pre-compiled WASM module.

[0] https://github.com/bytecodealliance/javy

[1] https://fostrom.io

You might be able to get away with running QuickJS compiled to WebAssembly: https://til.simonwillison.net/npm/self-hosted-quickjs

> Would that be a good use case for this?

That is an ideal use case

> Are there better alternatives?

Created microsandbox because I didn't find any

runsc / gVisor is interesting also as the runsc engine can be run from within Docker/Docker Desktop.

gVisor has performance problems, though. Their data shows 1/3rd the throughput vs. docker runtime for concurrent network calls--if that's an issue for your use-case.

It has a YAML config format to declare all of that so you could just do that once, or template it, generate it on the fly, fetch it from remote, or many other methods.

As a Nix user, I'm actually really excited to try this out.

I want to run sandboxes based on Docker images that have Nix pre-installed. (Once the VM boots, apply the project-specific Flake, and then run Docker Compose for databases and other supporting services.) In theory, an easy-to-use, fully isolated dev environment that matches how I normally develop, except inside of a VM.

Windows support is a work in progress. I haven't tested using VNC yet but it should be possible.