Nix microVMs

Nix microVMs

MicroVM.nix is a Nix flake which provides facilities to declaratively manage lightweight NixOS systems that can share the Nix store with the host machine. It supports multiple hypervisors including QEMU.

Basic setup

Setup a simple Nix flake:

# flake.nix
{
	description = "MicroVM flake";

	nixConfig = {
		extra-substituters = [ "https://microvm.cachix.org" ];
		extra-trusted-public-keys = [
			"microvm.cachix.org-1:oXnBc6hRE3eX5rSYdRyMYXnfzcCxC7yKPTbZXALsqys="
		];
	};

	inputs = {
		nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
		microvm.url = "github:microvm-nix/microvm.nix";
		microvm.inputs.nixpkgs.follows = "nixpkgs";
	};

	outputs = { self, nixpkgs, microvm }: 
		let
			system = "x86_64-linux";
		in {
			packages.${system} = {
				default = self.packages.${system}.my-vm;
				my-vm = self.nixosConfigurations.dist-server-vm.config.microvm.declaredRunner;
			};

			nixosConfigurations.my-vm = nixpkgs.lib.nixosSystem (
				import ./my-vm {
					inherit
						system
						nixpkgs
						microvm
					;
				}
			);
		};
}

Under the my-vm directory, add a default.nix file:

# my-vm/default.nix

{ microvm, system, ... }:

let
    hostname = "my-vm";
in
{
    inherit system;
    
    specialArgs = {
        inherit
            hostname
            system
        ;
    };
    
    modules = [
        microvm.nixosModules.microvm # Include microVM module
        
        ./virtual-machine.nix
        ./networking.nix
        ./management.nix
    ];
}

Virtual Machine configuration

MicroVM.nix provides a microvm configuration object that exposes options for the hypervisor and allocated resources to the virtual machine. Here is an example virtual-machine.nix file:

# my-vm/virtual-machine.nix

{ ... }:

{
    microvm = {
        hypervisor = "qemu";
        
        mem = 1024; # in Mib
        vcpu = 2;
        
        volumes = [
            {
                image = "/var/vm/my-vm-persist.img";
                mountPoint = "/persist-data";
                size = 1024; # 1024 MiB = 1 GiB
            }
        ];
        
        interfaces = [
            {
                type = "tap"; # QEMU user networking can also be used.
                id = "vm-a1";
                mac = "02:00:00:00:00:01" # Locally managed
            }
        ];
        
        shares = [
            {
                proto = "9p"; # "virtiofs" can be used with system-launched VMs
                tag = "ro-store";
                source = "/nix/store";
                mountPoint = "/nix/.ro-store";
            }
        ];
    };
}

QEMU is a good reliable choice for a hypervisor. Volumes can be defined with images on the host machine, and are allocated and mounted automatically. It's also recommended to share the host's Nix store to the virtual machine, which reduces the size of the virtual machine's boot image and provides speedup when building the Nix flake.

Networking

MicroVMs follow NixOS's default configuration and attempt to grab a DHCP address on any interface exposed to the VM. For a tap interface, this is usually not the case unless the host has a DHCP server setup or the tap interface is bridged to an external network. While this is appropriate for MicroVMs which are launched with a system at boot, MicroVMs which are launched as a Nix package without any system configuration are best served by QEMU user networking.

systemd-networkd is recommended and can be configured with Nix:

# my-vm/networking.nix

{ hostname, ... } :

{
	networking = {
		hostName = hostname;
		# This only disables dhcpcd.
		# For systemd-networkd, simply configure in the networks section.
		useDHCP = false;
		useHostResolvConf = false;
		useNetworkd = true;
		
		nameservers = [
			"1.1.1.1#one.one.one.one" "1.0.0.1#one.one.one.one"
		];
	};
	
	systemd.network.enable = true;
	systemd.network.networks."20-lan" = {
		matchConfig.Type = "ether";
		networkConfig = {
		    # Static network configuration.
			Address = [ "10.69.69.2/24" "fded::dead:beef:bebe::2/64" ];
			Gateway = "10.69.69.1";
			DNS = ["10.69.69.1"];
			IPv6AcceptRA = false;
			DHCP = "no";
		};
	};

	systemd.resolved = {
        enable = true;
        fallbackDns = [ "1.1.1.1#cloudflare-dns.com"
			"9.9.9.9#dns.quad9.net" "8.8.8.8#dns.google"
			"2606:4700:4700::1111#cloudflare-dns.com"
			"2620:fe::9#dns.quad9.net" "2001:4860:4860::8888#dns.google"
		];
	};
}

Management

For a test VM, it probably suffices to just set the root password to be empty. Otherwise, this is where you might want to put NixOS configuration for OpenSSH.

# my-vm/management.nix

{ ... }:

{
    users.users.root.password = "";
}

Services

The MicroVM is still a NixOS system. To add services or software, see the NixOS manual.

Running a VM

MicroVM.nix exposes a Nix package that can be run to start the virtual machine.

$ nix run flake.nix#my-vm

If you're using a tap interface, you probably have to create it with user privileges:

# ip tuntap add vm-a1 multi_queue mode tap user <USER>

and then assign it an IP as appropriate. To setup a NixOS machine dedicated to MicroVMs, see MicroVM.nix's documentation.