Shikari: A Multi-VM launcher using Lima

[Ranjandas]

I have been working on a small tool written in Golang that helps create multi-vm clusters using Lima. It is heavily opinionated and primarily geared towards launching clusters for HashiCorp tools (like Nomad, Consul, etc.) but is general-purpose enough to build anything based on Lima templates.

Ref: https://github.com/Ranjandas/shikari

DISCLAIMER: I am not a programmer, and the code quality might be terrible. However, the tool works wonderfully for me :)
I ended up creating this as I was searching for something similar for a while. I started using Lima and was amazed by its power, so I decided to base the project on Lima.

I’m always looking to improve, so I’m open to any feedback, suggestions, or constructive criticism you may have. Feel free to share your thoughts, and thank you for taking the time to check it out!

The UX of the tool is simple in that it only requires you to pass the following:

• Name for the cluster (--name/-n)
• Number of Client (--server/-s) and Server (--client/-c) VMs
• Path to the lima template to launch --template/-t
• Finally, the VM image to be used --image/-i (optional if already included in the template)

In addition, it lets you inject environment variables into the VMs using the --env/-e option.

The VMS are launched with predictable names in the form of <cluster-name>-srv-xx for servers and <cluster-name>-cli-xx for the clients, respectively. These names help the VMs to be addressed easily when using DNS.

Here is an example of how a multi-node K3S cluster is launched using Shikari.

$ shikari create -n k3s -s 1 -c 1 -t shikari-k3s.yaml

$ shikari list
CLUSTER       VM NAME          STATUS        SCENARIO            DISK(GB)       MEMORY(GB)       CPUS       IMAGE
k3s           k3s-cli-01       Running       k3s-multinode       100            4                4          https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img
k3s           k3s-cli-02       Running       k3s-multinode       100            4                4          https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img
k3s           k3s-cli-03       Running       k3s-multinode       100            4                4          https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img
k3s           k3s-srv-01       Running       k3s-multinode       100            4                4          https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img

To access the K3S cluster, the env helper command populates the KUBECONFIG environment variable.

$ shikari env -n k3s k3s
export KUBECONFIG=/Users/ranjan/.lima/k3s-srv-01/k3s.yaml

$ eval $(shikari env -n k3s k3s)

$ kubectl get nodes
NAME              STATUS   ROLES                       AGE    VERSION
lima-k3s-cli-01   Ready    <none>                      125m   v1.30.3+k3s1
lima-k3s-srv-01   Ready    control-plane,etcd,master   125m   v1.30.3+k3s1

Shikari also has a scale option, which lets you add or remove VMs from either the client or server pool.

$ shikari scale -n k3s -c 3

# added 2 additional worker nodes
$ shikari list
CLUSTER       VM NAME          STATUS        SCENARIO            DISK(GB)       MEMORY(GB)       CPUS       IMAGE
k3s           k3s-cli-01       Running       k3s-multinode       100            4                4          https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img
k3s           k3s-cli-02       Running       k3s-multinode       100            4                4          https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img
k3s           k3s-cli-03       Running       k3s-multinode       100            4                4          https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img
k3s           k3s-srv-01       Running       k3s-multinode       100            4                4          https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img

$ kubectl get nodes
NAME              STATUS   ROLES                       AGE    VERSION
lima-k3s-cli-01   Ready    <none>                      125m   v1.30.3+k3s1
lima-k3s-cli-02   Ready    <none>                      125m   v1.30.3+k3s1
lima-k3s-cli-03   Ready    <none>                      125m   v1.30.3+k3s1
lima-k3s-srv-01   Ready    control-plane,etcd,master   125m   v1.30.3+k3s1

Internals

Networking

All scenarios are expected to use socket_vmnet networking mode. I found this to be the best option when running multi-node, where the nodes can talk to each other and the services can also be accessed from the host.

DNS

The images used are pre-configured to use systemd-resolved with mDNS support. For scenarios using upstream images, the lima template provisioners make these necessary configurations. This allows every node to talk to the others using DNS names. Eg: lima-k3s-srv-01.local, lima-k3s-cli-01.local etc.

Scenarios

Ref: https://github.com/Ranjandas/shikari-scenarios

The tool is accompanied by a scenarios repository (shikari-scenarios) that contains the templates for creating clusters. This repository also includes a packer file that bakes an image that contains all the software required to launch the clusters. This choice was made so that the internet speed does not slow down the launch of multiple VMs while packages are installed on startup.

Demo

Here is an asciicast of launching a Kubernetes cluster based on the k3s-multinode scenario.

[AkihiroSuda]

Nice!

All scenarios are expected to use socket_vmnet networking mode. I found this to be the best option when running multi-node, where the nodes can talk to each other and the services can also be accessed from the host.

I'd rather suggest user-v2 network, as it doesn't need the root privilege on the host and works on Linux hosts too

[Ranjandas]

Thank you so much for your input.

I started using user-v2 initially but soon ran into issues due to how port mapping works. Here are some of the concerns:

Shikari caters to use cases such as spinning up clustered components (e.g., Kubernetes, Nomad, Consul, etc.). This means that multiple nodes would have services listening on the same port, which results in port-mapping complexities such as:

• Only one of the nodes port will be exposed to the host (whichever comes up first)
• It isn't easy to figure out which VM a port is mapped to
• The launch of the VMs will be full of port-mapping failure warnings for the rest of the VMs
• To connect to other services on non-port mapped VMs will have to be done by getting a shell into one of the other VMs (and connecting cross-vm)
• If there is already a cluster running (with existing port-mapping), a new cluster would not get exposed to the host at all (unless I define custom port mapping in the template)

These are all the reasons why I ended up picking socket_vmnet. It provides more traditional networking without port-mapping magic.

• It allows cross-vm networking.
• It allows direct access to ports on all VMs
• I can altogether disable port mapping (and guest agent for that matter), save some startup time, and get cleaner logs during startup

With my current experience using socket_vmnet, I wish it had the following additional features:

• Native DNS resolution for VM names (like user-v2). Currently, I am configuring systemd-resolved and using mDNS.
• Linux Support. This is one of my major disappointments. Currently, my scenarios are stuck on macOS due to socket-vmnet. 😢
• Being able to run without root privilege (this is not something that affects me that much, tbh)

The only major issue we are having with socket_vmnet is that sometimes the VMs don't get an IP Address and we haven't been able to figure out what is causing this. Sometimes a simple restart will solve the issue, and sometimes, deleting bridge100 and restarting is required. I had a recent encounter where whatever we tried, the VM was not getting an IP and I gave up. A systematic guide on troubleshooting this would be a great addition.

I am open to suggestions for the problems above, or any input that would make the experience better. Once again, thank you for the awesome project.

[afbjorklund]

One workaround for the Linux scenario is using a simple "socket" network, which extends slirp to multiple machines:

• Allow setting up "socket" network for VM-to-VM communication #682

It doesn't help much with access the individual machines directly, but you could use some other VPN solution for that.

https://norouter.io/docs/getting-started/vpn/

That is, the SSH access to the instances is still handled by the default port 22 tunneling and they all have the same IP...

The secondary interface is unique per instance, and is where they access the other cluster machines but not outside.

One of the machines is the "main", and hosts the private network socket that the other qemu machines connect to.

In the Kubernetes cluster scenario this was typically the (primary) control-plane node, as shown in the demo (#1185).