Tagged: wsl

minikube and WSL

I develop services that run on Kubernetes. During development minikube provides an convenient way to run a local Kubernetes “cluster” regardless of whether you use Windows, OS X, or a Linux distribution as your host OS.

Day-to-day I use minikube on Windows 10 and I prefer to use the Windows Subsystem for Linux (WSL) bash shell to have a scripting environment consistent with my colleagues, some of whom do not use Windows, and consistent with the CI system.

The Linux binary of minikube isn’t very useful in WSL since it doesn’t support the Hyper-V driver and the Virtualbox driver cannot deal with the path differences it sees within WSL compared to those reported by VboxManage.exe.

However, when running the Windows minikube.exe binary, many of the commands (e.g. start, stop, ip, dashboard) just work without any special configuration. Furthermore, creating a symlink so minikube can be executed on the PATH without the .exe extension easily improves the default experience. Beyond these initial commands though, some extra effort is required.

SSH can be a little flakey with minikube ssh so I find it better to create an alias to use WSL’s ssh client:

ssh -a -i "$(wslpath -u "$(minikube ssh-key)")" -l docker "$(minikube ip)"

You may get an error from this SSH command that the permissions of the identity file are too open. This is fixed in two steps. Firstly ensure you have added metadata to the automount options in your /etc/wsl.conf and restart your WSL session.

Secondly, change WSL’s view of the permissions of the key file:

chmod 0600 "$(wslpath -u "$(minikube ssh-key)")"

The minikube docker-env command doesn’t recognise the WSL environment and outputs the PowerShell environment commands instead. This can be worked around by passing the --shell bash arguments but the DOCKER_CERT_PATH environment variable value won’t work with the docker Linux binary as-is and the Windows binary needs the WSLENV environment variable set appropriately. These extra steps are enough to justify a helper script which I’ve published as a GitHub Gist. With this script dot-sourced, both the Windows and Linux binaries for the Docker client will work with Minikube’s Docker daemon.

Lastly, use the Windows binary for kubectl. The paths for Kubernetes certificates in the .kube/config file make it difficult to use the kubectl Linux binary and so far I haven’t found a problem with the Windows version.

wslpath and mktemp

The Windows Subsystem for Linux (aka WSL or Bash on Ubuntu on Windows) provides a fantastic reproduction of a local Linux environment without needing a virtual machine.

Even better than a virtual machine, WSL includes a lot of conveniences for interoperating with the host Windows file system and processes. That is, I can access my C: drive via /mnt/c/ and I can pop calc via calc.exe.

Naturally, the nature of file paths in Linux and Windows are quite different so WSL performs some translations where it can (e.g. for the current working directory) and provides the wslpath utility for explicit conversions where necessary.

Recently I discovered that even though the root filesystem of my particular WSL installation is accessible from Windows (via %LocalAppData%/lxss/rootfs in my case), WSL will not translate just any path within Linux to a path within this rootfs directory. And this is because WSL is designed with the idea that Windows processes should not modify WSL files.

However I work with various version controlled scripts shared amongst developers on Mac, Linux, and Windows (via Cygwin mostly) that use /tmp/ as a staging area (via mktemp) and when using WSL, Windows processes don’t see this directory. If the current working directory is in /tmp/, the working directory of the Windows process will become the Windows user profile directory instead. And running wslpath -w /tmp/ just returns Result not representable.

To avoid modifying the shared scripts to be WSL-aware, I instead converted my WSL tmp directory to be mounted from the Windows host file system via the following set of commands.

First, define the directory to use as WSL’s tmp, I chose C:\wsltemp\ out of convenience, but it could be any path you prefer.

$ mkdir -p /mnt/c/wsltmp
$ chmod 1777 /mnt/c/wsltmp # tmp dir should have the Sticky-bit set
$ sudo chown root: /mnt/c/wsltmp

Also, to ensure Linux’ case-sensitivity is honoured for this directory, from an elevated PowerShell, run:

> fsutil.exe file setCaseSensitiveInfo c:\wsltmp enable

While NTFS has supported opt-in case-sensitivity for a very long time, it has only recently supported setting it per directory.

Finally, define the mount in WSL and mount it:

printf '\n/mnt/c/wsltmp\t/tmp\tnone\tbind\t0\t0\n' | sudo tee -a /etc/fstab
sudo mount -a

Now your WSL session, and future sessions (assuming you haven’t disabled mountFsTab), will have a /tmp/ directory which will be correctly translated for Windows processes.

Warning: if you use ssh-agent in WSL, mounting /tmp/ to a DrvFS volume instead of LxFS will mean the ssh-agent socket (in /tmp/ssh-*/agent.*) will not be available for WSL processes to connect, it will only be accessible by Win32 processes and therefore not useful for typical scenarios.