Recursive virtualization

In order to fully test the KVM / Qemu / Spice stack, it will be useful to have virtual machines that use it. I can put them in a number of locations, but one thing that I wanted to try was to put them inside my Linux VM, using recursive virtualization.

What is recursive virtualization

Recursive virtualization is the ability to intall a virtual machine within a virtual machine. This is not entirely trivial to implement. A virtual machine monitor, for example, typically reserves part of the address space by “stealing bits” from the host address space. You can only do that so many times.

Activating recursive virtualization in VMware

Since VMware Fusion offers recursive virtualization, I thought it was very tempting to see how well it worked, and to install yet another virtual machine, but this time from within the VM. Another reason for doing that is that I am currently doing many things on the OSX side, and I’d rather not reboot all the time just to get to my Linux-hosted VMs.

Enabling recursive virtualization is just one click away. While I’m at it, I also added profiling support. All this is probably not too good for performance, but that’s not my objective right now.


Creating a VM with Boxes

Since I’m still a bit in my “exploration” phase, I wanted to see how well Boxes did under these circumstances. I NFS-mounted my Fedora 25 ISO image, connected that to my VM, and Boxes provided me with default parameters.

The original settings were 2GB of RAM for the guest, and 20GB of disk. Disk space seems a bit short (I ran out quickly last week with 20GB), so I bumped that to 30G. RAM seems a bit excessive, because the “host” (which is really a VM, remember) only has 2G itself. I should probably bump that up, but for now, let’s try with a meager 512M allocation for the guest.

I picked up 512M, which I think is way too low, to see:

  1. If Fedora 25 boots with only 512M. I remember when 512M was more than plenty, but time has passed, so I’m curious where the lower limit is nowadays.
  2. My VM is also currently building a Linux kernel, and is apparently using a good fraction of its 2G.

SE Linux and NFS ISO images

Booting the VM in that configuration fails. I get an SE Linux warning about using NFS for virtualization. The fix is easy:

# setsebool -P virt_use_nfs 1

BUG messages and stuck CPUs

The VM then boots correctly, but the boot shows a number of “BUG” messages complaining that some CPUs have been stuck for over 20 seconds.

NMI watchdog: BUG: soft lockup - CPU #0 stuck for 23s! [kworker/0:1:54]

That’s probably because both my VM and my host are quite busy. Need to wait a bit.

Well, even after unloading both my host and my guest, I still get plenty of BUG messages and stack traces. It really does not look good. This solution does not work well enough yet. I’m hardly surprised 😉

Recovering Windows-hosted Virtual Box VMs

I have a Windows host that has a number of Virtual Box VMs installed on it. I’m trying to revive them. There is a rather old Fedora there (Fedora 22 or 23, I believe). So an alternative for testing Spice is to setup a Fedora 25 guest that I can then access remotely. The machine is distant, it will  probably be a better test of the real benefits of Spice.

I will also try to build the exact same SHA1 for Linux that got me into trouble the first time.

Virtual Box offers an option to use disks using the Qemu format. I’ll pick that, and I’ll see if that helps me transfer VMs around.

Updating Linux to see if the problem goes away

Picked up today’s 4.10-rc3, SHA1 a121103c922847ba5010819a3f250f1f7fc84ab8, hoping that my problem booting in VMware would go away. But apparently, it’s still there.

I guess I should report it, but I don’t have much data to share yet. It’s really strange that the last character on the serial console is always a G. Often GG:

[   11.919074] [TTM] Initializing pool allocator
[   11.923006] [TTM] Initializing DMA pool allocator
[   11.926280] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[   11.931211] [drm] No driver support for vblank timestamp query.
[   11.935264] [drm] Screen Target Display device initialized
[   11.937851] [drm] width 800
[   11.939140] [drm] height 480
[   11.940463] [drm] bpp 32
[   11.950890] [drm] Fifo max 0x00040000 min 0x00001000 cap 0x0000077f
[   11.955451] [drm] Using command buffers with DMA pool.
[   11.962502] [drm] DX: no.

I thought VMware had accumulated the console (I clicked on the “Append” button), but no luck, I apparently have only the last one. Maybe when I edited the Serial port settings?

Will try to build regular v4.9 to see if the problem exists on a standard release. Well, kernel 4.9.0 boots OK, so this is a real kernel bug in this configuration. Will need to file / investigate / bisect.

Best Linux distro?

See this article

Emacs problem with zipped files…

Trying to get Emacs to “compress” and “uncompress” files with a different extension using this tip. There are many files that are really zip files inside, including most Office documents. But the tip does not work, but I don’t know why…

Added this to my .emacs:

(add-to-list 'jka-compr-compression-info-list
              "compressing" "gzip" ("-c" "-q")
              "uncompressing" "gzip" ("-c" "-q" "-d")
              t nil "\037\213"])

The variable jka-compr-compression-info-list is set correctly, but the files with extension .myext don’t open. I tried to modify the original jka-cmpr-hook.el and byte-compile it. Still no joy. I guess the files in question are dumped with the Emacs code, so it does not actually read the file from the filesystem. See this page for a bit of this prehistory, which I happen to know only because I once ported Emacs to OSX

Found the problem with the original tip: in order for the list to be re-evaluated and added to the proper hooks in Emacs, you need to toggle the auto-compression mode. The original tip specified a function that is no longer there to do that, toggle-auto-compression.

So in the end, the required code would be:

(add-to-list 'jka-compr-compression-info-list
              "compressing" "gzip" ("-c" "-q")
              "uncompressing" "gzip" ("-c" "-q" "-d")
              t nil "\037\213"])
(auto-completion-mode nil)
(auto-completion-mode t)

Except that this does not work either, because auto-completion-mode is very smart inside, so the code works if you do it manually, but not in a Lisp script… Aaargh.

So in the end of the end, the correct thing that does work is:

(add-to-list 'jka-compr-compression-info-list
              "compressing" "gzip" ("-c" "-q")
              "uncompressing" "gzip" ("-c" "-q" "-d")
              t nil "\037\213"])



Author: Christophe de Dinechin

I try to change the world, but that's work in progress. If you want to know me, google "Christophe de Dinechin". Keywords: concept programming, virtualization, OS design, programming languages, video games, 3D, modern physics. Some stuff I did that I'm proud of: the first "true" 3D game for the PC, HP's big iron virtualization, real-time test systems for car electronics, some of the best games for the HP48 calculator, a theory of physics that makes sense (at least to me).

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s