first lests start by demonstrating the differences between KVM & QEMU
QEmu is a complete and standalone software of its own. You use it to emulate machines, it is very flexible and portable. Mainly it works by a special ‘recompiler’ that transforms binary code written for a given processor into another one (say, to run MIPS code on a PPC mac, or ARM in an x86 PC).
To emulate more than just the processor, Qemu includes a long list of peripheral emulators: disk, network, VGA, PCI, USB, serial/parallel ports, etc.
In the specific case where both source and target are the same architecture (like the common case of x86 on x86), it still has to parse the code to remove any ‘privileged instructions’ and replace them with context switches. To make it as efficient as possible on x86 Linux, there’s a kernel module called KQemu that handles this.
Being a kernel module, KQemu is able to execute most code unchanged, replacing only the lowest-level ring0-only instructions. In that case, userspace Qemu still allocates all the RAM for the emulated machine, and loads the code. The difference is that instead of recompiling the code, it calls KQemu to scan/patch/execute it. All the peripheral hardware emulation is done in Qemu.
This is a lot faster than plain Qemu because most code is unchanged, but still has to transform ring0 code (most of the code in the VM’s kernel), so performance still suffers.
KVM is a couple of things: first it is a Linux kernel module—now included in mainline—that switches the processor into a new ‘guest’ state. The guest state has its own set of ring states, but privileged ring0 instructions fall back to the hypervisor code. Since it is a new processor mode of execution, the code doesn’t have to be modified in any way.
Apart from the processor state switching, the kernel module also handles a few low-level parts of the emulation like the MMU registers (used to handle VM) and some parts of the PCI emulated hardware.
Second, KVM is a fork of the Qemu executable. Both teams work actively to keep differences at a minimum, and there are advances in reducing it. Eventually, the goal is that Qemu should work anywhere, and if a KVM kernel module is available, it could be automatically used. But for the foreseeable future, the Qemu team focuses on hardware emulation and portability, while KVM folks focus on the kernel module (sometimes moving small parts of the emulation there, if it improves performance), and interfacing with the rest of the userspace code.
The kvm-qemu executable works like normal Qemu: allocates RAM, loads the code, and instead of recompiling it, or calling KQemu, it spawns a thread (this is important). The thread calls the KVM kernel module to switch to guest mode and proceeds to execute the VM code. On a privileged instruction, it switches back to the KVM kernel module, which, if necessary, signals the Qemu thread to handle most of the hardware emulation.
One of the nice things of this architecture is that the guest code is emulated in a posix thread which you can manage with normal Linux tools. If you want a VM with 2 or 4 cores, kvm-qemu creates 2 or 4 threads, each of them calls the KVM kernel module to start executing. The concurrency—if you have enough real cores—or scheduling—if not—is managed by the normal Linux scheduler, keeping code small and surprises limited.
when i deployed RHOS using the official documentation, after some struggles everything went fine, but when trying to launch instances i couldn’t the reason was that error:
Error: Failed to perform requested operation on instance "vmtest", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 6259d74b-8692-4b16-a4b1-5d5655ba2b89. Last exception: invalid argument: could not find capabilities for domaintype=kvm ].
the error means that nova is trying to use the hyper visor KVM but that feature is not there in the operating system, which might mean one of two things. first: the processor used doesn’t include the virtualization acceleration module VT for Intel or AMD. and you can check on that by the following command:
egrep -c '(vmx|svm)' /proc/cpuinfo
if the command returned no lines means that the operating system cannot see that processor have virtualization module enabled.
i tried to add the add manually the KVM support for intel, but it was giving an error as follows:
[root@overcloudssl-compute-2 ~]# lsmod | grep kvm [root@overcloudssl-compute-2 ~]# modprobe -a kvm_intel modprobe: ERROR: could not insert 'kvm_intel': Operation not supported
in my case if you can revert back to my article https://messeiry.com/2017/05/24/deploying-redhat-openstack-rhos-on-vmware-esxi-6-0u2/ RHOS is been deployed on top of vmware ESXi 6.0u2. i have enabled the virtualization module on the processor but still it was not yet recognized by the system.
the reason for that is that enabling the CPU/MMU/Virtualization from the vSphere Client is not enough as the configuration parameters have changed between version 5.5 to version 6.0 so you need to do this directly from the ESXi new web interface as following screen.
make sure you check the option Expose Hardware assisted virtualization to the guest OS.
after that restart the compute node or restart the compute service
systemctl restart libvirtd.service openstack-nova-compute.service
now when you check the capabilities you can find that kvm_intel is there
[root@overcloudssl-compute-2 ~]# lsmod | grep kvm kvm_intel 170181 0 kvm 554609 1 kvm_intel irqbypass 13503 1 kvm
now jif you try to create/launch instances it will run buit it will get stuck in booting either by showin “Booting from GRUB” or “Starting …. ” but it will kepp hanging there without a compute boot.
the reason for that is that KVM only can run on baremetal and in my case am running this on my vmware esxi lab.
to work around this we will need to use qemu instead of KVM
Edit the [libvirt] section in the /etc/nova/nova.conf file as follows:
[libvirt] ... cpu_mode=none virt_type = qemu
but lets go back and understand what i have just did, first i changed the virtualization from kvm to qemu, second i specifed to the qemu to choose the default model of CPU instead of reading it from the host.
Specify the CPU model of KVM guests
The Compute service enables you to control the guest CPU model that is exposed to KVM virtual machines. Use cases include:
- To maximize performance of virtual machines by exposing new host CPU features to the guest
- To ensure a consistent default CPU across all machines, removing reliance of variable QEMU defaults
In libvirt, the CPU is specified by providing a base CPU model name (which is a shorthand for a set of feature flags), a set of additional feature flags, and the topology (sockets/cores/threads). The libvirt KVM driver provides a number of standard CPU model names. These models are defined in the
/usr/share/libvirt/cpu_map.xml file. Check this file to determine which models are supported by your local installation.
Two Compute configuration options in the
[libvirt] group of
nova.conf define which type of CPU model is exposed to the hypervisor when using KVM:
cpu_mode option can take one of the following values:
Host model (default for KVM & QEMU)
nova.conf file contains
cpu_mode=host-model, libvirt identifies the CPU model in
/usr/share/libvirt/cpu_map.xml file that most closely matches the host, and requests additional CPU flags to complete the match. This configuration provides the maximum functionality and performance and maintains good reliability and compatibility if the guest is migrated to another host with slightly different host CPUs.
Host pass through
nova.conf file contains
cpu_mode=host-passthrough, libvirt tells KVM to pass through the host CPU with no modifications. The difference to host-model, instead of just matching feature flags, every last detail of the host CPU is matched. This gives the best performance, and can be important to some apps which check low level CPU details, but it comes at a cost with respect to migration. The guest can only be migrated to a matching host CPU.
nova.conf file contains
cpu_mode=custom, you can explicitly specify one of the supported named models using the cpu_model configuration option. For example, to configure the KVM guests to expose Nehalem CPUs, your
nova.conf file should contain:
cpu_mode = custom
cpu_model = Nehalem
nova.conf file contains
cpu_mode=none, libvirt does not specify a CPU model. Instead, the hypervisor chooses the default model.
NOW !! Restart compute services or restart the compute node and everything will work fine.