Yet another technical blog

First, some background:

We run a number of production VoIP PBXs, a number of which are Asterisk-based.

Historically one of the tricky things about pure-VoIP Asterisk deployments (those which talk pure SIP and IAX2) is that you still need a timing source if you want to do any conferencing, audio-playback or IAX2 trunking.

Asterisk doesn’t implement a timing source natively, but calls on DAHDI (formerly known as Zaptel) to achieve this.

For some time, Zaptel and now DAHDI ships with a “dummy” module (formerly zaptel_dummy, now dahdi_dummy), which uses the “best” available operating system timing source to generate the 1kHz timing signal required by Asterisk.

On production boxes, where timing can be critical, it has been commonplace to use hardware line cards to obtain the 1kHz timing signal, even though no analogue lines are in use. In the UK these can be picked up in PCIe format for around £50, so it’s no big deal – although it does seem quite wasteful. This is what we’ve been doing for almost 3 years. Using hardware cards isn’t without its difficulties (the timing signal reaches DAHDI via interrupts, which isn’t bulletproof), although that’s outside the scope of this post.

If you don’t use a hardware card, The kernel module dahdi_dummy has a series of compile-time checks to determine what the best hardware timing source is. If you’re lucky, and you have a recent Linux kernel (>2.6) and a motherboard built in the last 5 years or so, you get HPET, which is the High-Precision Event Timer. This has sufficient granularity available to generate 1000 interrupts every second without too much difficulty.

With the HPET timer enabled for dahdi_dummy, you can get some pretty workable results, although I probably wouldn’t use this on a production system – we’ve not done enough benchmarking to be sure how it would perform under load.

Now, back to the point of my post:

With such a big push toward virtualisation at the moment, it’s been frustrating to have been unable to virtualise Asterisk PBXs due to the aforementioned timing issue; using mainstream environments such as VMware ESXi, it’s not possible to enable a PCIe hardware timing source, and VMware doesn’t provide a virtualised HPET (this is actually very well documented in this PDF).

However, I’ve recently been doing some reading about VMI extensions (VMware overview here), which in the simplest terms should allow interaction between a guest OS kernel and the host (ESX) kernel. This paravirtualisation not only provides better performance, but it also extends further functionality to the guest OS.

To make things even simpler, the latest Debian distro (Lenny) has the “VMI Paravirt Ops” compiled into the kernel that ships with the OS. If you’re not on Lenny yet this is a pretty decent guide to compiling VMI into your Debian kernel.

What this all means is that you can install the latest Debian release straight into ESXi, tick the VMI checkbox in the virtual machine settings via the VMware Client and you should be up and running with a paravirtualised kernel.

Note: you have to use the i386 (32-bit) Debian kernel, since ESX and ESXi 3.5 use a 32-bit host kernel. Unsurprisingly VMI isn’t operable between different architectures. If you’re wondering how you’re able to virtualise a 64-bit OS on ESXi 3.5, it’s thanks to those natty Intel VT-x and AMD-V extensions.

So, some testing followed:

Having installed a fresh Debian Lenny, I quickly downloaded and compiled DAHDI in order to run some tests on the timing. Bear in mind at this stage I’m just using the out-of-the-box kernel.

These were the initial dahdi_test results:

vmi-test:~# dahdi_test
Opened pseudo dahdi interface, measuring accuracy...
99.902% 99.978% 99.954% 99.883% 99.919% 99.897% 99.964% 99.959%
99.946% 99.802% 99.811% 99.832% 99.971% 99.964% 99.998% 99.924%
99.887% 99.924% 99.946% 99.976% 99.935% 99.917% 99.920% 99.983%
99.850% 99.969% 99.975% 99.952% 99.927% 99.959% 99.967% 99.960%
--- Results after 32 passes ---
Best: 99.998 -- Worst: 99.802 -- Average: 99.929693, Difference: 99.994844

Here’s the dmesg output from dahdi_dummy loading (although be aware that the kernel will still ‘emulate’ an HPET source even when a hardware HPET isn’t available. In VMware guests, this causes some really interesting results!):

vmi-test:/# dmesg | grep dahdi
[  680.700134] dahdi: Telephony Interface Registered on major 196
[  680.702862] dahdi: Version: 2.1.0.4
[  681.768307] dahdi_dummy: Trying to load High Resolution Timer
[  681.768307] dahdi_dummy: Initialized High Resolution Timer
[  681.768307] dahdi_dummy: Starting High Resolution Timer
[  681.768307] dahdi_dummy: High Resolution Timer started, good to go
[  681.795983] dahdi: Registered tone zone 4 (United Kingdom)

Finally here’s the “proof” that VMI was operational:

vmi-test:/# dmesg | grep vmi
[    0.004000] vmi: registering clock event vmi-timer. mult=12551454 shift=22
[    0.193598] Booting paravirtualized kernel on vmi
[    1.067853] vmi: registering clock source khz=2992500

Conclusion?

It’s certianly a step forward from previous attempts that I’ve made at DAHDI / Zaptel in a virtualised environment, however the results aren’t so encouraging that I would use it on a production system. We’ll be doing some more testing in due course, and in the mean time we’re saving a couple of these VMs as backup PBXs in the event of hardware failure elsewhere. It’s just too versatile not too.