RSS Feeds

Creating a test VM to mirror the Webfusion Shared Service

In previous articles I have discussed how difficult it is for users of a shared web hosting service such as Webfusion's to develop and to debug applications on a service that they're using to provide services such as a blog or a forum to a client community over the web.  In my view, there are two main challenges to developing on such 'production' environments:

For these reasons, mature IT organisations invariably separate out development from live 'production' and use multiple environments for development teams, pre-production testing, user trials, production itself and support.  Not doing this is asking for trouble and is just bad practice, so I use VMs hosted either on my laptop or on my home Ubuntu server to do all such testing.  I find that a standard Ubuntu LAMP stack (such as on my laptop) is good enough for functional development.  However this type of LAMP stack is different in subtle but important ways from the Webfusion shared service architecture, so I have also set up a VM configuration which far more closely mirrors the Webfusion set-up. 

Just in case others would like to do this, I have used this article to explain how I did this in some detail.  Unfortunately it has grown quite large as I’ve documented this.  I apologise for its length.  To keep the article manageable, I have split it into the following sections: deciding on your requirements, choosing the VM for you and building and configuring the VM.  I will also be deferring some content to a later article.

Deciding on your requirements

I would recommend at a minimum for any developer for a web application:

Of course you might be able to buy extra environments from your service suppliers, but in my view this is a waste of money and not as effective as having a local environment over you have administrative control.  I will be discussing how to use a free VM product to configure a VM to use as a test service for the LAMP-based Webfusion shared services, but I first want to discuss a few issues.

Choosing the VM for you

The first thing that you need to decide is which virtualisation product to use.  There are two clear alternatives in my opinion:

I switched to using VBox about five years ago, and I would still recommend it overall because of its features, performance and wide platform support, so the remainder of this article assumes that you will be using VBox as your virtualisation platform.

The next step that you need to do its to understand your vendors hosting environment:

Building and Configuring the VM

See VirtualBox Downloads.  In the case of Windows and OS X this is an self-installing exe / package which you download and run.  In the case of Linux variants the Linux page tells you how to add the Virtual box repository to the sources so that you just use your standard package installer to do the initial install and track any product updates.  For this type of VM, VirtualBox is pretty much load and go.

I have used TurnKey's LAMP Appliance over VBox in this example.  You will need to tailor this script if you use another media (such as the Ubuntu-supplied 10.04-LTS CD distribution or the Ubuntu 10.04 LTS netboot mini.iso) or use VMware Player as your VM product.  The sweet-spot for this appliance is Amazon EC2, but it is still a good starting point for me to use here.  The advantages of this appliance is that it is packaged minimal Ubuntu 10.04-LTS LAMP server build (~ 210Mb download compared to 680Mb for the standard Ubuntu ISO), and has  some nice management tools preinstalled that novice sysadmins will find useful.  There are two possible approaches to using it: the first is to download the OVF version of the appliance, unzip it and use the VBox import function to convert / import into Virtualbox.  However, I find it easier just to use the ISO version and do a bare load.

Use the VirtualBox->New Menu option to create a new VM:

Now start the machine and hit enter to boot into the Linux installation.  Select "Guided – Use entire disk" as the partitioning method and hit enter.  Let the installer split the disk (into the root and swap partitions), select "Yes" and hit enter to continue.  Select "Yes" to the question "Install the GRUB boot loader to the master boot record?"  The system will now install the LAMP stack, prompt to remove the CD (which you do by hitting right-clt then selecting the Devices->CD->host drive then hitting enter; the sytem will now boot into Ubuntu.  This all takes a couple of minutes.

The Turnkey configuration menu first prompts for root and MySQL root passwords.  This is a test VM which will only be accessible locally from your PC, so I suggest using a common password for all these as there's little point in high security (but don't do this for live systems).  Skip the Turnkey Backup and Migration feature, and skip daily security updates.  You will now be presented with the URIs for web access, WebShell, Webmin, PHPMyAdmin, and SSH/SFTP.  Now select Advanced->Shutdown to close down the server.  When you restart, you now have all of the features that you need to tailor your server to your specific development needs. 

Two hints here: (i) you don't need to shut down the VM when you've finished using it; you can instead simply save it to disk by using the Hostkey(Right-Ctl)-Q option in the VBox console window. (ii) You don't need to have a VBox console window running (and keep capturing you mouse if you accidentaly select it) as you can do everything that you need to do with the WebShell, Webmin, PHPMyAdmin, and SSH/SFTP tools provided; instead use the VBoxHeadless -s {vmname} -v off from the command line to start your VM and VBoxManage controlvm {vmname} savestate to save it.

At this point you have a good platform to do most of your LAMP development.  However in my case the VM is different to the Webfusion service (WfS) in some ways that I accept.  However I want to mirror some of these, so I still need to have more work to do:

So what this really boils down to is that I've got to swap out mod_php for suPHP (this also requires dropping the PHP Xcache accelerator as this isn't supported under suPHP and isn't used on WfS anyway.  I also want to add some extra components to make debugging easier and facilitate transfer from VM to the hosting machine and the production VM.  My goal is to have the VM look identical to my Webfusion environment at a PHP programming level.  I am more relaxed about other differences between the VM and the production service that are visible at a PHP coding level so I have decided not to mirror these (for example the VM only runs one user service and uses an in-VM database and file storage, whereas Webfusion runs thousands of users on a server farm with the dedicated database dedicated servers and the user file storage on Network-attached storage (NAS).  The listings below gives the command scripts that I used to tailor the system. 

I will discuss copied the /etc updates (for Apache, suPHP, &c) in a following article.  You can just paste this into a WebShell window on your server to do this yourself.  (You will, of course, need to change this if you are using a service other than Webfusion shared LAMP service.)  I am currently only using this to test PHP scripts so I need to think about what I do about python support etc..

You need to set up entries in your hosts file to map test versions of your domain onto the VMs IP.  I allocate all my VMs static IPs in the address range 198.162.1.128-250, and so this is easy, eg.

192.168.1.245   ellisons.org.home blog.ellisons.org.home files.ellisons.org.home ... 

You can now use an SFTP, SSH, WebShell, etc. to access and set up the VM in very much the same way with the target service. 

Listings — VM tailor script

The first step is to install some additional tools that I find useful for debugging:

apt-get update
apt-get install linux-image-virtual \
   libapache2-mod-suphp php5-cli php5-gd php5-imagick php5-curl \
   php5-xdebug vim zip unzip bzip2 strace
halt

After restarting the VM booting the the virtual kernel (This will now be the default and is labelled linux-image-generic-pae for some reason), I strip out the Turnkey components that are really aimed at is Amazon EC2 offering:

apt-get remove linux-image-generic php5-xcache libapache2-mod-php5 lvm2 \
   tklbam tklbam-duplicity tklbam-python-boto webmin-tklbam 
apt-get clean
apt-get autoclean
apt-get autoremove
rm /var/cache/debconf/*old /var/cache/apt/*cache.bin
rm /etc/php5/conf.d/{imagick,xcache}.ini
rm -R /etc/tklbam /usr/lib/tklbam
rm /etc/cron.daily/tklbam-backup
DISABLE_MODS="authz_default authz_groupfile cgi perl python reqtimeout ssl"
ENABLE_MODS="auth_digest authn_anon authz_dbm expires headers include rewrite unique_id"
for m in $DISABLE_MODS; do rm /etc/apache2/mods-enabled/$m.*; done
for m in $ENABLE_MODS;  do ln -s ../mods-available/$m.load /etc/apache2/mods-enabled/$m.load; done

Lastly I set up a shadow account so that my working UID / directory is in the same location as on the Webfusion system.  Here I just execute this script with parameters /websites/LinuxPackage02/el/li/so ellisons.org.uk 9999999 terrye "Terry Ellison" :

NAMEROOT="$1"
DOMAIN="$2"
USERID="$3"
USERNAME="$4"
USERDESC="$5"
DOCROOT=$NAMEROOT/$DOMAIN
addgroup --gid $USERID $USERNAME 
mkdir -p $DOCROOT/logfiles
chgrp $USERNAME $DOCROOT/logfiles
chmod 750 $DOCROOT/logfiles
adduser --uid $USERID --ingroup $USERNAME --home $DOCROOT/public_html --gecos "$USERDESC" $USERNAME

To be be covered in next article: Apache config changes, Mysql account set up, ...