Terry Ellison's Blog

A slight case of ME/CFS – four years on

Tue, 07 Feb 2012 15:07:00 +0000

I’ve just finished reading my last update, written just over a year ago. From the ME/CFS perspective, I guess that things are pretty much on a plateau now. For example, my walking range hasn’t improved much over this last year and I still have to be very careful about what I eat, as my food intolerances can punish me if I am lax in this department. The first half of 2011 was constrained by my knee injury, surgery to repair the cartilage (or more accurately cut out the wrecked bits). Knee arthroscopy is now a routine operation, but even so it took about 6 months to recover strength and flexibility in my knee. No doubt this impacted my general mobility and exercise levels. I am still sometimes troubled by knee-pain when sitting or in bed.

So one decision that I did make was to set myself the target of reducing my body weight to what it was when I was 30 years of age. Like most people moving into late middle-age, my weight has been slowly and steadily creeping up over the years; nothing sudden, just the odd pound or so each year. Achieving this target involved a conscious dietary change. Given that I already avoid nearly all processed foods and the wheat family, you might think that there is little more that I could do. However, in essence I have materially eliminated carbohydrates and potatoes from my diet, and increased the amount of vegetables and fruit to compensate. Within six months, I’d lost maybe 25lb, and I’ve lost a few more since. This is largely what Dr Myhill calls a stone age diet, and it’s very similar to one that Prof Terry Wahls described in her very thought provoking TEDTalk: Minding Your Mitochondria. An interesting side-effect of moving onto this diet is that my bowels have really settled down, and I now seem to have no sign of “irritable bowel” symptoms.

It has still been a long slow road to recovery, but I regard myself as lucky because none of the ME/CFS sufferers that I know have made anywhere near the improvments that I have, and my life is a world-away from the year that I spend exhausted and bedridden four years ago.

A quick point about timing

Wed, 01 Feb 2012 18:12:00 +0000

I've just deployed my V3 blog engine to live. In doing so, the time per query jumped from <10 mSec per query to ~2.5s — which meant that the typical page response time for non-cached pages increased from an average 0.25s to nearly 3s. (Cached pages are loaded from a static copy, so the render time is typically under 0.2s. This was odd as the timing for V3 remained much the same as the V2 engine (< 10 mSec) for my development and test versions. After quick 'binary chop' through the code, looking at the micro-timing (I have debug routines to do this), I found that entire increase was down to a single MySQL query which I used in the initialisation of my extension to the mysqli class:

SELECT TABLE_NAME AS name
FROM information_schema.tables
WHERE TABLE_SCHEMA = ''
AND TABLE_NAME LIKE '%'

and this was taking 2.3 – 2.4 seconds to run, so I replaced this by a functional equivalent which ran in < 2 mSec:

SHOW TABLES LIKE '%'

I picked the former because I wanted to explicitly name the column in the result set. However, coding around with the second query only adds an extra line of code. For this extra line, my script time drops by 100x and the user response at the browser decreases by 10x. This all goes to show you can spend ages worrying about whether this particular way of coding (say select case vs. if esleif chains) which only changes runtimes by µSec, but if the runtime suddenly increases in a bizarre manner, then its usually some bizarre quirk that you hadn't anticipated, and you should do a timing drill-down to work out why.

More fun with the Webfusion configuration

Mon, 05 Dec 2011 01:20:00 +0000

This is a follow-up to my article on building a Webfusion test VM. My aim is to create a test environment which reflects my Webfusion environment at a file system organisation and PHP programmatic level. The Webfusion service itself uses name-based dynamically configured mass virtual hosting that is more complex than the examples in this Apache technical note, and its implementation requires patches to the Apache build. (See below for more details). In my case, it is a lot simpler to stick to a standard Debian Apache configuration, with a virtual host for each site to be tested in the /etc/apache2/sites_available directory as in listing 1 below, as this removes the need to add rewrite rules in the section and works with an “out of the box” Apache install. Note that 192.168.1.245 is my static IP address for my VM and this would need updating for another installation. With this configuration, I now have full access to the error and rewrite logs and can now fully debug my applications locally.

The last thing that I need to do is to set up my D/B account (either in phpMyAdmin or by executing the following from the MySQL root account, with the database name, username and password as appropriate):

   CREATE DATABASE dddddddddddddd   CHARACTER SET utf8;
   CREATE USER    'uuuuuuuuuuuuuu'@'localhost' IDENTIFIED BY 'pppppppppppppp';
   GRANT  ALL ON   dddddddddddddd.* TO 'uuuuuuuuuuuuuu'@'localhost';

And also adding the D/B server name (in my case cust_mysql01) as an alias for localhost in /etc/hosts.

Listing 1 – Apache configuration changes to mirror the Webfusion service

# Change the StartServers,MinSpareServers,MaxSpareServers,MaxClients to 2,1,3,50
sed -i -f - /etc/apache2/apache2.conf < /etc/apache2/sites-available/webfusion <
    ServerSignature      On
    UseCanonicalName     Off
    UseCanonicalPhysicalPort Off

    ServerName  ellisons.org.home
    ServerAlias blog.ellisons.org.home test.blog.ellisons.org.home

    AddDefaultCharset ISO-8859-1

    suPHP_Engine on

    DocumentRoot /websites/LinuxPackage02/el/li/so/ellisons.org.uk/public_html

    LogLevel  debug
    LogFormat "02 %{VHOST}e %V %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost
    CustomLog "/var/log/apache2/access.log" vhost

    DirectoryIndex index.html index.php index.htm

    RewriteLog "/var/log/apache2/rewrite.log"
    RewriteLogLevel 6

    Setenv DOCUMENT_ROOT_REAL /websites/LinuxPackage02/el/li/so/ellisons.org.uk/public_html
    RewriteEngine On

    
      AllowOverride All
      Options       SymLinksIfOwnerMatch IncludesNOEXEC Multiviews -Indexes
    

    
        Order deny,allow
        Deny from all
    

    RLimitCPU 15
    RLimitNPROC 7
    RLimitMEM 95000000

END

rm /etc/apache2/sites-enabled/000-default
ln -s ../sites-available/webfusion /etc/apache2/sites-enabled/000-webfusion

Some details on the Webfusion implementation

I have merged in the relevant content of the Webfusion Apache configuration into my VM’s Debian-type Apache configuration given above. The rewrite directives in the Webfusion configuration are perhaps worth a little futher discussion. These were:

     RewriteMap  wfvhost      prg:mod_rewrite_scripts/wf-rewrite-vhost.pl
     RewriteMap  wfgetbase    prg:mod_rewrite_scripts/wf-rewrite-base.pl

(1A) RewriteCond %{ENV:SSL}   ^1$
(1)  RewriteRule .*   -                 [skip=3]
(2)  RewriteRule ^(/websites/(LinuxPackage\d+|symlinks)/../../../([^/]+)/.*)$ \
                      $1                [E=VHOST:${wfvhost:$3},skip=5]
(3)  RewriteRule ^/websites/(LinuxPackage\d+|symlinks) \
                      x                 [F]
(4)  RewriteRule ^/~([^/_@]+[_@])?([^/]+)(/.*)?$ \
                      ${wfgetbase:$2}$3 [E=VHOST:${wfvhost:$2},skip=3]
(5A) RewriteCond %{HTTP_HOST}  ^$
(5)  RewriteRule .*   -                 [F]
(6)  RewriteRule .*   -                 [E=VHOST:${wfvhost:%{HTTP_HOST}}]
(7)  RewriteRule (.*) ${wfgetbase:%{ENV:VHOST}}$1
(8)  RewriteRule .*   -                 [E=DOCUMENT_ROOT_REAL:${wfgetbase:%{ENV:VHOST}}]

where I’ve removied the comment text and numbered the RewriteRules to make skip calculations easier. As far as I can see the Webfusion guys have modified the Apache code to use the environment variable DOCUMENT_ROOT_REAL as a way of dynamically initialising DOCUMENT_ROOT. However, this implementation is unnecessarily inefficient as each cycle can call up to 3 rewrite map functions. Also not only can the user .htaccess rewrites generate additional rewrite cycles, but also rules (2), (4) and (7) can all trigger another rewrite cycle likewise. Hence on my initial system test (which was a close copy of the Webfusion setup), my blog rules were generating 18-21 map calls per request.

Apache implements each prg: map by forking off a child process when the Apache server is started, and the rewriting engine then calls this as an out-of-process function: that is on each map-function lookup Apache sends the input key as a string on a socket connected to the process’s stdin and then reads its returned looked-up value as a string from its stdout. Hence 20 map requests will generate 40 process context-switches. However, the two perl scripts referenced in the maps cache previous lookups in a hash, so even though a database query is still required for any cache miss, any subsequent lookups are quite cheap with the main cost is that these are out of process calls.

However, there should be no need to do more than two calls per request, and most retries can easily be avoided by using the persistent environment variables. (I did a few tweaks to these rules and got this down to 4 map calls per request, but it’s not my job to debug the Webfusion installation.)

Creating a test VM to mirror the Webfusion Shared Service

Tue, 29 Nov 2011 23:02:29 +0000

In previous articles I have discussed how difficult it is for users of a shared web hosting service such as Webfusion's to develop and to debug applications on a service that they're using to provide services such as a blog or a forum to a client community over the web. In my view, there are two main challenges to developing on such 'production' environments:

The supplier of the hosting service will typically optimise the service for normal access – that is for live use – and this usually involves disabling all debug features and diagnostics.
The development process can temporarily 'break' functional code or introduce bugs. You will normally want to keep the existing service running in parallel to this development without its failing.

For these reasons, mature IT organisations invariably separate out development from live 'production' and use multiple environments for development teams, pre-production testing, user trials, production itself and support. Not doing this is asking for trouble and is just bad practice, so I use VMs hosted either on my laptop or on my home Ubuntu server to do all such testing. I find that a standard Ubuntu LAMP stack (such as on my laptop) is good enough for functional development. However this type of LAMP stack is different in subtle but important ways from the Webfusion shared service architecture, so I have also set up a VM configuration which far more closely mirrors the Webfusion set-up.

Just in case others would like to do this, I have used this article to explain how I did this in some detail. Unfortunately it has grown quite large as I’ve documented this. I apologise for its length. To keep the article manageable, I have split it into the following sections: deciding on your requirements, choosing the VM for you and building and configuring the VM. I will also be deferring some content to a later article.

Deciding on your requirements

I would recommend at a minimum for any developer for a web application:

Don't make untested changes to production. Use a development/test environment which is quite separate from the production (and use more if if is necessary to work on multiple aspects which you want to keep separate). Make sure that each such environment appropriately mirrors the live environment. (I'll discuss this later.)
Use some form of development cycle base around this / these environments:
- For each change decide what its scope is and how it needs to be tested.
- Develop and test the change on the development environment.
- When you are confident that the change is working successfully then release the change into production, making sure that you have some form of simple rollback strategy if things go wrong.

Of course you might be able to buy extra environments from your service suppliers, but in my view this is a waste of money and not as effective as having a local environment over you have administrative control. I will be discussing how to use a free VM product to configure a VM to use as a test service for the LAMP-based Webfusion shared services, but I first want to discuss a few issues.

What is an 'appropriate' development environment? At a minimum, you will need to have somewhere to run a LAMP stack. I use Ubuntu for all of my home environments, so my main laptop already supports a LAMP stack, and this is adequate for most purposes. I have a copy of my blog engine here that I use to author articles before "syncing" them to the public service. However if you are running Windows then running Apache/PHP/MySQL over Windows (WAMP) is a pain to set up and is also different to LAMP in many ways, so IMHO, it's just a lot easier to run a LAMP VM. A typical default LAMP stack is not the same as that configured by Webfusion, and these difference can sometimes be significant. It is a lot easier to tailor a dedicated LAMP VM to mirror the target production environment as I do in this article.
Release and Rollback. By 'release', I mean the process of moving a tested set of changes onto your production service. There are many ways to do this, but unless your application is absolutely huge, the simplest method is probably the best. What I have a _private/kits directory, and I copy any releases into this in the form of a .tar.bz2 file. (I could just as easily use a zip file, but using a single compressed file makes the copy a lot easier under FTP or a WebDAV-based explorer.) I also have a simple script for each application that takes the 'release ID' as a parameter and explodes the corresponding kit into the target application directory hierarchy. Rollback is easy: release the previous version using the same script. I use a remote command (as I discuss in this article) to execute this script, but you just as easily create a PHP variant which you invoke over the web and which takes the application, release ID, and enabling password and run as a child process. Remember to ensure that the release script does not overwrite any content subdirectories, if the application hierarchy includes these (e.g. using an --exclude option or equivalent in the script).
Consider using Source Version Control. I use Git in my development cycle, so I typically just keep the current and previous versions on my production server, as I can remake any historic version easily. However, setting up this sort of system can be quite complex, so you might just prefer to keep a local archive of your release kits.

Choosing the VM for you

The first thing that you need to decide is which virtualisation product to use. There are two clear alternatives in my opinion:

VMare Player. This is the simplest end-user VMware product to use within a desktop / laptop environment, and the one that I initially used at home because it's the market leader and I'd used VMware at work. I would avoid the free VMware products as they are either obsolescent or too complex for this type of simple VM application. One advantage of using VMware is its Appliance Store which provides simple pre-build LAMP appliances such as the free TurnKey LAMP Appliance which I discuss below.
VirtualBox (commonly abbreviated to VBox). This is the main market alternative to VMware, and in my view is the easiest product to install and to use on Windows, OS X and most major Linux variants. Unlike VMware Player, VBox includes a fully featured VM development environment. The disadvantage for starters is the lack of an integrated appliance store, but many of the store subscribers offer ISO downloads such as Turnkey's.

I switched to using VBox about five years ago, and I would still recommend it overall because of its features, performance and wide platform support, so the remainder of this article assumes that you will be using VBox as your virtualisation platform.

The next step that you need to do its to understand your vendors hosting environment:

Which versions of Apache / PHP / MySQL are you running. The starting point here is the response from a phpinfo() request as I've discussed in my earlier Webfusion articles, which tells me that I am running but I also ended up running httpd -v, php -v and mysql --version on the server to get the versions (Apache/2.2.3, PHP 5.2.16 and MySQL 5.1.52)
Which PHP and Apache modules are loaded and how they are configured. I exported a tarball of /etc/httpd and /etc/php* to work this out as PHP runs as a cli child image of mod_suphp rather than an in-process library within mod_php.
You need to replicate this closely on your VM and be aware of any material differences (e.g. some nice PHP features such as anonymous functions were only released with PHP 5.3 and don't work under 5.2). However, the main issue here is to cover the feature that you use, e.g. you might decide not to bother with some modules that you don't use in your applications.

Building and Configuring the VM

Step 1: Install the VM product

See VirtualBox Downloads. In the case of Windows and OS X this is an self-installing exe / package which you download and run. In the case of Linux variants the Linux page tells you how to add the Virtual box repository to the sources so that you just use your standard package installer to do the initial install and track any product updates. For this type of VM, VirtualBox is pretty much load and go.

Step 2: Initial install of the VM product

I have used TurnKey's LAMP Appliance over VBox in this example. You will need to tailor this script if you use another media (such as the Ubuntu-supplied 10.04-LTS CD distribution or the Ubuntu 10.04 LTS netboot mini.iso) or use VMware Player as your VM product. The sweet-spot for this appliance is Amazon EC2, but it is still a good starting point for me to use here. The advantages of this appliance is that it is packaged minimal Ubuntu 10.04-LTS LAMP server build (~ 210Mb download compared to 680Mb for the standard Ubuntu ISO), and has some nice management tools preinstalled that novice sysadmins will find useful. There are two possible approaches to using it: the first is to download the OVF version of the appliance, unzip it and use the VBox import function to convert / import into Virtualbox. However, I find it easier just to use the ISO version and do a bare load.

Use the VirtualBox->New Menu option to create a new VM:

Linux/Ubuntu VM;
System: 384 Mb;
Boot:CD+HDD, no floppy;
Storage: the IDE Controller is sufficient, so add a new 2Gb HDD to the IDE primary master (this will leave about 1Gb for your Application + database, so increase this if you need more), add the LAMP distribution ISO to the secondary master; delete the SATA controller;
Network: enable a single adapter bridged;
Audio and USB: disabled;
Everything else is as per the defaults.

Now start the machine and hit enter to boot into the Linux installation. Select "Guided – Use entire disk" as the partitioning method and hit enter. Let the installer split the disk (into the root and swap partitions), select "Yes" and hit enter to continue. Select "Yes" to the question "Install the GRUB boot loader to the master boot record?" The system will now install the LAMP stack, prompt to remove the CD (which you do by hitting right-clt then selecting the Devices->CD->host drive then hitting enter; the sytem will now boot into Ubuntu. This all takes a couple of minutes.

The Turnkey configuration menu first prompts for root and MySQL root passwords. This is a test VM which will only be accessible locally from your PC, so I suggest using a common password for all these as there's little point in high security (but don't do this for live systems). Skip the Turnkey Backup and Migration feature, and skip daily security updates. You will now be presented with the URIs for web access, WebShell, Webmin, PHPMyAdmin, and SSH/SFTP. Now select Advanced->Shutdown to close down the server. When you restart, you now have all of the features that you need to tailor your server to your specific development needs.

Two hints here: (i) you don't need to shut down the VM when you've finished using it; you can instead simply save it to disk by using the Hostkey(Right-Ctl)-Q option in the VBox console window. (ii) You don't need to have a VBox console window running (and keep capturing you mouse if you accidentaly select it) as you can do everything that you need to do with the WebShell, Webmin, PHPMyAdmin, and SSH/SFTP tools provided; instead use the VBoxHeadless -s {vmname} -v off from the command line to start your VM and VBoxManage controlvm {vmname} savestate to save it.

Step 3: Tailor the VM to mirror your production service

At this point you have a good platform to do most of your LAMP development. However in my case the VM is different to the Webfusion service (WfS) in some ways that I accept. However I want to mirror some of these, so I still need to have more work to do:

Apache: (WfS:2.2.3 vs VM:2.2.14). Accept as there are no material function changes between these versions. WfS loads a richer set of authorisation modules but I don't use this functionality. It also uses mod_suphp and mod_expires as opposed to the VM which currently supports mpd_cgi and mod_php, and these have different runtime characteristics so I want to fix up these.
MySQL: (WfS: 5.1.52 vs. VM:5.1.41). Accept as there are no material function changes between these versions.
PHP: (WfS: 5.2.16 vs.VM:5.3.2). Accept as there's little that I can do about it as obtaining a 5.2 version is a pretty stretch. The main issue here is that I need to avoid using the new PHP 5.3 functionality.

So what this really boils down to is that I've got to swap out mod_php for suPHP(this also requires dropping the PHP Xcache accelerator as this isn't supported under suPHP and isn't used on WfS anyway. I also want to add some extra components to make debugging easier and facilitate transfer from VM to the hosting machine and the production VM. My goal is to have the VM look identical to my Webfusion environment at a PHP programming level. I am more relaxed about other differences between the VM and the production service that are visible at a PHP coding level so I have decided not to mirror these (for example the VM only runs one user service and uses an in-VM database and file storage, whereas Webfusion runs thousands of users on a server farm with the dedicated database dedicated servers and the user file storage on Network-attached storage (NAS). The listings below gives the command scripts that I used to tailor the system.

I will discuss copied the /etc updates (for Apache, suPHP, &c) in a following article. You can just paste this into a WebShell window on your server to do this yourself. (You will, of course, need to change this if you are using a service other than Webfusion shared LAMP service.) I am currently only using this to test PHP scripts so I need to think about what I do about python support etc..

Step 3: Install the application on the VM

You need to set up entries in your hosts file to map test versions of your domain onto the VMs IP. I allocate all my VMs static IPs in the address range 198.162.1.128-250, and so this is easy, eg.

192.168.1.245   ellisons.org.home blog.ellisons.org.home files.ellisons.org.home ...

You can now use an SFTP, SSH, WebShell, etc. to access and set up the VM in very much the same way with the target service.

Listings — VM tailor script

The first step is to install some additional tools that I find useful for debugging:

apt-get update
apt-get install linux-image-virtual \
   libapache2-mod-suphp php5-cli php5-gd php5-imagick php5-curl \
   php5-xdebug vim zip unzip bzip2 strace
halt

After restarting the VM booting the the virtual kernel (This will now be the default and is labelled linux-image-generic-pae for some reason), I strip out the Turnkey components that are really aimed at is Amazon EC2 offering:

apt-get remove linux-image-generic php5-xcache libapache2-mod-php5 lvm2 \
   tklbam tklbam-duplicity tklbam-python-boto webmin-tklbam 
apt-get clean
apt-get autoclean
apt-get autoremove
rm /var/cache/debconf/*old /var/cache/apt/*cache.bin
rm /etc/php5/conf.d/{imagick,xcache}.ini
rm -R /etc/tklbam /usr/lib/tklbam
rm /etc/cron.daily/tklbam-backup
DISABLE_MODS="authz_default authz_groupfile cgi perl python reqtimeout ssl"
ENABLE_MODS="auth_digest authn_anon authz_dbm expires headers include rewrite unique_id"
for m in $DISABLE_MODS; do rm /etc/apache2/mods-enabled/$m.*; done
for m in $ENABLE_MODS;  do ln -s ../mods-available/$m.load /etc/apache2/mods-enabled/$m.load; done

Lastly I set up a shadow account so that my working UID / directory is in the same location as on the Webfusion system. Here I just execute this script with parameters /websites/LinuxPackage02/el/li/so ellisons.org.uk 9999999 terrye "Terry Ellison":

NAMEROOT="$1"
DOMAIN="$2"
USERID="$3"
USERNAME="$4"
USERDESC="$5"
DOCROOT=$NAMEROOT/$DOMAIN
addgroup --gid $USERID $USERNAME 
mkdir -p $DOCROOT/logfiles
chgrp $USERNAME $DOCROOT/logfiles
chmod 750 $DOCROOT/logfiles
adduser --uid $USERID --ingroup $USERNAME --home $DOCROOT/public_html --gecos "$USERDESC" $USERNAME

To be be covered in next article: Apache config changes, Mysql account set up, ...

Oh, the embarrassment!

Fri, 25 Nov 2011 00:35:00 +0000

I’ve just discovered a bug in my blog: any comment posts get thrown into the bit-bucket and are never passed to me for acceptance / publishing if the user making the comment is a guest (that is not a blog author) and accessing the public instance on my blog.ellisons.org.uk domain rather than my private and locally installed development system.

I can only plea in mitigation is that this combination made it easier to miss this bug in my module and integration tests. I did a quick check on my access logs (I've got the last 18 months archived), and I've work out how many comment posts were made in this period – roughly 4 a week. If I ignore the 21 spammer probes that were 404'ed, there were a total of 295 reader comments. The comments themselves are gone forever, but appended an article league table for comments, with a cut at 5 comments. My sincere apologies to all these blog readers, and I really regret losing this valuable feedback.

There were a couple of factors that helped my missing this. The first is that I decided to do a complete reimplementation of the blog engine based on some of the conclusions that I've come to in these articles on PHP performance, and I've been working on that on my development system. The second is that I had some heavy commitments in apache.org that took most of my spare time for a couple of months. Both of these will be the subject of future articles. However, I first needed to fix this bug before I could publish this article from my test system to live – otherwise I would have whithered with shame!

A postscript one month on

I continued to track the Apache logs and saw regular POSTs to articles but none were arriving on my administration queue for approval, so I added small diagnostic to dump any POSTs to the article page to a debug file. Yup lots of posts but all spam, which my "simple sum" validation was defeating. Nobody loves me after all :-(

League table of lost comments

Article	No of posts
VBA vs OOo Basic Performance II	46
A PHP Cache Overview and Cache Tuning	27
Building a VMware Appliance Playpen	17
My blog’s templating engine	16
phpBB Performance – Reducing the data cache overhead	16
Migrating the OpenOffice.Org forums to a common code-base	11
phpBB Performance – Reducing the script load overhead	11
VBA vs OOo Basic Performance	11
Using .htaccess files on a Webfusion shared service	10
Access and Permissions on a Webfusion shared service	9
Error Handling in VBScript - Part I	8
Use cases for phpBB	8
An example of how to use VBA/COM over Outlook	7
So why CSS Sprites?	7
Differences in Error Handling between VBA and OOo Basic	6
Performance in a Webfusion shared service and guidelines for optimising PHP applications	6
More on optimising PHP applications in a Webfusion shared service	6
A miserable case of ME/CFS – three years on	5
Putting it all together: my blog engine architecture	5

More on using Rewrite rules in .htaccess files

Mon, 21 Nov 2011 16:58:00 +0000

This article is a further discussion how to use rewrite rules on a shared hosting service (SHS) such as the one supplied by Webfusion and that I use. It develops some earlier discussions in the following blog articles:

The main documentation source is the Apache HTTP server documentation, the .htaccess files tutorial and the documentation on mod_rewrite. The former contains a lot of useful information on use of .htaccess files except anything related to rewrite which is covered by the latter which is now a complete section covering the more detailed aspects in eleven separate sub-sections, three of which are essential reading.

Mod_rewrite reference documentation. This is the standard reference content on the module and its directive definitions.
Introduction to regular expressions and mod_rewrite. This is describes what regular expressions are and how they are used within mod_rewrite. A must-read for beginners.
RewriteRule Flags. A good summary of what all the rewrite flags mean. Note that the DPI flag is only available for Apache versions 2.2.12 and later.

Of the remaining eight, two are useful in specific cases and the rest really only apply where you have access to the base Apache configuration and therefore aren’t so relevant to SHS users.

Using mod_rewrite for redirection and remapping of URLs. This contains some standard patterns for common uses of mod_rewrite, including detailed descriptions of how each works.
Using mod_rewrite to control access. More patterns, this time based on the HTTP_REFERER variable, to deny access or redirect to alternative sites
Dynamic virtual hosts with mod_rewrite
Dynamic proxying with mod_rewrite
Using RewriteMap
Advanced techniques and tricks
When NOT to use mod_rewrite
Technical details

Most of what you need to understand is covered somewhere in these chapters, but it's well worth scanning for “htaccess” and “per-directory” when you go through because this content is fragmented across them. What I then want to do here is to cover the main points about the interaction of the rewrite functionality and .htaccess processing in a single article.

The overall Rewrite architecture

Rewrites fall into two categories:

A system administrator can put them into the Apache configuration (SHS providers will typically do this to map the individual user domains onto the correct root directory) and these will apply to the entire server.
Non-privileged users have to use the directory-specific processing (referred to “Perdir” in the rewrite logs) that interprets the rewrite rules in the .htaccess files.

The rewrite engine essentially does a loop:

do
  execute server rewrites (in the Apache Config)
  execute vhost rewrites (in the Apache Virtual Host Config)
  find the "Perdir" .htaccess file
  if found(.htaccess)
    execute .htaccess rewrites (in the user's directory)
 while rewrite occurred

Note that this loop only terminates after a pass where no rewriting has occurred and thus if you aren’t careful this can result in a loop which terminates with an error.

"Perdir" Rewrites

If you have access to the Apache config (for example when you are buying a VM service such as Webfusion VPS or Amazon EC2) then best practice is to use the server and vhosts rewrites and disable .htaccess use altogether. SHS providers can't do this as users will need to have access to rewrite functionality through "Perdir" .htaccess files and this is this category that I want to discuss in detail in this article.

What access files get processed

Processing of .htaccess files can only be enabled within the Apache configuration, and this is done as standard within SHS offerings. (Note that the filename of the access files can be changed from the default .htaccess in the Apache configuration, but this is rarely done.) When processing a request for any resource, Apache will try to open all possible access files on the path. So in the case of an example, /blog/includes/tinymce/license.txt, it tries to open the following access files:

DOCUMENT_ROOT/.htaccess
DOCUMENT_ROOT/blog/.htaccess
DOCUMENT_ROOT/blog/includes/.htaccess
DOCUMENT_ROOT/blog/includes/tinymce/.htaccess
DOCUMENT_ROOT/blog/includes/tinymce/license.txt/.htaccess

where DOCUMENT_ROOT is the root directory of the user’s directory tree (in my case this is /websites/LinuxPackage02/el/li/so/ellisons.org.uk/public_html). If any of the above files exists then it is read and cached during the processing of the request. Doing a putative open and then handling the error condition if the file doesn’t exist may seem an odd implementation, but it is a cheap operation (in terms of runtime and system overhead) – say 0.1 mS – so this is an efficient way of traversing all possible .htaccess files. Even this last case which might seem an odd one to do, but there is some logic in this: if licence.txt were a directory, then its .htaccess file would need to be processed; the error code in this case is “not a directory” which therefore acts as a file / directory test for licence.txt.

Note that the .htaccess file will only take part in rewrite processing if it includes a RewriteEngine on directive and Options FollowSymLinks has been set in the Apache server configuration (which is always the case in the case of SHS offerings).

My advice is to keep the number of access files to a minimum, and delete any that you don’t need. You can do all the rewrite processing, access control, etc., from a single DOCUMENT_ROOT/.htaccess and I prefer this approach. However, if you have multiple applications under your root (for example I’ve got six including a blog, a test wiki and a forum) then a sensible compromise is to add a second tier of access files with one per application at the next directory level, so that application-specific rewrite processing can be split per application. This also has the benefit that you can create a test directory (hierarchy) to develop changes to your rules, and do this without causing the live applications to fail, but at the cost of an extra .htaccess file open and Perdir processing cycle.

Some Misconceptions

.htaccess rewrites are inefficient. The Apache documentation uses phrases like “incredible”, “these are reached a very long time after the URLs have been translated to filenames”, “mod_rewrite should be considered a last resort”, … This is just silly: the Apache child process starts to process the .htaccess in DOCROOT less than 0.2 mSec after reading GET request, and the entire rewrite processing typically takes a few mSec at most if rewrite logging is disabled (as is always the case on an SHS configuration) and the .htaccess files are precached in the server’s Virtual Filesystem Cache (VFC). To put this time overhead into context, PHP image activation can take a typical 80 mSec, and reading a single file from a network storage mounted directory (the typical implementation for SHS server farm architectures), if not precached in the VFC or NAS cache, can take 200 mSec. So the rewrite overheads are typically at least a couple of orders of magnitude less than PHP image activation and file access overheads. The only material overhead is in reading in the .htaccess file(s), and since these are the ‘hottest’ files in an SHS application they will typically have excellent cache-hit ratios.
.htaccess rewrites are a minefield that are best avoided. Unfortunately SHS application developers have no viable alternative for many Apache functions, and so using this functionality can’t be avoided. Yes, the .htaccess architecture and its implementation are complicated as a result of its evolution, but this also largely compounded by weaknesses in the current documentation, which does a poor job in explaining how to use .htaccess files to implement rewrite functionality.

How PerDir rewrite processing works

Processing is multi-pass. Dropping through the last rewrite or setting a “last” flag on a successful rewrite forces the end of a pass. If any processed rewrite rule has matched and has changed the URI or the query string, then this could mean that another rule might apply, so this is treated as an “INTERNAL REDIRECT” at the end of the pass, and processing is restarted. This cycle is terminated when the last pass makes no change to the URI or the query string.
Each pass uses a single .htaccess file. The deepest .htaccess file on the URI path with RewriteEngine on is used. For historic reasons, this processing is convolved.
By this second stage the URI has been converted into a putative filename based on the path relative to DOCUMENT_ROOT. This is then split into a “Per Dir” part (which includes the trailing /) and a relative URI part (where the leading / is missing). So using an example which is generated initially by a request to http://blog.ellisons.org.uk/includes/tinymce/license.txt:
- HTTP_HOST is blog.ellisons.org.uk
- Per Dir is /websites/LinuxPackage02/el/li/so/ellisons.org.uk/public_html/
- The bare URI is includes/tinymce/license.txt (note the missing leading /).
Any “?” delimiter and request parameters are also stripped from the URI before it is then used as the match string in any RewriteRule statements. Thus you can’t examine request parameters in a RewriteRule.

The rewrite logic consists of a sequence of RewriteRule statements. By default these are executed sequentially until the last one or a [last] flag is set on a successful rule. However the order of rules can be modified by use of flags such as [chain] and [skip].
Any rule can be preceded by one or more RewriteCond statements. These are used if the RewriteRule pattern matches. The condition statements essentially evaluate to a go/no-go against the first parameter which is interpolated, and which can therefore include expanded variables, back-references, etc. (Hence request parameters can only be accessed through using the %{QUERY_STRING} variable.) This is then matched against the second parameter which is a condition pattern. RewriteCond statements are used for two main purposes: to set match variables which can then be used as back references in the associated RewriteRule replacement string or to provide a guard to stop an endless loop substitution. Note that the execution order of the rules and conditions are actually the other way around.
So in this example the root .htaccess applies and if it contained the following the URI would be rewritten as blog/includes/tinymce/license.txt and the flag would trigger internal redirection:

RewriteCond %{ENV:REDIRECT_STATUS} =””
RewriteCond %{SCRIPT_FILENAME}  !-f
RewriteCond %{HTTP_HOST}        (blog|wiki|forum)\.ellisons\.org\.uk   [nocase]
RewriteRule  ^(.*)              %1/$1                                  [last]   

# This is evaluated as follows. Note that the REDIRECT_STATUS environment variable is set by any 
# redirect processing cycle, so this condition is a safety check to ensure that this rule is only
# applied on the first rewrite pass and not on any subsequent loops. 
#
#If URI_pattern.match(‘^(.*)’)   and
#    %{ENV:REDIRECT_STATUS} = ”” and     # This variable which is set at the end of a 
#    %{SCRIPT_FILENAME}  !-f     and
#    %{HTTP_HOST}.match(‘(blog|wiki|forum)\.ellison6\.co\.uk’) then
#    URI_pattern = “%1/$1”   # where %1 is blog, wiki or forum and $1 is the URI from the RewriteRule
#    Break processing
#Endif

A second rewrite pass using DOCUMENT_ROOT/blog/.htaccess now applies with:

Per Dir is /websites/LinuxPackage02/el/li/so/ellisons.org.uk/public_html/blog
The REQUEST_URI match string is includes/tinymce/license.txt

So the rule patterns ($1 etc.) can be used in any condition string. Any condition patterns (%1 etc.) can by used in the next condition string, or (in the case of the last condition pattern) the rewrite rule string.
Remember match patterns are on the left and strings (which can include variable expansion) or the right in the RewriteRule statements, and this is visa-versa in the case of RewriteCond statements: strings (which can include variable expansion) are on the left and match patterns on the right.
Remember to specify a RewriteBase (usually / for your document root .htaccess).
Following a successful rewrite the URI match string is replaced by the interpolated RewriteRule substitution string any following statements.
The rewrite engine will treat any RewriteRule substitution string with leading / as a (new) absolute directory discarding the current relative path. How these are processed varies according to the Apache version, so the safest thing to do if you do want to specify an absolute path is to terminate the current rewrite cycle with a [last] flag and restart a new cycle (with possibly a new .htaccess file).
The rewrite engine will treat any RewriteRule substitution string containing a “?” as setting a new QUERY_STRING which will replace the existing one unless the [qsappend] flag is set in which case the old string is appended.
Hence my advice is to use a relative path (no leading /) for the RewriteRule substitution string wherever practical.

Write down all of the rules that you need, and work out the dependency groups that will require chaining, and order the groups / singletons in approximate order of frequency. If you are a programmer comfortable writing procedural logic then you might find it easier to write a version using if/then/else bracketting first , then manually convert it to the rewrite ladder logic form by adding the necessary [or], [chain], [skip], [next] and [last] flags. Your aim should be to order the rules so that a single pass executes the rewriting (or two passes one root and one application .htaccess in the case of a two level split.

Make sure that you've got the Regexp syntax correct, so that each gives the expected match variables for a set of test patterns.
Don’t forget that Perdir processing is cyclic and this is an intrinsic characteristic of how the engine works. Nonetheless, I regard such looping as a bad practice to be avoided wherever possible.
Make sure that you add the necessary conditions for each rewrite rule (i) to generate any back references needed in the replacement string, and (ii) to prevent unnecessary refiring of the rule on following internal redirection passes.

If you don’t have access to your own test Apache instance where you can set Rewrite debugging, test the rules in a separate test subdirectory, adding them one at a time, because the only diagnostic that will be available to you are the status returns code on failure. Be aware that patterns might fail when you don’t expect them to. For example in the above rule had been “^(.+) %1/$1” then this would fail on the URI http://blog.ellisons.org.uk/ as the match string would be “” and (.+) requires at least one character.

So wrapping this up this example if I adopted a two level .htaccess for my blog, then this would give me this set of rules in the DOCUMENT_ROOT/blog/.htaccess file:

RewriteEngine on
RewriteBase /blog

# If the URI maps to a file that exists then stop. This will kill endless loops

RewriteCond %{REQUEST_FILENAME}     -f
RewriteRule .*                      -                   [last]

# If the request is HTML cacheable (a GET to a specific list and with no query string, the
# user is not logged on) and the HTML cache file exists then use it instead of executing PHP

RewriteCond %{HTTP_COOKIE}                                              !blog_user
RewriteCond %{REQUEST_METHOD}%{QUERY_STRING}                            =GET               [nocase]
RewriteCond %{DOCUMENT_ROOT}/blog/html_cache/$1.html                    -f
RewriteRule ^(article-\d+|index|sitemap.xml|search-\w+|rss-[0-9a-z]*)$  html_cache/$1.html [last]

# Anything else pass to index.php

RewriteRule (.*) index.php?page=$1    [qsappend,last]

One complication here is the use of the RewriteBase. This doesn’t apply to the rewrite rule pattern, but it is added in to the substitution strings and the REQUEST_URI variable. You can see this in an extract of the rewrite log (with debugging enabled) for a request to http://blog.ellisons.org.uk/article-59 as follows (note that I’ve trimmed some of the header fields and replaced the document root by DOCROOT to save space.)

init rewrite engine with requested uri /article-59
pass through /article-59
[perdir DOCROOT/] strip per-dir prefix: DOCROOT/article-59 -> article-59
[perdir DOCROOT/] applying pattern '^(.*)' to uri 'article-59'
[perdir DOCROOT/] RewriteCond: input='' pattern=''  => matched
[perdir DOCROOT/] RewriteCond: input='DOCROOT/article-59' pattern='!-f' => matched
[perdir DOCROOT/] RewriteCond: input='blog.ellison6.home' pattern='(blog|wiki|forum)\.ellison6\.home' [NC] => matched
[perdir DOCROOT/] rewrite 'article-59' -> 'blog/article-59'
[perdir DOCROOT/] add per-dir prefix: blog/article-59 -> DOCROOT/blog/article-59
[perdir DOCROOT/] trying to replace prefix DOCROOT/ with /
strip matching prefix: DOCROOT/blog/article-59 -> blog/article-59
add subst prefix: blog/article-59 -> /blog/article-59
[perdir DOCROOT/] internal redirect with /blog/article-59 [INTERNAL REDIRECT]
[perdir DOCROOT/blog/] strip per-dir prefix: DOCROOT/blog/article-59 -> article-59 
[perdir DOCROOT/blog/] applying pattern '.*' to uri 'article-59'
[perdir DOCROOT/blog/] RewriteCond: input='DOCROOT/blog/article-59' pattern='-f' => not-matched
[perdir DOCROOT/blog/] strip per-dir prefix: DOCROOT/blog/article-59 -> article-59 
[perdir DOCROOT/blog/] applying pattern '^(article-\d+|index|sitemap.xml|search-\w+|rss-[0-9a-z]*)$' to uri 'article-59'
[perdir DOCROOT/blog/] RewriteCond: input='blog_email=XXXXXXXX; blog_user=XXXX; blog_token=XXXXXXXXXXXXXXXX' pattern='!blog_user' => not-matched
[perdir DOCROOT/blog/] strip per-dir prefix: DOCROOT/blog/article-59 -> article-59 
[perdir DOCROOT/blog/] applying pattern '(.*)' to uri 'article-59'
[perdir DOCROOT/blog/] rewrite 'article-59' -> 'index.php?page=article-59'
split uri=index.php?page=article-59 -> uri=index.php, args=page=article-59
[perdir DOCROOT/blog/] add per-dir prefix: index.php -> DOCROOT/blog/index.php
[perdir DOCROOT/blog/] trying to replace prefix DOCROOT/blog/ with /blog
strip matching prefix: DOCROOT/blog/index.php -> index.php
add subst prefix: index.php -> /blog/index.php
[perdir DOCROOT/blog/] internal redirect with /blog/index.php [INTERNAL REDIRECT]
[perdir DOCROOT/blog/] strip per-dir prefix: DOCROOT/blog/index.php -> index.php
[perdir DOCROOT/blog/] applying pattern '.*' to uri 'index.php'
[perdir DOCROOT/blog/] RewriteCond: input='DOCROOT/blog/index.php' pattern='-f' => matched
[perdir DOCROOT/blog/] pass through DOCROOT/blog/index.php

Diagnosing .htaccess usage

There is a simple fact of life here: debugging .htaccess files on a shared service is very difficult, and this is for one main reason: even though Apache server has facilities to carry out detailed logging, enabling logging incurs major performance penalties, so service providers do not enable this through the Apache configuration. So you as an SHS user/developer are left with a process of trial and error where the only error diagnostics that you get is the HTTP status return code if the rewrite fails. Hence I do all of my .htaccess rule debugging on a local test VM which mirrors my SHS environment. Here I can set the RewriteLogLevel to dump diagnostics for the Apache instance to a rewrite.log file. If you can’t do this then another trick is to use environment variables to pass parameters between rules and to the executing script, writing intermediate test strings to environment variables using the [E=VAR:val] constructs which are then accessible by the invoked script as the environment variable REDIRECT_VAR.

Because I use a Linux laptop as my PC, as I’ve discussed on previous articles, I can also use the strace system utility to log all of the system calls executed by the Apache child processes as follows:

# Start an strace on all www-data children
sudo rm /tmp/strace.*
sudo strace  -u www-data -tt -ff -o /tmp/strace $(ps -o "-p %p" h -u www-data)

This strace diagnostic enables me to look at the relative timelines, and file / directory access and the writes to the rewrite.log can then be tied up to the corresponding log records. This information, plus code inspection of the mod_rewrite source enables me to retro-engineer the module’s processing.

I can examine the timelines, what processing takes place, for roughly how long and when, and what I/O takes place. Unfortunately doing this is really in the skill domain of an experienced developer and probably a bit daunting for the average SHS user. However, the reasons that I’ve included this footnote are (i) for those readers who would like to drill down themselves to understand what is going on in the rewrite processing; and (ii) to underline that the recommendations that I make in the preceding section are not simply a matter of opinion, but are underpinned by hard timing and file access data.

HTTP Caching Revisited

Wed, 29 Jun 2011 18:13:04 +0000

I find the issues around of Web performance very interesting. I’ve researched pretty comprehensively and written a few articles in this area. I also routinely use various web tools to instrument sites that I visit to see how they perform, and quite frankly a lot are middling to terrible in performance terms. An example is the MoneyCorp GPS application that I discussed in a previous article, I have subsequently found worse sites (e.g. the RadioTimes TV website which scores 18-25 depending on page), though in these cases the main effect is a case of loading slowly, rather than failing to load at all as in the case of GPS.

However, what I have noticed with some sites is that they seemed to perform reasonably well from a user-perspective even though Google PageSpeed marks them down for not specifying caching parameters. The relevant Google recommendations on “Leverage browser caching” describe the use of the HTTP headers Expires, Cache-Control max-age, Last-Modified and Etag for resources that you wish cached by the client browser, as I have discussed perviously. This guidance really only relates to the mandatory rules for browser caching detailed in RFC 2616 (HTTP/1.1) in section 13.2. However, in addition to these mandatory rules, most browsers also implement an advisory rule that is discussed in section 13.2.4, and which is based on any Last-Modified header if provided. If present, then life of the cached resource is the age at download (the delta of Date and Last-Modified values) divided by a factor X. The factor “X = 10” is suggested in the RFC and this is what IE, Firefox and Chrome use. So by example, if a resource is 10 weeks old at download, then its cached copy will be treated as valid for one week from download.

Apache and IIS both supply Date and Last-Modified headers by default for file-based resources, hence despite the general “best practice” advice on specifying these headers, such file-based resources will frequently be cached.

Conversely, script-based resources will not be unless the code explicitly emits the correct headers.

A good web application spoilt by poor Internet performance

Sat, 11 Jun 2011 18:01:22 +0000

My wife and I have a cottage on the Greek island of Alonissos – at the very top of the village in the photo to the right. We like the remoteness, food, walking, swimming, our terrace with a fantastic view over Aegean and the life-style in general. Internet connectivity isn’t a high priority when I am on the island, so I haven’t gone through all the hassle of getting an ADSL line; I just buy a drink or two at one of the local tavernas and use its WiFi when I need Internet access. This works fine for all the access that I need: managing the websites that I look after, Skype, access to my various bank / credit cards services, YouTube, etc. – with one annoying exception: I need to transfer money routinely from my UK Sterling bank account to my Greek Euro one to cover living expenses.

I use MoneyCorp GPS to do this. MoneyCorp is a market leader in this Forex sector that offers competitive rates and seems to be widely recommended (e.g. by the Telegraph). The GPS application provides all functionality that I need except that it is unusable on these taverna connections: it hangs and times-out. It seems to need a reasonable internet bandwidth to work robustly, and luckily for me there is an Internet service in the main port on the island which offers Enet connection and a reasonable Internet bandwidth, so this an inconvenience rather than a show-stopper. Even so, for an application implementing a money transfer service, I would expect it to provide a functional service 24x7 from pretty much anywhere in the world, given a reasonable minimum Internet service.

So following the theme of my last few articles on optimising web applications for network and browser performance, I decided to drill down into the GPS application to see just why it was the worst application that I had measured with Google Page Speed, scoring 36/100 (my next article will discuss a few more that I have subsequently found). As far as I can see, the designers may have developed a good application in functional terms, but they seem to have ignored some basic rules for web optimisation:

None of the text content is compressed. Rendering the GPS home page requires downloading some 29 HTML, CSS and JS files. None is compressed. This can be enabled by a simple configuration option in the appropriate IIS web.config files, so I am not sure why this hasn’t been done. Compressing these 29 resources totalling 460 KiB would reduce this size by roughly a factor of 10. Some admins cite performance reasons for not doing so, but this is a bogus argument because (i) the CPU overhead on modern Intel CPUs in minimal; (ii) there is a CPU and context switch overhead saving in avoiding the larger network transfers, and (iii), the cost of compression also has to be offset against the saving in only having to encrypt the 10% of the bytestream in the case of HTTPS. This one is a no brainer.
Most static files do not have an expiry date set. There are 22 CSS and JS files which do not have an associated Expires tag; this is again a simple IIS web.config setting. The client browser can safely use any locally cached resources when they are tagged with a future expiry date, without querying the server. If any document has an associated Etag but no valid Expires date, then the browser must issue a conditional request to the server to download if Etag is mismatched. This is in effect an RPC call that optimises out unnecessary downloads. However, even when the download is optimised away, each RPC call itself still has an associated round trip time. Also whilst modern browsers support multiplexing of requests, most limit the concurrent number to six, so issuing these requests will block and serialise other load activities. So again, setting expiries at say now+1 month is a no brainer.
The main document is 80% boilerplate. There is a ticker-tape currency bar on the GPS home page when logged in: occasionally a nice little goodie, but not really core functionality. However, this content is common to all user home pages and it comprises 80% of the document (some 180 KiB out of 223 KiB). Scrolling is already handled by a javascript so why on earth doesn’t the script just do an AJAX-style asynchronous (compressed) XML/JSON load of this table data? The ticker-tape is above this business content so the
for the ticker tape is easiest placed ahead of this content in the HTML body, but there is absolutely no reason to place the ticker-tape script here. Best practice is to place all such furniture scripts at the foot of the body so that their invocation will not block or or prevent the actual main page content loading. Doing this and compressing the remaining document content would reduce the main document size from the current 223 KiB to less than 12 Kib. This is a simple application change using standard functionality within the ComponentArt Web UI toolkit used by this application. There are various other dubious implementation details, and in my experience the current implementation is something that a novice might do.
Marshal CSS and JS files. This is more of a nice-to-have as most users will be repeat visitors for this use-case, and therefore (if static CSS and JS have appropriate expiry dates) the browser will optimise away any such server requests by using locally cached content. Nonetheless, applications are usually more responsive, especially on initial load, if the CSS files are grouped and collated into composite style sheets: 3 big style sheets are better than 22 small ones. Likewise, server-side javascript aggregation helps improve performance. There seem to be two main group of script files: the GPS application ones, and the Web UI ones. It is simpler and better to aggregate and “minify” the former into a compressed composite as a one-off server-side function, and then load this once into the local cache of each user. In the Web UI case, again these are loaded one file per UI component, through I note that the ComponentArt website does marshal Web UI components on its own pages to reduce the numbers of script files. Another benefit of marshalling is that any compression can be done as a pre-process to remove the per-request overhead as I did in my TinyMCE loader.

Having browsed some of the other pages such as the Transfers and Payments functions, I see that javascript components seem to be randomly loaded from either WebResource.axd or ScriptResource.axd, with the latter being compressed, but not the former. Bizarre.

So in previous articles I’ve discussed how to optimise your own web applications and packages such as phpBB. Here we have an example of a corporate B2C application that is undermined by sloppy implementation details and ignoring some basic rules for good web performance:

don't send content over the net that doesn't need to be (i.e. can be cached in the client browser);
set up the transfer defaults so that the browsers and servers don't even need to “talk” about transfer if this isn't needed;
when you do need to send content, make sure that it's compressed;
sensibly glob up content that you do need to send, because this is more efficient and responsive.

I am not sure of the exact reasons for the time-out because I haven’t any access to the server-side logs, but the system is clearly struggling. The secure HTTPS protocol is a must for this type of financial application, and this complicates some techniques for page optimisation. Even so, GPS is sending out 223 – 595 KiB (depending on caching) over this HTTPS stream, when with a few tweaks it needs to send 15 KiB to render the home page, plus another 15 KiB or so asynchronously to initialise the ticker-tape. This poor implementation means that the system traffic is perhaps 5-10x greater than it needs to be, and any poor performance is very much self inflicted. All I hope is that MoneyCorp manage to fix this before the autumn, so that I can do my Forex transactions from my local taverna on my next visit to the island!

The Anatomy and Timing of a Web Request – Part II

Mon, 04 Apr 2011 22:12:26 +0000

In Part I of this analysis, I looked at the overall timeline of viewing a webpage, and my main recommendations were:

The correct webserver configuration is essential to ensure that files are correctly cached at the client’s browser and compressed to ensure that any network transfer times are kept to an absolute minimum when content is transferred over the internet.
The application should be written to minimise the number of supporting files needed to render the page by aggregating content where practical. It should also ensure that when the client browser revalidates scripted content that the script processes the request correctly and issues a status 304 request when appropriate.

Whilst the application changes are beyond the scope of most application installers, getting the webserver correct, through properly configured .htaccess files can easily improve response times by a factor of 3 or more. Having done this though, the application response for the delivery of the main document content becomes the main performance constraint and I want to use phpBB to explore the factors which drive this response time. Whilst I realise that this article is in many ways a reprise of two earlier articles, it is clear from my dialogue on the phpBB developers forum that we continue to talk at cross purposes, so I wanted to drill down and parameterise some of these performance factors to put quantitative numbers of this responsiveness.

To do this, I wrote a small script (see Listing 1) which I ran on local server and used this to “hit” the harness (see Listing 2) at my Webfusion based forum (pointed to by $site) using this script with the output logged to a file. Running the three timings serially like this is safe because the file itself is in the target directory so all of the path Inodes are resolved outside the timing windows, and the three variants load disjoint versions of the phpBB common.php include hierarchy:

the ‘out-of-the box’ load which hierarchically loads 14 source files comprising some 15.5K lines of source;
a compacted version of these 14 files which comprises 1 source file with whitespace collapsed, comments removed and therefore a reduced 11K lines of source;
a variant of the 1 source file where the single source file is gzip compressed and included through a PHP zlib stream.

I let this run for 36 hrs and dropped the results into a spreadsheet for analysis. Figure 1 to the right (click to enlarge) shows a scatter plot and linear regression fits for the three cases with the delay between queries in minutes as the X axis and time taken in mSec as the logarithmic Y axis (Hence the linear trends looking bendy). The formulae give the time in millseconds with N the number of minutes delay.

14 files: 77 + 15 N (Blue X’s and trend line)
1 file: 71 + 0.1 N (Red Squares and trend line)
1 gzip file: 77 + 0.4 N (Yellow Diamonds and trend line)

Figure 2 shows the error histograms around these three linear fits. The 14 file case has a standard deviation of 211 mSec and a typical log-normal skew so the top 10% are > 250 mSec greater than the trend. The variance on the other two cases is bugger-all (3 mSec for the 1 file case), so in other words the response for the 1 file compacted case is a solid 71-75 mSec, the 14 file load is double that at a 5 minute interval, treble at a 10 minute interval and the actual times are all over the place, so the top 10% about 6x the single file load version. The 71 mSec pretty much represents a base load as this inclusion is roughly 75% of the code needed to be compiled.

To put this in context, the complete time to execute the PHP script for, say, viewforum.php is as follows. I have given the ballpark times in mSec for two cases: (a) where phpBB is running on a dedicated server or VM with the system, Apache, PHP and MySQL tuned to memory-cache hot code, tables, and files so as to avoid unnecessary physical I/O, etc, and (b) on a shared service such as Webfusion’s.

Time (mSec) Dedicated	Time (mSec) Shared	Description of Component
0	80	PHP impage activation. Not needed on Dedicated as PHP is permanently mapped into Apache through mod_php
0	70	Compile ~20,000 lines of PHP needed to run the viewtopic function. Not needed on Dedicated as PHP code is cached in Opcode Cache.
0	10 – 50	Fetch cached variables from ACM file cache. Not needed on Dedicated as variables cached in memory ACM cache.
15	15	Execute PHP opcodes needed to run the viewtopic function.
15	30	Execute SQL queries needed to run the viewtopic function. Significantly less on Dedicated as (i) connection to the mysql instance will be by local socket rather than TCP to database server; (ii) the tables will be memory cached on a dedicated D/B.
0	50-500	Delays in loading data from NFS served files. On Dedicated, these will be fully cached in the VFC.
30	30	Apache Handling Overhead and stream compression.
0	0 – 1500	Download any uncached images if Apache caching not properly defined in `.htaccess` files. This will usually be correctly set up on Dedicated.
60	285-2275	Total

The reason that I’ve said “ballpark” about these timings is that I’ve collected them on separate timing tests. One thing that I want to add to phpBB is a DEBUG_TIMING option which can be turned on in config.php, so that these stats can be properly to the error log. Nonetheless, the overall pattern is correct: phpBB is a fairly lightweight application, and the per-request load an a dedicated VM or system should be excellent. However the timings for running it on a shared server are different. The service time roughly 0.3 – 2.5 seconds mainly depending how the forum administrator has configured the service (through the .htaccess file) to maximise caching.

File access is the main performance killer on shared services, since Virtual Filesystem Cache (VFC) is rapidly flushed by contention with the other applications. However, the data storage architecture on Linux services is mitigates this because the VFC acts as a first level cache, with the file system then accessing the shared NAS using an RPC tunnelled NFS protocol. The NAS itself has a large RAM cache, so active files are normally in cache. As a consequence the second order file I/O penalty is that of the RPC transport itself, which is again mitigated because the network drivers including the TCP stack run in kernel mode and the IP Gbit Enet switched infrastructure use jumbo frames, and the NFS (at least on the Webfusion services) is configured with a 32 Kbyte blocksize. Hence the overhead for a VFC cache-miss is RPC to the NAS server and this is only a few milliseconds per I/O if the data content is itself in the NAS cache.

However, because phpBB needs to read some 30+ code and data files per web request in its default configuration, as the gap between transactions increases a higher percentage of these I/Os result in RPC overheads which rapidly aggregate up to become the major component of request response.

Lastly note that all of the times relate to the service time for the request. As soon as the server starts to become resource limited then queuing will add multiplier factors to these delays.

Listing I

file="$site/common_comp.php
for ((i=240; $i; i=$i-1)); do
  sleep $($php -r 'echo rand(1,900);')
  date "+%d %H:%M:%S"
  $wget -o /dev/null -O - "$file?v=0"
  $wget -o /dev/null -O - "$file?v=1"
  $wget -o /dev/null -O - "$file?v=2"
done

Listing 2


//the float version of microtime truncates the LSDs 
function now() {$x = explode(' ', microtime()); return ( ($x[1] % 1000) + $x[0] ); }  
$path=realpath('.');
$common=array('./_debug_common.php', './common.php', "php://filter/zlib.inflate/resource=common.php.gz");
$v = (int) $_GET['v'];
$start = now();
define('IN_PHPBB', true);
$phpbb_root_path = (defined('PHPBB_ROOT_PATH')) ? PHPBB_ROOT_PATH : './';
$phpEx = substr(strrchr(__FILE__, '.'), 1);
header("Content-Type: text/plain");
//echo "including $common[$v]\n"; 
$status = require( $common[$v] );
$elapsed = now() - $start;
printf("%-50.50s %9.6F (require status=%u)\n", $common[$v], $elapsed, $status);

The Anatomy and Timing of a Web Request – Part I

Sat, 02 Apr 2011 13:39:49 +0000

My academic background was mathematics, specialising in operations research and statistics. I put this to good use when I first started my career in an IT consultancy, in working one the development and use of detailed simulations of large-scale Army Command and Control Systems. You might wonder what on earth modelling large voice communications networks over thirty years ago has to do with using modern web services, but in fact the conceptualisation and analytic techniques are very similar, (though the time constants involved have shrunk from seconds to milliseconds). This foundation in modelling and analysing large communicating sequential systems has proved invaluable in my work in systems optimisation during my career, and has influenced my approach to systems engineering.

What I show in this article is that:

Correctly optimising your webserver configuration to ensure that files are correctly cached at the client’s browser and compressed to ensure that any network transfer times are kept to an absolute minimum when content is transferred over the internet.
The application should be written to minimise the number of supporting files needed to render the page by aggregating content where practical. It should also ensure that when the client browser revalidates scripted content that the script processes the request correctly and issues a status 304 request when appropriate.

Using the Google Chrome network analyser

I now want to explore a typical phpBB query in depth, and in one specific case: displaying the phpBB community forum’s board index. To understand what goes on in the interval between a user clicking the board index link and the time taken to complete page is assembled for viewing, you need to have a suitable tool to instrument this process. I recommend using Google Chrome, because the developer tools that you need are part of the standard download (though most modern browsers have an add-on pack which provides similar functionality). The main view that I will use to do this instrumentation is the Network Panel. You can access this when visiting any website with Chrome by typing Shift+Ctrl+i whilst viewing the page.

What I want to do first is to discuss the base case when the browsers local cache is empty. This Chrome explorer window is active so you can click on different options and objects to drill down for more information. Alternatively, you can right-click and select “Export all to HAR” which copies the analysis content to the clipboard in JSON format, then paste this into a test editor and save it to file. I have written a simple PHP filter, (which you can download from here) to convert this HAR format into tab-separated variable (TSV) for loading into Calc or Excel for further analysis (which is what I do). To give you some idea of what this tool does, the screen snapshot to the right shows the display for this index page. (Click on the image to get the full-size version).

If you look at the zoomed figure, then you will see that it took my browser some 5.02 seconds to download the 55 files comprising some 314 Kbytes needed to render the page. Revisiting this page later in the same browsing session took some 1.06 seconds and this time nearly all files were already cached locally on the PC with only 2 files comprising 9 Kbytes needing to be downloaded. If I did a “page refresh” then the time increased to 3.51 seconds but with still only 2 files comprising 9 Kbytes needing to be downloaded.

Note that the figures that I quote here are the main milestone reported by the network tool, and that is when the main page has been rendered and is stable to the viewer. The DOM content load might be an earlier time milestone, but if images of HTML/CSS-undefined size are to be viewed, the view will still jerk around as these load. Further images may continue load and javascript download and fill in content (but this are not visually distracting to the user reading the page) before the “onload” event fires. So the network tool reports all three separately.

The user experience on these extremes is very different: on the first view the page is terribly sluggish and stuttered during loading; on the second it is almost a case of click-2-3-bang and the page was displayed. Note that whilst the exact timing will vary from request to request, the general timing pattern will be similar. I want to look at little more into the loading timeline and consider why were these are so different. In the HTTP 1.1 protocol, the individual requests are multiplexed over shared TCP/IP connections, and by convention, the browser will open up at most two such streams per host site referenced. Each request can be thought of as a Remote Procedure Call (RPC) from the browser, where the input parameters are a set of request headers, and the output is a set of response headers with an optional content.

The timeline for the phpBB communitiy website bulletin index

Timeline (mSec) (cache empty) (pre-cached)	Description
0 - 873 0 - 975 0 – 898	The core index.php request does the bulk of the application processing and outputs HTML document. As pbpBB does all its processing before it generates its output, this appears as delay then a content download from the browser perspective. The download is streamed into the browser parser, and the first thing that it does parse the HTML header, so the header javascript and CSS files referenced can be processed before the content has been loaded.
598 – 2,132 667 - 706 643 - 1.498	The 5 javascript files linked in the header are downloaded. These total 107Kbytes and are uncompressed.
1,903 – 3,204 708 - 732 1,255 – 1,986	The 9 CSS files linked in the header are downloaded. These total 85Kb and the static CSS files are uncompressed. Once these are downloaded the browser now has enough information to start to render the page. This takes a few tens of mSec. If the image tags have defined dimensions then the browser can allocate frame-space to hold each image ahead of loading it and subsequently avoid reflow of the page as images download.
3,246 - 5,018 733 - 1,024 2,043 – 3,658	The 40 files linked in the CSS and the HTML body are downloaded. Image files are already in a compressed format, so no further compression is necessary. These total 124 Kbytes. As each is downloaded, the browser adds it to the page.

There is a big difference for the user between a 1 second and a 5 second delay in rendering a web page. These are typical timings when the phpBB site was relatively lightly loaded. The variance in response means that the worst quartile will be a lot longer than these times. The phpBB admins have got a reasonable set of Apache configuration settings have also configured a Varnish caching accelerator to boost response, but these settings still fall short of optimum. I find the refresh time of ~3.5 seconds quite worrying. A typical phpBB installer using a shared hosting service with default Apache settings would perform at or below this 5 second mark. This is quite unnecessary as the .htaccess files can be used to achieve most of the required performance gains.

I have also got an ODS spreadsheet detail (here) for those interested in looking at the detailed figures. But the main points that I want to emphasise are:

Loading a single web page results in a cascade of secondary requests needed to view the complete page.
Most of these (the static elements) can be cached on the user’s local browser cache and you need to ensure that your web server is configured wherever possible (in the section or .htaccess files on Apache and the web.config file on IIS) to inform the client browser that they can be cached.
You need to be aware that many users configure their browsers to revalidate local cache entries once-per-session or even on every page, so you need to configure the Etag or Last-Modified responses so that the browser can bypass the download. Scripted content can also make use of cache negotiation, but again the script author must let the browser know that it can negotiate by setting these responses, and I cover this point in further detail below..
If the content is being generated by script then remember that users will typically view your site in a burst sequence, visiting maybe 4-6 pages in one session, so script authors need to ask themselves “is this information likely to change on every view, once-per-session, or even very occasionally and set the caching parameters accordingly.
I have configured my OpenOffice.org phpBB forums to maximise possible caching, but the split is still roughly 75% of user page requests which have locally cached all of this “content furniture” vs. 25% of requests which either download or renegotiate these additional files. This 25% miss-rate may seem a good ‘batting average’, but remember that this will trigger 30+ further requests to the webserver.
Whenever possible configure the browser (or PHP in the case of scripts) to compress the output. This has minimal compute overhead on the server, but can make a dramatic impact on perceived user response.
Applications developers should minimise the number of files needed to display a given page, because each separate file brings with it network overheads and per request load on the server. (For example, the Google homepage needs 12 requests, my blog needs 7 requests and the phpBB community board index needs 55 requests.) This is done by aggregating content whenever possible:
- Images used for enriching the page can be grouped together using a technique know as CSS sprites. See So why CSS Sprites? for a discussion of how I did this on my blog.
- Files such as cascading style sheets and javascripts can be aggregated server-side on a one-per-change basis and the aggregate version reference in the HTML page for a single inclusion. (This is the approach adopted by TinyMCE and it uses a PHP server-side loader / aggregator that I helped develop. See Rounding off the use of TinyMCE for WYSIWYG editing for more details.)
The per-request overhead is also dependent on the number of cookies defined for that site as the browser passes a copy of all cookies for a given site back to the server with every request. In the case of phpBB extra 0.5Kb upload for 54 of the requests. For this reason, many sites use a resource-only shadow site to server the static furniture. As this is a second (cookie-less) site, these requests don’t carry this overhead. Also most browsers will parallel up request streams to separate sites. So most high performance sites use this trick. (Google uses ssl.gstatic.com, Wikipedia uses bits.wikimedia.org.)
Applications developers also need to be aware of the inter-dependencies between files needed to render the webpage. These can serialise download activities and extend the total time needed to render the page. Examples here include:
- Load the CSS needed to render the page ASAP, and this means putting all CCS definitions and links in the HTML header. Don’t nest CCS style sheets using the @import directive.
- If Javascript links are placed in HTML header then the browser will load them before rendering the page, so if they only implement action processing (e.g. on user clicks) them move them into the HTML body.

These guidelines are discussed in detail on the Google Web Performance Best Practices page.

Specific comments on the phpBB website

Just looking at this page, I can cast these general observations about phpBB and this site. Given the expertise of the phpBB team, The fact that I can make them about this site shows that they can make quite a difference to site performance if you aren’t careful. This is more an issue of website configuration rather than an application coding one, because my OOo community forums run over 2x faster than the phpBB ones (1.78s vs 5.02s for clear-cache load and 0.47s vs 1.06s for primed-cache load on the trial that I did.)

[Defect] The Apache setting for the phpBB site do not enable compression on static javascript and CSS files. (Note that style.php enables zlib compression, but the Chrome tool doesn’t report this. I did some double checking here because the network delay was not consistent with the reported size. This turns out that if the response does not contain a Content-Size header then Chrome reports the final size not the size transferred. I checked this by commenting out the ob_start('ob_gzhandler'); statement and letting the output fall through to be compressed by the mod_deflate handler . Chrome now reports the transfer as 67.45KB / 12.79KB but the timing is the same.)
[Defect] The phpBB style.php script neither defines nor processed the Etag or Last-Modified responses, so the entire stylesheet can be unnecessarily downloaded on page refresh.
[Enhancement] Javascripts that are linked into the HTML header but should be moved into the HTML body
[Enhancement] The current template downloads (5) multiple javascripts, when these could be marshalled and compressed for a single aggregate inclusion in the HTML body.
[Enhancement] The same aggregator callback approach could be adopted for style sheets to reduce the number and size to be downloaded. Say in this case from 9 files comprising ~150Kbyte to 1 of 40Kbytes.
[Enhancement] The number of images could be significantly reduced using a CSS sprite implementation. They can’t be conveniently used for all images, but nonetheless the the number of images to be downloaded could easily be reduced from 40 to 20.
[Enhancement] Image tags should have their explicit dimensions set in the document HTML.
[Enhancement] By allowing the downloading of static files from a configurable second site, e.g. static.phpBB.com (and this might just be a DNS synonym for forum site), (i) this avoids having to upload these cookies on all static references, (ii) the browser will open extra TCP channels to this site to improve download concurrency.

Programmatic cache negotiation

There are three aspects to ensuring that PHP scripts carry out optimum negotiation with the client browser:

Ensure that the script emits the correct Expires and Cache-Control directives to let the browser know when it can safely cache content/
Generate Etag or Last-Modified headers to facilitate negotiation when the browser needs to revalidate content.
If the browser has previously cached content with either or both of these headers, it will attempt to revalidate content by supplying If-None-Match or If-Modified-Since request fields. Unfortunately the CGI 1.1 specification does guarantee that these will be supplied as, but if HTTP_IF_NONE_MATCH and HTTP_IF_MODIFIED_SINCE, are supplied then the script can use these to bypass unnecessary processing and return a 304 status code (Not Modified).

What still constrains performance is the server-side internal overheads of responding to this request, and this will be the subject of my next article.