Social Buzz
« Skype call phishing scams! | Main | How to measure the performance of the Top 100 UK e-tailers in 11min 28secs. »
Wednesday
Jun082011

Calculating your website’s Apdex using Atomic Lab’s Pion

Apdex (Application Performance Index) is a metric promoted by the Apdex Alliance as a KPI for your website’s performance.

Apdex is pretty easy to understand – you take the response time from your application and divide it into 3 buckets “Satisfied”, “Tolerated” and “Frustrated” that represent a customer’s perspective of the application’s performance.

You then count the number of responses in each category over your preferred sample time and then calculate the Apdex:

Apdex(t) = (Satisfied Count + Tolerated Count / 2) / Total Samples

The alternative formulation is, of course,

Apdex(t) = (Satisfied Count + Tolerated Count / 2) / (Satisfied Count + Tolerated Count + Frustrated Count)

For technical reasons the “Tolerated” threshold is set to four times the “Satisfied” Threshold so if your “Satisfied” threshold (t) was 3 seconds then:

0 to 3 seconds = Satisfied
3 to 12 seconds = Tolerated
over 12 seconds = Frustrated.

So basically you end up with an Apdex score between 0 and 1 that represents a weighted index of how happy your users are with your application.

So what does this mean for your website? Well, if we apply Apdex to web page load time you have a great KPI that you can track and measure over time to see that your website is meeting your customer’s performance needs.

For web page load performance I would set the performance threshold “t” to be 1 second, as my performance benchmark is to always serve pages in under 3 seconds. Sadly in the current version of Apdex I am stuck with a fixed 4x ratio so my “tolerated” threshold becomes 4 seconds but I can live with that for now.

So, for a Apdex(1) for web page performance we’d have these thresholds.

0 to 1seconds = Satisfied
1 to 4 seconds = Tolerated
over 4 seconds = Frustrated.

I was curious to see how easy it would be to build a “real-user experience Apdex dashboard” in Atomic Lab’s Pion so I set about building one. In the end it took about half a day of playing about to get a proof of concept up and running, including a bit of Python scripting help from my friend Jackson at Atomic Labs (thanks Jackson!).

So what does it look like?

image

Basically there are 4 elements:

image

(1) a “Transformation” reactor “AssignApdexCategory” to create a new term “ApdexCategory” and assign it a value of “satisfied”, “tolerated” or “frustrated” based on my thresholds.

image

(2) An “Aggregation” reactor to count the number of occurrences in each unique category “ApdexCategory” (of “satisfied”, “tolerated” or “frustrated”). In this example I count every 1m for 60 minutes

image

(3) A custom “Python” reactor to do the actual Apdex calculation as per the formula show earlier

image

(4) Pion can only graph the output of Aggregation reactors, so we just pipe the output event from the Python reactor that contains the “ApdexScore” into an Aggregation reactor so we can create the required graph.

image

And that’s it!

All that’s left is to create a Dashboard to display the results:

image

Technically I am cheating a bit, since I am plotting Apdex on a “times 100” scale since 0 to 100 is easier to graph than 0 to 1, but you get the idea.

My “full performance dashboard” showing Apdex, Pageviews, Pageload time averages and Data Start time averages is shown below.

image

I’ve created a video walkthrough on YouTube to show you exactly how it’s done.

Video walkthrough of the Apdex solution in Pion

 

If you want to give this a try for yourself just let me know Smile

 

Reader Comments

There are no comments for this journal entry. To create a new comment, use the form below.

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>