Measuring page load performance at a network level

The jetlag has kicked in and I’m sat in my hotel room in Santa Clara wide awake at 4am so I figured I might as well do some more performance testing!

I wanted to do a quick “proof of concept” test of a larger study I want to do comparing different web performance timing methods but for now I wanted to compare Webpagetest (a browser plugin) with Atomic Lab’s Pion (promiscuous-mode network level).

I have a aiCache test environment running on EC2 testing acceleration of aicache.applicationperformance.com so I thought I would use that for my tests. I used Webpagetest.org Canadian IE8 node to test (just a random choice as I was testing some other stuff at the time).

I tested the URL 5x (first view only) and the results were very interesting.

Pion’s network level timings were within 300 – 500ms of the Webpagetest.org timings, which is impressive given that Pion can measure EVERY visit to your website, not just be used a manual test tool a la WPT.

To be fair, there are a number of caveats here – the test page is fairly straightforward HTML without any fancy onLoad AJAX requests etc but as a proof of concept I am happy with that

The raw data comparing WPT timings versus Pion are here:

	Document Complete		Fully Loaded		Pion Timing
Test Number	Time	Requests	Time	Requests	Requests	Time	Difference (seconds)
1	4.961s	52	5.615	53	53	5.070	0.545
2	5.392s	52	5.864	53	53	5.480	0.384
3	5.985s	52	6.454	53	53	6.089	0.365
4	5.188s	52	5.636	53	53	5.263	0.373
5	4.756s	52	5.205	53	53	4.829	0.376

This shows the raw data in Pion (using Pion’s Monitor Events view)

Inter-continental mobile performance testing!

I figured I might as well be productive whilst I was on my way to the airport so I set my phone (Android HTC Desire) to do some automated performance testing using 3PMobile.com’s Browser2.

Since I am on my way to #VelocityConf what better site to test that http://velocityconf.com/velocity2011?

I set it up as the first test on my test page here - http://demo.seriticonsulting.com/3pmobile/index.htm and you can see the Javascript required for the testing below. It tests the URL and then waits 120secs and loops back around. I can stop it at any time from the app UI on the phone.

<script>

function do_velocity_loop_test()

{

var urls = ""; // The list of URLS to AUTOTEST

urls += "http://velocityconf.com/velocity2011\n";

urls += "loop delay 120\n";

js5o9.autotest( urls );

}

</script>

You can see the tests following the route from my home in Guildford, Surrey, to London Heathrow Terminal (about a 35 min drive when the traffic is good).

The results averaged out at around 10secs or so for the 620Kb, 44 object page.

Fast forward about 12 hours and I am in the taxi from SFO to the Avatar Hotel, Santa Clara which will be my home for the next week.

So what’s the performance like roaming on AT&T?

Utterly abysmal is the answer!

My first test, with a reasonable signal strength of 26 ASU, came in 3x slower at 30secs.

The big culprit here, compared to my Vodafone network in the UK, is that the AT&T network still uses the older EDGE network, instead of the much faster HSDPA.

_di_cellular_network_type	=>	EDGE
_di_cell_tower_current_time	=>	06/12/2011 03:05:55 PM PDT
_di_cell_tower_current_id	=>	GSM cid/7651 lac/6039 mcc/310 mnc/410
_di_cell_tower_current_signal_strength	=>	26 asu

You can see the test location below as we drove out of the SFO carpark in the taxi.

The other tests just got slower and slower, taking nearly 2 mins in some cases!

If we look at a test that took 1m50sec we can see from the Webkit timing events (see table at the bottom of the blog) that basically no external events were registered between ~11 seconds and ~88seconds.

Basically the AT&T network took a holiday…

Anyway, once I got to the hotel I decided to do a “back to back” test of the Avatar hotel’s free speed-limited 512Kbps Wifi with the mobile network whilst sat around the pool… and the wifi romped it in with 16.49secs compared to the 55sec it took over the mobile network (still Edge, 16 ASU).

I look forward to doing some more testing later in the week!

Standard WebKit Timing Events (for the 1m50sec test on AT&T)

#	Time Absolute	Time Relative	Time Since Last Event	Event Name	Event Description
1	1307916760779	0	0	loadUrl	http://velocityconf.com/velocity2011/
2	1307916760786	7	7	onNewPicture	hitcount=1 width=320 height=576
3	1307916760789	10	3	onDraw	hitcount=1
4	1307916760801	22	12	onPageStarted	URL=[http://velocityconf.com/velocity2011/]
5	1307916760833	54	32	onProgressChanged	10 percent
6	1307916760833	54	0	onLoadResource	URL=[http://velocityconf.com/velocity2011/]
7	1307916760977	198	144	onDraw	hitcount=2
8	1307916763523	2744	2546	onLoadResource	URL=[http://velocityconf.com/stylesheets/general.css?1307677801]
9	1307916767245	6466	3722	onLoadResource	URL=[http://ajax.googleapis.com/ajax/libs/jquery/1.2.6/jquery.min.js]
10	1307916767257	6478	12	onLoadResource	URL=[http://velocityconf.com/javascripts/application.js?1307677801]
11	1307916767311	6532	54	onLoadResource	URL=[http://velocityconf.com/javascripts/ui/minified/jquery.ui.all.min.js?1307677801]
12	1307916767316	6537	5	onLoadResource	URL=[http://velocityconf.com/javascripts/ui-ext/ui.bgiframe.min.js?1307677801]
13	1307916767320	6541	4	onLoadResource	URL=[http://velocityconf.com/javascripts/ui-ext/ui.stars.pack.js?1307677801]
14	1307916767324	6545	4	onLoadResource	URL=[http://velocityconf.com/javascripts/ui-ext/ui.dimensions.js?1307677801]
15	1307916767329	6550	5	onLoadResource	URL=[http://velocityconf.com/javascripts/ext/jquery.form.min.js?1307677801]
16	1307916767376	6597	47	onLoadResource	URL=[http://velocityconf.com/velocity2011/public/template/css/228/1307121630]
17	1307916767379	6600	3	onLoadResource	URL=[http://assets.en.oreilly.com/1/eventprovider/1/2011v1_common.js]
18	1307916767383	6604	4	onLoadResource	URL=[http://velocityconf.com/velocity2011/public/template/js/228/1307121630]
19	1307916767390	6611	7	onLoadResource	URL=[http://assets.en.oreilly.com/1/eventprovider/1/2011v1_css.css?061011]
20	1307916767437	6658	47	onLoadResource	URL=[http://assets.en.oreilly.com/1/event/60/velocity2011_css.css?03]
21	1307916767441	6662	4	onLoadResource	URL=[http://velocityconf.com/javascripts/superfish.js]
22	1307916767446	6667	5	onLoadResource	URL=[http://assets.en.oreilly.com/1/eventprovider/1/live_modalpanel.js]
23	1307916767500	6721	54	onLoadResource	URL=[http://assets.en.oreilly.com/1/eventprovider/1/print_css.css]
24	1307916767507	6728	7	onProgressChanged	13 percent
25	1307916767907	7128	400	onProgressChanged	14 percent
26	1307916768995	8216	1088	onProgressChanged	15 percent
27	1307916769503	8724	508	onProgressChanged	16 percent
28	1307916770462	9683	959	onProgressChanged	17 percent
29	1307916771234	10455	772	onProgressChanged	18 percent
30	1307916771512	10733	278	onLoadResource	URL=[http://velocityconf.com/stylesheets/themes/enation/enation.all.css]
31	1307916771528	10749	16	onLoadResource	URL=[http://velocityconf.com/stylesheets/ui.stars.css]
32	1307916771580	10801	52	onLoadResource	URL=[http://velocityconf.com/stylesheets/global.css]
33	1307916771892	11113	312	onProgressChanged	19 percent
34	1307916772485	11706	593	onProgressChanged	20 percent
35	1307916772842	12063	357	onProgressChanged	21 percent
36	1307916777808	17029	4966	onProgressChanged	22 percent
37	1307916779863	19084	2055	onProgressChanged	23 percent
38	1307916780821	20042	958	onProgressChanged	24 percent
39	1307916781972	21193	1151	onProgressChanged	25 percent
40	1307916782446	21667	474	onProgressChanged	26 percent
41	1307916782924	22145	478	onProgressChanged	27 percent
42	1307916783335	22556	411	onProgressChanged	28 percent
43	1307916784698	23919	1363	onProgressChanged	29 percent
44	1307916785185	24406	487	onProgressChanged	30 percent
45	1307916785907	25128	722	onProgressChanged	31 percent
46	1307916786765	25986	858	onProgressChanged	32 percent
47	1307916787723	26944	958	onProgressChanged	33 percent
48	1307916788583	27804	860	onDraw	hitcount=3
49	1307916788871	28092	288	onProgressChanged	34 percent
50	1307916789702	28923	831	onProgressChanged	35 percent
51	1307916792961	32182	3259	onProgressChanged	36 percent
52	1307916793519	32740	558	onProgressChanged	37 percent
53	1307916795115	34336	1596	onProgressChanged	38 percent
54	1307916796452	35673	1337	onProgressChanged	39 percent
55	1307916797059	36280	607	onProgressChanged	40 percent
56	1307916798782	38003	1723	onProgressChanged	41 percent
57	1307916800115	39336	1333	onProgressChanged	42 percent
58	1307916841236	80457	41121	onReceivedTitle	Velocity 2011 - OReilly Conferences, June 14 - 16, 2011, Santa Clara
59	1307916841294	80515	58	onReceivedTouchIconUrl	URL=[http://assets.en.oreilly.com/1/eventprovider/1/orm-iphone-icon.png]
60	1307916841362	80583	68	onLoadResource	URL=[http://assets.en.oreilly.com/1/eventprovider/1/scripts_functions.js?012611]
61	1307916841444	80665	82	onLoadResource	URL=[http://assets.en.oreilly.com/1/eventprovider/1/scripts_init.js?012611]
62	1307916841527	80748	83	onLoadResource	URL=[http://www.google-analytics.com/urchin.js]
63	1307916841532	80753	5	onLoadResource	URL=[http://assets.en.oreilly.com/1/event/60/velocity2011_mobile_header.jpg]
64	1307916841599	80820	67	onLoadResource	URL=[http://assets.en.oreilly.com/1/event/14/web2008_live_widget_top.gif]
65	1307916841604	80825	5	onLoadResource	URL=[http://assets.en.oreilly.com/1/event/60/velocity2011_bullet.png]
66	1307916841656	80877	52	onLoadResource	URL=[http://assets.en.oreilly.com/1/event/60/velocity2011_footer_top.gif]
67	1307916841698	80919	42	onLoadResource	URL=[http://assets.en.oreilly.com/1/eventprovider/1/oreilly_conferences_banner.gif]
68	1307916841702	80923	4	onLoadResource	URL=[http://assets.en.oreilly.com/1/eventprovider/1/banner_oreilly_ug.gif]
69	1307916841743	80964	41	onLoadResource	URL=[http://assets.en.oreilly.com/1/event/60/velocity2011_banner_webmetrics.gif]
70	1307916841785	81006	42	onLoadResource	URL=[http://assets.en.oreilly.com/1/event/60/velocity2011_cisco_cloud_banner.gif]
71	1307916841827	81048	42	onLoadResource	URL=[http://assets.en.oreilly.com/1/eventprovider/1/makerfaire11_ny_banner_call.gif]
72	1307916841833	81054	6	onLoadResource	URL=[http://assets.en.oreilly.com/1/event/60/keynote_728x90.jpg]
73	1307916841875	81096	42	onLoadResource	URL=[http://assets.en.oreilly.com/1/event/60/velocity2011_cotendo_banner.jpg]
74	1307916841918	81139	43	onLoadResource	URL=[http://assets.en.oreilly.com/1/eventprovider/1/oreilly-white-logo-footer-transparent.png]
75	1307916841965	81186	47	onNewPicture	hitcount=2 width=930 height=2493
76	1307916841974	81195	9	onDraw	hitcount=4
77	1307916841984	81205	10	onDraw	hitcount=5
78	1307916842991	82212	1007	onProgressChanged	44 percent
79	1307916843438	82659	447	onProgressChanged	45 percent
80	1307916843500	82721	62	onProgressChanged	48 percent
81	1307916843584	82805	84	onProgressChanged	49 percent
82	1307916843673	82894	89	onLoadResource	URL=[http://assets.oreilly.com/js/s_code.js]
83	1307916844135	83356	462	onProgressChanged	51 percent
84	1307916844265	83486	130	onNewPicture	hitcount=3 width=930 height=2493
85	1307916844285	83506	20	onDraw	hitcount=6
86	1307916844288	83509	3	onNewPicture	hitcount=4 width=930 height=2493
87	1307916844293	83514	5	onNewPicture	hitcount=5 width=930 height=2493
88	1307916844295	83516	2	onDraw	hitcount=7
89	1307916844631	83852	336	onProgressChanged	52 percent
90	1307916844633	83854	2	onNewPicture	hitcount=6 width=930 height=2493
91	1307916844666	83887	33	onDraw	hitcount=8
92	1307916844908	84129	242	onProgressChanged	53 percent
93	1307916845187	84408	279	onProgressChanged	54 percent
94	1307916845273	84494	86	onNewPicture	hitcount=7 width=930 height=2493
95	1307916845291	84512	18	onDraw	hitcount=9
96	1307916845515	84736	224	onProgressChanged	55 percent
97	1307916845519	84740	4	onNewPicture	hitcount=8 width=930 height=2493
98	1307916845552	84773	33	onDraw	hitcount=10
99	1307916845630	84851	78	onNewPicture	hitcount=9 width=930 height=2493
100	1307916845641	84862	11	onDraw	hitcount=11
101	1307916846021	85242	380	onProgressChanged	56 percent
102	1307916846145	85366	124	onProgressChanged	57 percent
103	1307916846598	85819	453	onProgressChanged	59 percent
104	1307916848064	87285	1466	onProgressChanged	60 percent
105	1307916848171	87392	107	onProgressChanged	62 percent
106	1307916848780	88001	609	onProgressChanged	64 percent
107	1307916848821	88042	41	onProgressChanged	68 percent
108	1307916848822	88043	1	onNewPicture	hitcount=10 width=930 height=2493
109	1307916849562	88783	740	onProgressChanged	71 percent
110	1307916849574	88795	12	onNewPicture	hitcount=11 width=930 height=2493
111	1307916849846	89067	272	onProgressChanged	72 percent
112	1307916851545	90766	1699	onProgressChanged	73 percent
113	1307916863023	102244	11478	onProgressChanged	74 percent
114	1307916863825	103046	802	onProgressChanged	75 percent
115	1307916864185	103406	360	onProgressChanged	76 percent
116	1307916864312	103533	127	onProgressChanged	77 percent
117	1307916864867	104088	555	onProgressChanged	79 percent
118	1307916865601	104822	734	onProgressChanged	80 percent
119	1307916866165	105386	564	onProgressChanged	81 percent
120	1307916866387	105608	222	onProgressChanged	82 percent
121	1307916866555	105776	168	onProgressChanged	83 percent
122	1307916866711	105932	156	onProgressChanged	85 percent
123	1307916867097	106318	386	onLoadResource	URL=[http://metrics.oreilly.com/b/ss/orvelocity,orglobal/1/H.20.2/s39131448343396?AQB=1&ndh=1&t=12/5/2011%2015%3A14%3A26%200%20420&ns=oreilly&pageName=confs%3Avelocity2011%20home&g=http%3A//velocityconf.com/velocity2011/&cc=USD&ch=confs&events=event2&c1=co... (Continued) Total length = 755
124	1307916867181	106402	84	onLoadResource	URL=[http://www.google-analytics.com/__utm.gif?utmwv=1.4&utmn=1483074935&utmcs=UTF-8&utmsr=320x514&utmsc=32-bit&utmul=en&utmje=0&utmfl=10.1%20r92&utmdt=Velocity%202011%20-%20OReilly%20Conferences%2C%20June%2014%20-%2016%2C%202011%2C%20Santa%20Clara&utmhn=v... (Continued) Total length = 508
125	1307916867272	106493	91	onLoadResource	URL=[http://assets.en.oreilly.com/1/event/60/velocity2011_tab_social_open.png]
126	1307916867339	106560	67	onLoadResource	URL=[http://assets.en.oreilly.com/1/event/60/velocity2011_tab_video_open.png]
127	1307916867409	106630	70	onLoadResource	URL=[http://assets.en.oreilly.com/1/event/60/velocity2011_tab_speakers_open.png]
128	1307916867474	106695	65	onLoadResource	URL=[http://assets.en.oreilly.com/1/eventseries/11/velocity_stay_connected_bottom.png]
129	1307916867531	106752	57	onLoadResource	URL=[http://assets.en.oreilly.com/1/eventseries/11/velocity_stay_connected_top.png]
130	1307916867587	106808	56	onLoadResource	URL=[http://assets.en.oreilly.com/1/eventseries/11/velocity_stay_connected_bg.gif]
131	1307916869991	109212	2404	onProgressChanged	86 percent
132	1307916871391	110612	1400	onProgressChanged	87 percent
133	1307916871441	110662	50	onReceivedIcon	Width=16:Height=16
134	1307916871493	110714	52	onPageFinished	URL=[http://velocityconf.com/velocity2011/]

Measuring the difference “location, location, location” makes to Mobile web performance

The new version of the www.3pmobile.com mobile performance testing app was out yesterday and it has support for automated testing. Given that my wife was taking the car to the garage today for its annual MOT inspection this was a great chance to easily do some real location-dependent mobile testing.

So I installed the performance test browser on her Android HTC Desire, went to the test page and fired off the “30sec Google Loop Test”. This is a simple test that goes to www.google.com, beacons the performance data back to www.3pmobile.com, pauses for 30 secs and then repeats.

Since the performance browser is location-aware via GPS I was able to track the my wife’s journey and see the performance along the route. The pin markers represent a test location, and they are colour coded via the performance thresholds you can see in the legend below.

The variation in performance is truly enormous… the fastest being about 5.2 secs (stationary, just before leaving home, low signal strength (1ASU) but on HSDPA).

The slowest, by comparison… nearly 90secs, whilst moving, (_di_gps_current_speed=>16.75) over GPRS (_di_cellular_network_type=>GPRS). Unusually slow for my wife’s driving, she must have been behind a truck

The Google Map’s view can show you exactly where the person was when the test was taken but, sadly, not whether there was a truck in front of them at the time. Although a enhanced reality version that overlaid video data with page loading performance test data would be cool…

Poor Vodafone doesn’t come off to well in the network rankings but given the smaller number of beta testers at the moment one or two bad tests can probably really skew the ranking data.

It’s hard to draw any “hard and fast” conclusions at this stage about the key factors re location-based performance – we need to crunch some larger datasets to come up with statistically valid data.

One interesting fact – two tests, 90 secs apart, went from ~5sec to ~38sec.

The key difference?

The cell tower changed.

_di_cell_tower_current_id => GSM cid/30340667 lac/206 mcc/234 mnc/15

_di_cell_tower_current_id => GSM cid/30340518 lac/206 mcc/234 mnc/15

The ASU after the change dropped from 23 ASU to 3 ASU (although the network stayed HSDPA).

It’s interesting to speculate on the performance impact cell switching and/or signal strength has on performance.

I look forward to learning “Just Enough Stats to be dangerous” at John Rauser’s session at Velocity and hopefully once we get some larger datasets when can do some correlation.

Ditto hopefully I can talk to some of the Vodafone guys at Dylan’s London Ajax Mobile Conference and find out more about the technicalities of mobile cell handover and data performance.

If you have an Android phone and want to join the beta test programme hop over to www.3pmobile.com and sign up. For “sponsor” pick “Seriti Consulting” and put my name “Steve Thair” in the box below.

Technorati Tags: mobile,3pmobile,mobile web performance,web performance,webperf,Browser2,android mobile web performance,vodafone,velocity Conference,GPRS,HSDPA

Skype call phishing scams!

A new twist on an old scam?

I got a Skype call from someone called “ONLINE HELP” (Dratimonlinehelp) this morning.

Out of curiosity I answered to see what they had to say. A computerised voice told me that “my computer protection was out of date” and I was “at serious risk” if I didn’t go to www.IWantToP0wnYourComputer.com*.

With all phishing scams I like to apply the “Grandma & Grandpa” test and see whether or not my “not that computer literate elderly relatives” would be likely to fall for the scam… and in this case, given how “personal” getting a phone call is (automated or not) compared to a random email (that hopefully gets blocked by spam filters) I’d have to rate this one quite highly.

*(Ok, that wasn’t the real address, but you get my point!)

Technorati Tags: skype,spam,phishing,hacking

Calculating your website’s Apdex using Atomic Lab’s Pion

Apdex (Application Performance Index) is a metric promoted by the Apdex Alliance as a KPI for your website’s performance.

Apdex is pretty easy to understand – you take the response time from your application and divide it into 3 buckets “Satisfied”, “Tolerated” and “Frustrated” that represent a customer’s perspective of the application’s performance.

You then count the number of responses in each category over your preferred sample time and then calculate the Apdex:

Apdex(t) = (Satisfied Count + Tolerated Count / 2) / Total Samples

The alternative formulation is, of course,

Apdex(t) = (Satisfied Count + Tolerated Count / 2) / (Satisfied Count + Tolerated Count + Frustrated Count)

For technical reasons the “Tolerated” threshold is set to four times the “Satisfied” Threshold so if your “Satisfied” threshold (t) was 3 seconds then:

0 to 3 seconds = Satisfied
3 to 12 seconds = Tolerated
over 12 seconds = Frustrated.

So basically you end up with an Apdex score between 0 and 1 that represents a weighted index of how happy your users are with your application.

So what does this mean for your website? Well, if we apply Apdex to web page load time you have a great KPI that you can track and measure over time to see that your website is meeting your customer’s performance needs.

For web page load performance I would set the performance threshold “t” to be 1 second, as my performance benchmark is to always serve pages in under 3 seconds. Sadly in the current version of Apdex I am stuck with a fixed 4x ratio so my “tolerated” threshold becomes 4 seconds but I can live with that for now.

So, for a Apdex(1) for web page performance we’d have these thresholds.

0 to 1seconds = Satisfied
1 to 4 seconds = Tolerated
over 4 seconds = Frustrated.

I was curious to see how easy it would be to build a “real-user experience Apdex dashboard” in Atomic Lab’s Pion so I set about building one. In the end it took about half a day of playing about to get a proof of concept up and running, including a bit of Python scripting help from my friend Jackson at Atomic Labs (thanks Jackson!).

So what does it look like?

Basically there are 4 elements:

(1) a “Transformation” reactor “AssignApdexCategory” to create a new term “ApdexCategory” and assign it a value of “satisfied”, “tolerated” or “frustrated” based on my thresholds.

(2) An “Aggregation” reactor to count the number of occurrences in each unique category “ApdexCategory” (of “satisfied”, “tolerated” or “frustrated”). In this example I count every 1m for 60 minutes

(3) A custom “Python” reactor to do the actual Apdex calculation as per the formula show earlier

(4) Pion can only graph the output of Aggregation reactors, so we just pipe the output event from the Python reactor that contains the “ApdexScore” into an Aggregation reactor so we can create the required graph.

And that’s it!

All that’s left is to create a Dashboard to display the results:

Technically I am cheating a bit, since I am plotting Apdex on a “times 100” scale since 0 to 100 is easier to graph than 0 to 1, but you get the idea.

My “full performance dashboard” showing Apdex, Pageviews, Pageload time averages and Data Start time averages is shown below.

I’ve created a video walkthrough on YouTube to show you exactly how it’s done.

Video walkthrough of the Apdex solution in Pion

If you want to give this a try for yourself just let me know

Technorati Tags: Apdex,web performance,Atomic Labs,Pion,real-user monitoring