Over on the Performance Specialist Group on LinkedIn someone asked “Any pointers for feature comparison between Keynote, Gomez and Webmetrics?”.
My first answer was – “There are a lot more options than those 3””. Take a look at Site Confidence, Watchmouse, CatchPoint, AlertSite, Pingdom etc etc etc.
But it got me thinking about what features people do (or should) look for in a good external website monitoring solution.
I quickly came up with 10, although I am sure there are a lot more:
- How easy is it to use the portal and to create the kind of KPI reports you want. I would recommend starting with the KPI's and the data you want and working backwards! Can they reports be automated and emailed out every day/week/month? Make sure you look at Operations, QA and Dev's requirements. They all have different needs.
- Testing frequency - what interval can you test at? 1min, 5min, 15min? Or it is like Gomez where you might buy "12 tests per hour" but you can't guarantee WHEN in the hour they 12 tests might occur... so you might have a large gap between tests.
- How accurate is the testing? Is it representative of a real user experience? Is it reliable i.e. when testing the same page at the same time do you get consistent timing? If it isn’t reliable it’s useless… if you cant be sure that a change in the reported site performance reflects a real change in how your site is responding then the data is worthless.
- Drill down - can you drill down into waterfall charts and then into each object right down to the HTTP headers?
- False positives - when it sends an alert is something REALLY wrong? Many monitoring services are famous for paging you in the middle of the night when there is really nothing wrong…
- Alerting mechanisms - email, SMS, web service call, twitter, whatever
- Alerting profiles and escalation - how easy is it to send different alerts to different people, with automatic escalation if an alert isn't acknowledged or resolved
- User journeys - single page tests are easy. Robust user journey scripts are harder but they are where the real value lies. Get them to demonstrate monitoring an end-to-end e-commerce journey on your site - search for a product, view product page, add to basket, checkout.
- International monitoring nodes - do you care about user performance in other countries, and does the provider have a monitoring presence in those key countries?
- Support & Professional Services - ask to talk to customers to find out what they think about the service they get from the provider... because often when something goes wrong you want someone on the end of the phone that knows what they are talking about to help you diagnose the problem.
What do you look for? What would be on your list of key features?