Using Web Analytics to Assess CAPTCHA Usability

Recent articles about the impact of CAPTCHAs on conversion rates caused me to look at the web analytics for the CAPTCHA we use on the account signup page for our online game, Warrior Epic .

Like many sites, we use the reCAPTCHA service, although we have customized our implementation to make it fit better with the design of the site.

CAPTCHA on Warrior Epic signup form
CAPTCHA on Warrior Epic signup form

I haven’t done any A/B testing to see how the CAPTCHA is affecting our signup form conversion rate (it’s a little tricky to set up in Drupal).

However, I am able to see to what extent visitors to the page are having difficulty with the CAPTCHA by looking at how often they reload it, request an audio version, or click for help.

Here are the stats. Over a recent one month period:

  • Reload the CAPTCHA and get a new image = about 9% of pageviews.
  • Listen to the audio version of the CAPTCHA = approx. 2% of pageviews.
  • Get help with the CAPTCHA = slightly less than 1% of pageviews.

Of course, the numbers are approximate as someone filling out the form may have reloaded the CAPTCHA multiple times and then listened to the audio version.

However, with the large sample size that we have, it’s fairly safe to assume that most people who had trouble with the CAPTCHA interacted with it once or twice at most.

I can live with the numbers for requesting the audio version and clicking for help. However, nearly one in ten people filling out our signup form had difficulty with the CAPTCHA and had to reload the image.

That seems high to me and worth investigating. Clearly, some people are having difficulty reading it which could be affecting our account creation rates.

Digging further into the analytics I saw that 40% of the CAPTCHA reloads were made by visitors from Turkey (we are actively marketing the game in this country), followed by the US at 22%, and then Germany at 7%.

From this I can conclude that visitors who are using the Turkish version of the signup form are having trouble with the English language CAPTCHA. Which is not all that surprising, really.

So, what to do? One option would be to use a non-text-based CAPTCHA — such as Microsoft’s ASIRRA project — for the Turkish signup form.

Another might be to use a question or calculation (what is 3 + 5?) for these users.