I'm glad that Jason has decided to start throwing data behind his design decisions. It's certainly easy to go overboard on data collection, but a little good data is better than no data at all.

Quick story - a company I used to work for decided to A/B test their signup page. Everyone in the company could submit their "version" of the page. In the end, the best performer was the secretary, while the worst performer was the marketing guy.

Not surprising. To do sales well you really have to be able to step back and think like someone who knows nothing about what you're selling. Hard for someone who's been buried deep in a project for a long time.

One of the reasons I love trade shows. By the end of a few days talking to potential customers at a booth you'll have a rock solid pitch.

Agreed. It also gives a good place to point back to when recommending to go simple rather than extravagant on design decisions.

One caveat in A/B tests -- you have to be careful in deciding what metric you use to pick the winner. They used the number of immediate conversions. But maybe they just increased signups of freeloaders who will abandon the account within 30 days. And maybe not. Another, possibly better, metric would be the number of paying customers after 30 days, rather the number of conversions today.

That's a huge difference in sales from subtle changes in wording and font sizes. Definitely shows the importance of A/B testing.

We played with multivariate testing at the last company I was at, using an expensive platform built by Optimost (now part of Interwoven):

http://www.interwoven.com/components/pagenext.jsp?topic=PROD...

(We also paid a company that did optimization via Taguchi methods, and found them to be a bit of a scam.)

Besides this and Google Website Optimizer, I'd be curious if anyone on HN has used other AB or multivariate testing tools? I would absolutely love something open source that I could plug into Django templates on the server side, rather than depend on external Javascript code to rejigger the page.

Interesting – I wonder what the variance in the “kind” of sign-up might be. Perhaps those designs that highlight the “free” aspects will encourage a lot of try-outs that then bail after the 30 days; while the designs that don’t will see more longer term sign-ups.

The results of a follow-up after 30 days to see how many stay could also be enlightening.

Here is similar article related to design - etc (http://www.boalt.com/blog/2008/10/great-design-is-a-bottom-l...)

Advertising is a medium of information, not a substitute for it. I think the moral is that people love free trials and it's essential to test your landing pages.

I don't follow you... There's not much difference in the amount of information between the different versions.