Lately I've found myself repeatedly encouraging startups to use PickFu, so I thought it's time to share that advice with a wider audience. Here's how it works.
You give PickFu $5 and then they get 50 people via Mechanical Turk to do an A/B test for you in the form of a question, e.g. which page/image looks better and why?
In addition to recommending them to others, I've used it twice so far myself. I made the results public so you can see them. First, I asked about two different homepage logo designs. Second, I asked about two different about page designs. (The latter test I paid a bit more, $9 for 100 responses.)
It was interesting to me that in both cases there wasn't a clear winner. My original concern was people would just pick one and you'd expect an equal split in that case. And I'm still not sure that isn't happening.
On the other hand, I found the comments people left somewhat compelling. They make sense and so I'm inclined believe most people are actually taking it seriously.
More to the point though, I found the feedback valuable enough to iterate on the About page design and ended up with more of a hybrid approach. I wouldn't have got there without reading those comments. It's really a cheap and quick form of useful feedback.