• Scott Brinker

    Good point.

    You’re absolutely right. When you’re trying brand new experiments — rather than systematically tweaking individual elements of an existing page, one at a time — you won’t be able to isolate the individual components that contributed to its success or failure.

    The flip side, however, is that if you only tweak one variable at a time for everything you do, it will take you a very long time to cover new ground. In optimization, there’s also the common “local maximum” problem — that you can get stuck on a suboptimal peak in the performance landscape that you can’t escape by trying one variable at a time. Eric Ries has a great post about this in the context of the Lean Startup.

    I’d humbly suggest that there’s a role for both: using big experiments to move much farther across the landscape of possibilities, and then when you find a winner, using subsequent optimization techniques to hone some of the individual components.

    In computer-based optimization, this could be considered a simulated annealing like algorithm — big jumps to explore the landscape, followed by smaller increments to optimize the winners.

  • Scott Brinker

    Thanks! I love Paper.

  • Dean Marsden

    Thanks for sharing your experiences Scott. People do make such a difference after experiencing working at Koozai, I can say that having the right fit for a company is number 1. Experience can be worked on but a persons attitude is difficult to change.

    Interesting about your experience with multiple pages. I love Rand’s quote about website design, I totally agree although I see some websites where they still get sales despite a really poor design because they are the only seller of a product. I always just imagine how many more sales they could get by redesigning their pages!