Has Anyone Every Tested if Testing is Really The Best Way to Figure Out What Works?

Test, Test, Test… That’s all every marketer ever says and hears. Test Everything. The thing is, aside from the simple reality that most people don’t run true scientific tests, and most use the little data they gather as anecdotal evidence to support whatever position they are trying to back up. The Real Truth is that most people don’t want to hear or see the truth. They are afraid it will be too frightening to face.So, I really wonder, is testing really the best way or the only way to arrive at the truth, or more importantly, is it even an accurate reflection of reality? After all, correlation is hardly causation.

Once you get in the murky waters of Search Engine Marketing, like SEO or PPC, where there are literally thousands or perhaps milliosn of variables that can play a role in results, and you can hardly get a true control number to test against, I still wonder if attribution modeling just hasn’t caught up to the cause and effects that are steps removed from the original action. I can probably prove to you that when you run PPC, you will get more organic traffic, but I can hardly prove to you the reason why. I can postulate a dozen reasons, starting with the brand exposure, continuing to search funnels, using different devices or browsers, or other attribution concerns… Yet, so many people insist that tracking is the only way to know what truly works.

Malcolm Gladwell in his book, “BLINK” argues that intuition is much more effective than using data and analytics. He proved that decision making during war games the general in the thick of the battle won over the folks with massive data and analytics tracking everything.

Silicon Valley insists that We measure what we track. Dan Kennedy says never spend a penny on advertising without having a way to track it. I have seen countless companies become a slave to their marketing mix or even their adwords campaigns because they just didn’t quite know what worked and what didn’t.

Personally, I believe tracking everything is a very different concept that Testing everything. When you track to see results. When you Test, you are forced to interpret data through your own filter that may or may not account for what is actually true and real.

Makes me wonder if a business would be better off just looking at their starting balance and ending balance every month and thats it.

I mean, tracking technology costs money. Data is used constantly to support faulty assumptions. analyzing data not only costs money and uses up resources. It could drive fear from doing big things. I know personally, I have missed out of dozens of major opportunities because I was not able to get proper tracking setup to test the campaign. I probably would have been better off spending the money, and seeing if I went out of business or not, and than figuring out how granular to get.

On a more practical level, how many PPC accounts and campaigns got ruined by reactionary decisions to partial data sets driven by rushed decisions?

I am all for testing… But let’s atleast test if Testing itself produces better results!!!