The CIA, Hedgefunds, and Online Marketers All Have This Problem. Here’s The Solution.

Read the words in the triangles above: I’ll explain why in a couple moments.

A few days ago I got a text message from a friend who owns a hedge fund and happened to be in town for the day asking if I am able to get together with him. This friend designs complex investment models for institutional investors including Blackstone. Much of what he does overlaps with web analytics and digital marketing, in the sense that he is analyzing millions of variables, and interpreting that data to make better informed decisions in an uncertain environment and more importantly, an ever changing environment(i.e. every searcher and website visitor is a different person.)

What many people don’t realize is that the CIA operates in the same environment. They are gathering and analyzing data, interpreting it, and making decisions in an uncertain environment.

Ok, remember that picture I shared above with the sentences in the triangles? I’m willing to bet that you actually misread those sentences, just like I did, when I read them originally. (Unless you have seen this before or are incredibly perceptive.)

Now look back at the words in the triangles and see if you can figure out your mistake. If you can’t, I will shed some light in another few moments.

Whether you see the mistake yet or not, the lesson is important, no matter how smart you are, there is a good chance your mindset and its natural filters of the senses can cause you to make a mistake when the truth is staring you in the face clear as day.

In fact, we are bombarded with so many stimuli to your sense all day that we naturally filter out most of it. When it comes to data analysis, especially when the stakes are high, including life or death for CIA analysts, it is extremely important to recognize the existence of these filters and understand them.

Luckily for us marketers and hedge fund managers, the stakes are significantly higher for the CIA and they have invested very heavily into understanding how to analyze and interpret data in an uncertain environment. In fact, many years ago, the CIA commissioned a deep report which is now a book specifically about the analysis of information. It’s called.

The Psychology of Intelligence Analysis  by Richard Heuer

According to my friend from the hedge fund, this book is the best investment book every written, and as I read it, I am leaning towards calling it the best internet marketing and especially media buying and paid search advertising book every written.

The first picture in this post, with the words in the triangles, is taken out straight out of this book. (If you are wondering what your mistake was, all three triangles have a word written twice but most people read it as one instance of the word because our minds expect only one word and are somewhat familiar with these popular phrases. )

As you can see, we can be completely blind to plain as Jane facts sitting right in front of us.

Many moons ago I wrote a blogpost about the marketing mantra: Test, Test, Test. In that post I wondered if someone who doesn’t analyze everything about their marketing might get similar results to someone who does. While my hypothesis was rejected by most, it seems that Chapter 5 of this CIA handbook actually brings proof to a similar concept.

First though, let’s look at a couple more pictures and examples of how our minds can interpret the same data in two different ways.

Look at this picture starting from the first image and what do you see? Now, look at it starting from the last image and what do you see?

If you start from the beginning , you probably see a man’s face in all the images. If you start from the bottom, you probably see a woman in most the pictures.

What this demonstrates is that what you actually see and perceive is directly influenced by what you are EXPECTING to see based on preconception.

The first step in being a great data analyst is to recognize this truth. You can talk about rooting out prejudice and being objective, but those are fantasies that don’t really apply to the real world. What you need is a second opinion, a fresh set of eyes, or one of many different ideas that I haven’t gotten to yet in the Psychology of intelligence analysis book yet.

Here is another, more classic image that I first saw in  Steven Covey’s 7 habits of highly effective people book, but it’s also in this CIA guide on intelligence analysis.

Looking at this picture you can either see a young lady or an old woman. Which one do you see? Now look carefully and see if you can see both. If you start from the top, you likely see a young lady, if you start from the bottom you likely see and old woman.

This exercise is usually prefaced by two groups of people, one shown a regular pic of a young lady and the other a regular pic of an old woman, and than letting them fight with each other over what this picture actually depicts.

The Bottom Line is that You See what Your Mind EXPECTS To See. (Not what you want to see, what you expect to see and it takes conscious effort to change that perception.)

It takes a split second to make a snap judgement, but it takes a herculean effort to change an existing perception.

Hopefully by now you are sold on the idea that you have filters that you are probably not conscious of and therefore can’t really protect against. (most people believe they are very self aware, but very few are aware of all the things that color their perception.) In fact, the solution here is less about knowing what your preconceived perceptions are but rather are about accepting the existence of these filters, allowing you to keep an open mind to second opinions, especially from fresh eyes with no preconceived perceptions about this data.

This leads a much more important point and deep insight into data analysis.

If your perception of something is colored by preconceived notions, than when you get your data incrementally instead of all at one time, the first (incomplete) data you saw, and the snap judgement or even assumption you made about it, will color how and what you perceive from the rest of the data as it comes in.

We already know that we filter out stimuli, the same way we filtered out the double words in the image with the words in triangles, so once you have an idea about something, even if you know its incomplete, you will still be impacted by the filters that see what they expect to see as more information comes in.

This was a particularly difficult challenge for CIA analysts who get the information in piecemeal as it comes in. Luckily, policy makers who only get the report at the end of the data collection have less preconceived notions and anecdotal evidence from this dataset.

The CIA Manual cites a study of horse racing handicappers who were asked to analyze varying amounts of data about a specific race, they were asked to provide the 5 most important factors with their weights, the 2o most, and the 40 most. They than compared their accuracy based on having only those 5 pieces of data or 20 or all 40, and invariably their accuracy was virtually the same.

More Surprising is that their Confidence with the extra data grew exponentially, while their accuracy stayed the same.

In other words they were over-confident with the extra information, but it didn’t improve their accuracy.

What this means is that the first few metrics that they saw led to them buying into some sort of narrative, and the rest of the data was filtered to support that initial assumption.

The critical Question about paid search marketing or any advertising or data analysis is, “Do You Have Too Much Information?”

We all like to believe that knowledge is power and that the more information the better. We are obsessed with big data and fancy modeling, etc…

But, if the net effect of having that extra information is that it only improves our confidence about our interpretations but doesn’t actually impact our interpretation, too much information can lead to mistakes, close mindedness, and potentially poorer results.

Don’t get me wrong, we need the facts…But do we need More Data?

Perhaps the advertising world has got it all wrong. Maybe it’s not about more data or better data but rather better analysts who are more open minded, who seek out second and third opinions, and who recognize the inherent flaws in human perception and analysis and take those into account when making critical business decisions.

I am blown away by the amount of third party data available to advertisers today. Yet, when I asked a friend who is a major media buyer about the effectiveness of using these awesome data sets he replied that as best as he can tell it is never worth the extra cost in the media buying. (first party data is a different story.)

The thing about the above discussion that I found new insight into is not the idea that the data is just too expensive but rather that our flaw is focusing on more data instead of better analysis.

Maybe the programmatic advertising world has got it all wrong. Maybe they are thinking about things backwards. Instead of better targeting abilities and More Data, we need fresher ways to think about digital marketing in 2015.