Magic keywords on Google and the consequences of tailoring results

 

On September 6th, 2012 at 2PM eastern time, 131 DuckDuckGo users searched Google at the same time for abortion, gun control and obama. This was the first in a series of experiments we're doing to explore the state of Google search tailoring -- the fact that different people see different results on Google based on who they are.

Who Cares?

Sometimes tailoring can be good, e.g. we show local weather in our instant answer box. However, in other cases it can put you in a bubble, seeing more and more of what you already agree with, and less and less (relevant and important) opposing viewpoints. This effective filtering is troublesome when looking for raw information on a subject.

For example, consider you're an undecided voter being bombarded with propaganda from both sides. What do you do to research the candidates? If you're like most people, you use Google as your starting point.

Yet when searching Google for raw information, say on gun control, we found people getting very different results. That's why at DuckDuckGo we show everyone the same links by default--so you can escape your personal filter bubble. We believe any tailoring of the organic results should be opt-in and not opt-out.

In a recent Pew Report, 65% of people said personalized search was a "bad thing" since "it may limit the information you get online and what search results you see" compared to just 29% who said it was a "good thing" because "it gives you results that are more relevant to you." When asked a different way 73% said they were "Not OK" with it because it was an invasion of privacy.

Non-Partisan Disclaimer

DuckDuckGo does not have a political affiliation. Individuals within DuckDuckGo have many different (and opposing) political viewpoints, and also live in many different countries.

We chose to use these keywords (abortion, gun control and obama) because they are both a) searches where many people want unbiased results and b) particularly relevant given the 2012 presidential election in the US.

Our first study was initially covered in Talking Points Memo (a thought-of-as-liberal org) and then in the Wall Street Journal (a thought-of-as-conservative org).


Raw Data

We have users worldwide who participated in the study. When you look across countries, people saw all sorts of different things. While still against our policy (we think you should opt-in to region and language tailoring), it is pretty understandable why you might see some variation across countries.

So we concentrated on just the United States as the terms were relevant to the election here. We asked people to take screenshots of their Google results in both a "on" (signed in) and "off" state. The off state in this experiment was mostly signed out, but in some cases included being in private browsing mode or tweaking the various personalization settings on Google.

What you see below is the raw data in terms of how many different orderings people saw in just the first five links, for each of the searches and in both on and off versions. We treated the news block as one link and just counted the first news result as a link (labeled N), even though that masked more variation.


US: abortion (off)
SUBJECTS: 48
ORDERINGS: 11
15      31%     1,2,3,4,5
12      25%     1,2,4,3,5
5       10%     1,2,10,3,11
5       10%     1,2,4,5,N1
3       6%      1,2,10,3,5
2       4%      1,2,10,4,5
2       4%      1,2,4,4,5
1       2%      1,2,10,11,5
1       2%      1,2,10,N2,5
1       2%      1,2,10,11,N1
1       2%      1,2,9,3,5


US: abortion (on)
SUBJECTS: 43
ORDERINGS: 8
11      26%     1,2,3,3,11
10      23%     1,2,4,3,11
6       14%     1,2,10,3,11
6       14%     1,2,4,3,5
4       9%      1,2,4,5,N1
3       7%      1,2,3,4,5
2       5%      1,2,3,4,11
1       2%      2,1,13,10,N1


US: gun control (off)
SUBJECTS: 43
ORDERINGS: 10
19      44%     1,2,3,10,4
9       21%     1,2,3,10,6
6       14%     1,2,3,4,N1
2       5%      1,2,8,9,6
2       5%      1,2,8,9,10
1       2%      1,2,3,4,10
1       2%      1,2,3,10,10
1       2%      1,2,3,N1,10
1       2%      1,2,8,6,N1
1       2%      2,3,10,4,14


US: gun control (on)
SUBJECTS: 40
ORDERINGS: 8
10      25%     1,2,3,10,6
8       20%     1,2,3,10,4
7       18%     1,2,8,9,10
5       12%     1,2,8,9,N1
4       10%     1,2,3,4,N1
3       8%      1,2,8,9,6
2       5%      1,2,3,N1,4
1       2%      1,2,3,4,6


US: obama (off)
SUBJECTS: 49
ORDERINGS: 11
16      33%     N1,2,3,7,8
10      20%     N4,2,3,7,8
9       18%     N4,2,3,4,5
7       14%     N1,2,3,4,5
1       2%      N4,2,3,12,17
1       2%      N9,2,7,8,8
1       2%      N4,2,3,15,16
1       2%      N4,2,3,12,13
1       2%      N1,2,3,12,8
1       2%      N2,2,3,7,8
1       2%      N5,2,3,4,8


US: obama (on)
SUBJECTS: 45
ORDERINGS: 11
12      27%     N4,2,3,4,5
8       18%     N1,2,3,4,5
7       16%     N4,2,3,7,8
5       11%     N4,2,3,4,4
4       9%      N4,2,3,5,4
3       7%      N1,2,3,5,4
2       4%      N1,2,3,7,8
1       2%      N1,2,3,7,4
1       2%      N2,2,3,4,5
1       2%      N9,2,3,4,5
1       2%      N5,2,3,4,5


The number of subjects varied a bit in each group because a) we had to throw out some screen shots that didn't contain enough information and b) we asked people who didn't have Google accounts to just do the "off" state.

I should also note that link ordering matters a lot because people click on the first link way more than the second, and so on. 


Headline Results

  • You cannot turn Google personalization off, as I've explored before. We saw about as much variation in the "off" state as the "on" state. That could be in part due to our users generally not having a lot of data stored on Google, but nevertheless the "off" state showed a lot of variation given that you'd expect very little variation (say, from Google running isolated tests).

  • No ordering received a majority across the whole study, and several orderings were only seen by one or two people. In fact, the data only includes the top five links -- if you open it up to the whole first page (usually 7-11 links) it fragments a lot more.

  • We saw a lot of different links (not just orderings). And we also saw a lot of different news results within the news blocks.

  • The news varied a lot. In the obama search, news was the first link. Some people were getting Fox News while others got the LA Times and a few people got other stuff.

  • Individual people often saw the same things on the off and on versions, but there was of course more variation person to person. That this tailoring exists even when making an attempt to de-personalize (signing out or going incognito) makes it impossible for an individual to pop their own Google filter bubble. Google has been personalizing for signed out users since 2009, but we didn't expect it to be so all-encompassing.

  • We stumbled upon magic keywords.


Magic keywords

As the Wall Street Journal confirmed in its own study, Google has been significantly altering its search results to highlight Obama-related results, but not Romney-related results (more on that later).

These Obama-related results are being inserted because obama is a magic keyword on Google. A magic keyword is a search that can transform the Google results of later searches.

We first noticed magic keywords in the screen shots from our study.  If you use a magic keyword and then search a related topic, you'll likely get results inserted based on the magic keyword. For instance a search for social security after taxes yields this:

social-security-taxes.PNG

As you can see, I got a couple of results inserted based on the intersection of taxes and social security. The first inserted result is labeled (You recently searched for taxes). For some reason, Google doesn't label the other ones it inserts, but you can tell which ones they are because they bold the magic keyword in addition to the search terms (see second red arrow pointing to taxes).


Magic Keyword Variation

Magic keywords create more variation than you might initially expect because from person to person Google personalizes:

  • how many links they insert (we saw 0-4);
  • which links they insert (we saw different ones per search);
  • what order they insert them in (we saw different orderings of the inserted links); 
  • where they insert them (we saw different placements within the page, e.g. after link #3 vs #6).
  • which subsequent searches they transform (different people saw different searches transformed);
  • how many subsequent searches they transform (we saw upwards of ten searches later in our tests).


In other words, magic keywords have their own internal variation, presumably based on personal factors.

Also, we saw magic keyword transformations about equally across the on and off states. This means that if you start in private browsing mode (with cookies initially cleared) and start searching, you're still effected by magic keywords.

We also saw in Google chrome that magic keyword transformations sometimes jumped incognito sessions, meaning that if you started a new incognito mode, got a transformation, then shut it down and started a new incognito mode you could sometimes see the same transformation again (without searching for the same magic keyword again). This weird behavior was not reproducible in Firefox's private browsing mode.


Black Box

Magic keywords are just one of many (perhaps tens) of personalization signals Google uses. Google tailors results based on your search and click history as well as other things they know about you (connected accountsnews / ad preferencesYouTube history, etc.). Since Google changed its privacy policy in Mar, 2012, they can bring all that information to bear on your search results and their other sites.

All of this is a black box, meaning we don't really know what is going on. All we really know is it is happening and it is impossible to escape on Google. We're doing our best to quantify the effects.


Obama, not Romney

After stumbling upon magic keywords, we began to further explore their prevalence. We subsequently found there are a lot of magic keywords related to the upcoming US presidential election, such as taxessocial securityhealth careabortionohio, election and many others. 

We found Obama transforms more subsequent searches than any other magic keyword by far. In the WSJ study they found 80% subsequent variation compared to 25% variation for the keyword election.

Since there is so much overlap between Obama and election issues, it makes sense the obama magic keyword would transform a large subset of election-related queries. It even works on a search for Romney himself!

obama-romney2.png

What's weird is that Romney is not a magic keyword. The gut reaction is duh, Romney isn't searched as much. 

Yet that isn't true in the recent past. Here's a chart from Google trends of the last 90 days in the US for Obama (blue), Romney (red) and taxes (yellow).


obama-gtrends3.png


Most of the Obama-related results that are inserted are super-recent: news articles from The Huffington Post were the most commonly inserted results in our study. If the signal is based on recent results, I would expect the volume of recent searches to matter.

If you restrict to just US news searches, it is similar.

obama-gtrends2.png


And finally, we don't have data for searching something after something else, but as a proxy for that we can look at searches for obama iran and romney iran, which are again similar.

obama-iran3.PNG

I doubt it's intentional, but it nevertheless has arguably pretty wide consequences. Google's AdWords Keyword Tool shows obama is searched for a lot monthly.

obama-keyword.png


This data lags so we can expect it to be significantly higher over the past couple of months given the first chart above. Needless to say, obama has likely been searched over a hundred million times in the US during 2012.

Given that magic keywords can transform searches throughout a whole search session, we can expect a lot of Obama-related results have been inserted this election year. As far as we can tell, no Romney-related results have been similarly inserted.

To see it in practice, if you search Google for obama and then iran you'll likely see something like this:

obama-iran.PNG

Those red arrows are pointing to Obama-related results that Google inserted into their results based on the obama search. 


Conclusions

I believe our first experiment was a clear confirmation that the Filter Bubble is real and important. As such, we plan to do more experiments to shed more light on this under-studied phenomenon.

Inserting Obama-related results but not Romney-related results to tens of millions of people strikes me as a well-intentioned algorithm run amok. It is the best example I've seen so far of how search personalization could likely have had widespread political consequences. 

Consider again that undecided voter casually researching the candidates and issues. They would have likely seen Obama-related results inserted across the scope of their session, but not Romney-related results.

More generally, with so many personalization signals floating around I think we're going to see more and more unintended consequences of tailoring.

Magic keywords in particular are attractive for study because they are easy to see. I suspect there is similar bias lurking in other areas beyond politics. Some other (non-political) magic keywords I noticed just playing around for a bit are iphone, nexus, tablet, twilight, computer, speech, iraq, sports, and stock. There must be hundreds or even thousands of them.

We also put together a 1min video summarizing the results of our first study:



I'm the CEO & Founder of DuckDuckGo, the search engine that doesn't track you. More about me.
Get new posts:
Check out the left column for top posts and here are recent posts.
Have a comment? Please let me know.

About Me

RSS.