On September 6th, 2012 at 2PM eastern time, 131 DuckDuckGo users searched Google at the same time for abortion, gun control and obama. This was the first in a series of experiments we're doing to explore the state of Google search tailoring -- the fact that different people see different results on Google based on who they are.
Sometimes tailoring can be good, e.g. we show local weather in our instant answer box. However, in other cases it can put you in a bubble, seeing more and more of what you already agree with, and less and less (relevant and important) opposing viewpoints. This effective filtering is troublesome when looking for raw information on a subject.
For example, consider you're an undecided voter being bombarded with propaganda from both sides. What do you do to research the candidates? If you're like most people, you use Google as your starting point.
Yet when searching Google for raw information, say on gun control, we found people getting very different results. That's why at DuckDuckGo we show everyone the same links by default--so you can escape your personal filter bubble. We believe any tailoring of the organic results should be opt-in and not opt-out.
In a recent Pew Report, 65% of people said personalized search was a "bad thing" since "it may limit the information you get online and what search results you see" compared to just 29% who said it was a "good thing" because "it gives you results that are more relevant to you." When asked a different way 73% said they were "Not OK" with it because it was an invasion of privacy.
DuckDuckGo does not have a political affiliation. Individuals within DuckDuckGo have many different (and opposing) political viewpoints, and also live in many different countries.
We chose to use these keywords (abortion, gun control and obama) because they are both a) searches where many people want unbiased results and b) particularly relevant given the 2012 presidential election in the US.
We have users worldwide who participated in the study. When you look across countries, people saw all sorts of different things. While still against our policy (we think you should opt-in to region and language tailoring), it is pretty understandable why you might see some variation across countries.
So we concentrated on just the United States as the terms were relevant to the election here. We asked people to take screenshots of their Google results in both a "on" (signed in) and "off" state. The off state in this experiment was mostly signed out, but in some cases included being in private browsing mode or tweaking the various personalization settings on Google.
What you see below is the raw data in terms of how many different orderings people saw in just the first five links, for each of the searches and in both on and off versions. We treated the news block as one link and just counted the first news result as a link (labeled N), even though that masked more variation.
US: gun control (on)
The number of subjects varied a bit in each group because a) we had to throw out some screen shots that didn't contain enough information and b) we asked people who didn't have Google accounts to just do the "off" state.
I should also note that link ordering matters a lot because people click on the first link way more than the second, and so on.
- You cannot turn Google personalization off, as I've explored before. We saw about as much variation in the "off" state as the "on" state. That could be in part due to our users generally not having a lot of data stored on Google, but nevertheless the "off" state showed a lot of variation given that you'd expect very little variation (say, from Google running isolated tests).
- No ordering received a majority across the whole study, and several orderings were only seen by one or two people. In fact, the data only includes the top five links -- if you open it up to the whole first page (usually 7-11 links) it fragments a lot more.
- We saw a lot of different links (not just orderings). And we also saw a lot of different news results within the news blocks.
- The news varied a lot. In the obama search, news was the first link. Some people were getting Fox News while others got the LA Times and a few people got other stuff.
- Individual people often saw the same things on the off and on versions, but there was of course more variation person to person. That this tailoring exists even when making an attempt to de-personalize (signing out or going incognito) makes it impossible for an individual to pop their own Google filter bubble. Google has been personalizing for signed out users since 2009, but we didn't expect it to be so all-encompassing.
- We stumbled upon magic keywords.
As the Wall Street Journal confirmed in its own study, Google has been significantly altering its search results to highlight Obama-related results, but not Romney-related results (more on that later).
These Obama-related results are being inserted because obama is a magic keyword on Google. A magic keyword is a search that can transform the Google results of later searches.
As you can see, I got a couple of results inserted based on the intersection of taxes and social security. The first inserted result is labeled (You recently searched for taxes). For some reason, Google doesn't label the other ones it inserts, but you can tell which ones they are because they bold the magic keyword in addition to the search terms (see second red arrow pointing to taxes).
Magic Keyword Variation
Magic keywords create more variation than you might initially expect because from person to person Google personalizes:
- how many links they insert (we saw 0-4);
- which links they insert (we saw different ones per search);
- what order they insert them in (we saw different orderings of the inserted links);
- where they insert them (we saw different placements within the page, e.g. after link #3 vs #6).
- which subsequent searches they transform (different people saw different searches transformed);
- how many subsequent searches they transform (we saw upwards of ten searches later in our tests).
In other words, magic keywords have their own internal variation, presumably based on personal factors.
Also, we saw magic keyword transformations about equally across the on and off states. This means that if you start in private browsing mode (with cookies initially cleared) and start searching, you're still effected by magic keywords.
We also saw in Google chrome that magic keyword transformations sometimes jumped incognito sessions, meaning that if you started a new incognito mode, got a transformation, then shut it down and started a new incognito mode you could sometimes see the same transformation again (without searching for the same magic keyword again). This weird behavior was not reproducible in Firefox's private browsing mode.
All of this is a black box, meaning we don't really know what is going on. All we really know is it is happening and it is impossible to escape on Google. We're doing our best to quantify the effects.
Obama, not Romney
After stumbling upon magic keywords, we began to further explore their prevalence. We subsequently found there are a lot of magic keywords related to the upcoming US presidential election, such as taxes, social security, health care, abortion, ohio, election and many others.
Since there is so much overlap between Obama and election issues, it makes sense the obama magic keyword would transform a large subset of election-related queries. It even works on a search for Romney himself!
Yet that isn't true in the recent past. Here's a chart from Google trends of the last 90 days in the US for Obama (blue), Romney (red) and taxes (yellow).
Most of the Obama-related results that are inserted are super-recent: news articles from The Huffington Post were the most commonly inserted results in our study. If the signal is based on recent results, I would expect the volume of recent searches to matter.
I doubt it's intentional, but it nevertheless has arguably pretty wide consequences. Google's AdWords Keyword Tool shows obama is searched for a lot monthly.
This data lags so we can expect it to be significantly higher over the past couple of months given the first chart above. Needless to say, obama has likely been searched over a hundred million times in the US during 2012.
Those red arrows are pointing to Obama-related results that Google inserted into their results based on the obama search.
We also put together a 1min video summarizing the results of our first study: