The Best of 2013: The Top Posts and People on the Moz Blog

Posted on December 31, 2013 by

Continuing what has become an annual tradition at Moz, on the last day of 2013 we’re excited to bring you a roundup of the very best of this year’s posts on the Moz Blog and YouMoz.

After asking Roger to work overtime to crunch these numbers, we’ve whittled the posts down to the cream of the crop, and have organized them in several different ways:

This year’s data was more difficult to collect, as we migrated to our new domain in May. My eternal gratitude goes to Cyrus Shepard for helping make sure the right numbers were pulled.

Top comments by number of thumbs up

We’re always impressed by the discussions we see in the comments below blog posts. In addition to the great many insightful points that add to what the authors say, one of our favorite parts is the support our community members show for one another. Here are the most thumbed-up comments from 2013.

1. Stephan_Boehringer | September 24
When Keyword (not provided) is 100 Percent of Organic Referrals, What Should Marketers Do? – Whiteboard Tuesday

2. gfiorelli1 | July 19
Heart to Heart About Link Building – Whiteboard Friday

3. MarkTraphagen | August 20
Amazing Correlation Between Google +1s and Higher Search Rankings

4. jcolman | May 29
Goodbye SEOmoz. Hello Moz!

5. randfish | August 29
The Web Developer’s SEO Cheat Sheet 2.0

6. Dr-Pete | April 15
The Difference Between Penguin and an Unnatural Links Penalty (and some info on Panda too)

7. Bill Sebald | July 16
9 Experts and a Summary: What Makes an Ideal SEO Employee?

8. KeriMorgret | February 7
Going Beyond Moz Metrics to Answer: “Why is this Site Outranking Me?”

9. evolvingSEO | February 12
Stop Clicking Here! 7 Superior SEO Alternatives to Generic Links

10. KeriMorgret | March 8
6 Ways to Use Fresh Links & Mentions to Improve Your Marketing Efforts – Whiteboard Friday

Most active users by number of comments

While quality certainly trumps quantity in most cases, we’re continuously impressed by the ability of our community members to cover both bases. On average, the folks on this list (which intentionally omits our own staff and associates) have left a comment on every second or third post we’ve published, and we couldn’t appreciate their contributions more.

4. Dubs
mozPoints: 946 | Rank: 85

9. skifr
mozPoints: 234 | Rank: 407

About Trevor-Klein — Trevor is the editorial specialist at Moz—a proud member of the content team. He manages the Moz Blog, helps craft and execute content strategy, and wrangles other projects in an effort to provide the most valuable content possible for the Moz community.

Foursquare Quietly Unlocks Its Own "Local Data Aggregator" Badge

Posted on December 30, 2013 by

I was wrong about Foursquare.

While five of my 2013 local search prognostications came to fruition, my sixth prediction—that Foursquare would be bought—doesn’t look like it will (unless Apple has silently acquired Foursquare in the last couple of days).

In fact, Foursquare has been turning away from an acquisition path, setting off on a fundraising spree in 2013. While this quest for cash has struck some analysts as a desperate tactic, PR from the company indicates that it remains focused on growing its userbase and its revenues for the foreseeable future. It’s one of the few companies in tech to successfully address both sides of the merchant and consumer marketplace, and as a result, might even have a chance at an IPO.

As the company matures, we hear less and less about mayorships, badges, and social gamification—perhaps a tacit admission that checkins are indeed dying as the motivational factor underlying usage of Foursquare.

Foursquare: the data aggregator

Instead, the company is pivoting into a self-described position as “the location layer for the Internet.”

Google, Bing, Nokia, and other mapping companies have built their own much broader location layers to varying degrees of success, but it’s the human activity associated with location data that makes Foursquare unique. Its growing database of keyword-rich tips and comments and widening network of social interactions even make predictive recommendations possible.

But I’m considerably less excited about these consumer-facing recommendations than I am about Foursquare’s data play. If “location layer for the internet” is not a synonym for “data aggregator,” I’m not sure what would be.

In the last several months, Foursquare has been prompting its users to provide business details about the places they check-in at, like whether a business has wi-fi, its relative price range, delivery and payment options, and more. It’s also accumulating one of the biggest photo libraries in all of local search. For companies that have not yet built their own services like StreetView and Mapmaker, Foursquare “ground truth” position is enviable.

So from my standpoint, Foursquare’s already achieved the status of a major data aggregator, and seems to have its sights set on becoming the data aggregator.

Foursquare: The Data Aggregator?

That statement would have sounded preposterous 18 months ago, with “only” 15 million users and 250,000 claimed venues.

But while many of us in the local search space have been distracted by the shiny objects of Google+ Local and Facebook Graph Search, Foursquare has struck deals with the two largest up-and-coming social apps (Instagram and Pinterest) to provide the location backbone for their geolocation features. Not to mention Uber, WhatsApp, and a host of other conversational and transactional apps.

And buried in the December 5th TechCrunch article about Foursquare’s latest iOS release was this throwaway line:

“Foursquare has a sharing deal with Apple already — it’s one of over a dozen contributors to Apple’s Maps data.”

So, doing some quick math, we have

All of a sudden that’s a substantial number of people contributing location information to Foursquare. Granted, there’s considerable overlap in those users, but even a conservative 80-100 million would be a pretty large number of touchpoints.

In fact, one thing that Wil Reynolds and I realized at a recent get-together in San Diego is that for many people outside the tech world, Foursquare and Instagram are basically the same app (see screenshots below). I’m seeing more and more of my decidedly non-techie Instagram friends tagging their photos with location. And avid Foursquare users like Matthew Brown have always made photography their primary network activity.

Providing the geographic foundation for two apps—Pinterest and Instagram—that are far more popular than Foursquare gives it a strong running start on laying the location foundation for the Internet.

What’s next for Foursquare?

While Facebook is undoubtedly building its own location layer, Zuckerberg and company have long ignored local search. And they’ve got plenty of other short- and mid-term priorities. Exposing Facebook check-in data to the extent Foursquare has, and forcing Instagram to update a very successful API integration, would seem to be pretty far down the list.

As I suggested in my Local Search Ecosystem update in August, to challenge established players like Infogroup, Neustar, and Acxiom, in the long run Foursquare does need to build out its index considerably beyond the current sweetspots of food, drink, and entertainment.

But in the short run, the quality and depth of Foursquare’s popular venue information in major cities gives start-up app developers everything they need to launch and attract users to their apps. And Foursquare’s independence from Google, Facebook, and Apple is appealing for many of them—particularly for non-U.S. app developers who have a hard time finding publicly-available location databases outside of Google or Facebook.

Foursquare’s success with Instagram and Pinterest has created a self-perpetuating growth strategy: it will continue to be the location API of choice for most “hot” local startups.

TL;DR

Foursquare venues have been contributing to a business’s citation profile for years, so hopefully most of you have included venue creation and management in your local SEO service packages already. Even if you optimize non-retail locations like insurance agencies, accounting offices, and the like, make one of your 2014 New Year’s resolutions be a higher level of engagement with Foursquare.

The bottom line is that irrespective of its user growth and beyond just SEO, Foursquare is going to get more important to the SoLoMo ecosystem in the coming year.

About David-Mihm — David Mihm is one of the world’s leading practitioners of Local search engine marketing. He has created and promoted search-friendly websites for clients of all sizes since the early 2000’s. David co-founded GetListed.org, which he sold to Moz in November 2012. His annual Local Search Ranking Factors project is among the most important studies of Local SEO.

When 2 Become 1: How Merging Two Domains Made Us an SEO Killing

Posted on December 30, 2013 by

This is a story of recovery, despondency, occasional despair, and a pretty big gamble that paid off. It’s the why, the how, and the what of the things you might be able to gain from merging two significant domains into one unified site.

Regular Moz readers should recall WPMU.org from our fairly dramatic Penguin story from 2012 (tl;dr: the Penguin hit us hard, but then we recovered. It was pretty scary).

But all ended remarkably well. After our initial recovery, things went from good to, well, better:

Weekly organic traffic at WPMU.org took a nasty slug, and made a solid recovery.

Parties all round at Incsub HQ. Hell yeah. Let’s go hire a bunch of new writers, let’s go wild, let’s double this next year, etc.

I imagine you can guess what happened next…

Dear Search Lord, Why Has Thou Forsaken Us So? Dear Search Lords, Why Hast Thou Forsaken Us So?

Now, usually I’d be the first to see that and say something along the lines of, “well guys, you’re clearly doing it wrong.”

And in fact that’s exactly what I thought, pretty much from day one, so we got bloody busy. Specifically, we:

Hired some absolutely awesome and highly qualified new writers who took our standards up an absolute ton;
Spent ages working out quality and style guidelines for copy and media, and followed them like subeditors who had tucked into wayyy too many cans of V;
Brought in the best guest writers and paid them the best rates in a systematised editorial process;
Dramatically increased our social and email presence and published stuff that generated it’s own awesome links;
Tried every on-site SEO tactic we could, killed duplicate content, limited and focused our categories and tags, and essentially gave Google everything that she wanted: really quality, fresh, and engaging content.

And yet it was all for naught, we were, to put it mildly, in a hole. Going nowhere fast. We’d tried everything, pulled every string and ticked every box, we were doing stuff better than ever, but still we were failing.

So, we figured, let’s do something dramatic. Let’s kill WPMU.org and merge it with her sister site WPMU DEV.

This is where I get to insert the video, right?

[embedded content]

Awesome. Happy now. Moving on.

Why on earth would you kill such a well-known site?

It’s a good question, and it’s not one we arrived at lightly. Essentially though we were ready to take the punt for a bunch of different reasons, not the least of which being that seven months of declining organic results are enough to make anyone more risk-friendly than averse. But, more specifically:

Latent penguin / penalty Issues

Let’s face it, clearly Google had some pretty serious issues with us, and just because we recovered so well from Penguin, that didn’t mean we went off their radar, or the strategies we’d been employing (all white-hat, incidentally) weren’t falling close to the boundary line.

There was every reason to believe that a hex of some sort had been placed on wpmu.org. It was a monkey we just couldn’t shift, and to stretch the metaphor a little, those kinda monkeys aren’t in the trees, they’re clinging firmly to you day after day.

I always knew I’d get to use this image in a post one day

We had to shake the chimpguin!

Dilution to concentration, juice-wise

Back in the day, I set up WPMU.org as an “independent” site, the main business being WPMU DEV (at the extraordinarily bad, and still bad, premium.wpmudev.org domain).

Same, but different, kinda, look, it's complicated

Same, but different, kinda, look. It’s complicated.

And it has been that, we take no affiliate revenue, have no editorial agenda as regards any company outside of us and aim to give fair, balanced and decent coverage to all things WordPress. We’re really trying to be the same as the Moz blog, for WordPress.

But let’s face it, it’s WPMU DEV’s blog, and more to the point, we were generating organic links and engagement with a site that wasn’t our main business, while at the same time trying to do the same with WPMU DEV. It was a little nuts; both sites had thousands of unique domains linking to them, so they were both moderately powerful. Why on earth didn’t we just merge them together, and have one super-powerful site rather than two middling-to-strong ones.

Brand, brand, brand

And last, but certainly not least, there’s the small matter of Google and our brand… and if there’s a primary lesson in this piece, this could well be it.

Put simply, a search for ‘wpmu’ or ‘wpmu.org’ rendered a very different group of results to one for ‘wpmu dev’:

Somebody's got the SEO right for one of these grabs...

Somebody’s got the SEO right for one of these grabs…

I wonder what the impact of all those high-quality and fresh posts could be along with the WPMU DEV brand? Hmmmmm.

Technical time: merging two domains into one

It’s actually remarkably straightforward, here’s how you go about it:

First up, download and print this Moz infographic, and keep it by you at all times.

Second, fire up Asana; this will be fabulously useful if you are on your own and even more so if there are a bunch of you.

If there are a bunch of you, sit very close, or jump into a hangout, and (here we go)…

Decide on the new URL. We moved wpmu.org to /blog/ on premium.wpmudev.org, so it was pretty easy to transfer our staging. (Oh yeah: Get a staging server too, or just set things up with a modified hosts file.)
Dynamically (or manually, yawn) 301 everything, here’s your complete guide to redirection
Go through your dbase and theme files and replace every link via find and replace. (I.e. replace “wpmu.org” with “premium.wpmudev.org/blog”.)
Test the heck out of it. Give yourself at least a few hours to try pretty much every page (and make some user personas, too).
Use Open Site Explorer to find the major links to your site, and email whoever wrote the articles or manages the site, asking them to change their links to the new site.
Test some more.
Go tell Google using Webmaster Tools (and Bing if you have some extra time). 😉
Keep a good eye on things, and also run a Moz Analytics campaign on the new setup to pick up Crawl Diagnostics.
Ask everyone you know to look at the new setup and find issues (they will). Fix them.
Sit back and wait to see how well it works.

So, how was it for us?

I was expecting that we’d take a hit.

Before the move I’d said that up to a 30% hit would be manageable; we could build back from that, and it was to be expected by the dilution of link juice coming from 301s. Anything more would be a big problem, but we’d battle through.

Here’s how it actually went:

Before and after shots Before and after organic shots

On the Monday before we picked up 10,371 organic visits. On the Monday following, 14,627.

On the Tuesday prior, 10,458, and after, 14,546.

The two days taken together were almost exactly 40% up.

Not. Bad.

However, we did note that there was no significant change in organic visits for non /blog/* results at WPMU DEV, in fact over the two days (mostly Tuesday) we saw a slight decline of around ~1000 visits (around 2.5% of the overall traffic, but around a 6% variation in the original WPMU DEV traffic), which might indicate that the whole “concentrating juice on one domain” theory might not be the right one.

In conclusion

From this experience we’ve learnt a bunch of stuff, which I’m going to try to summarize in three main areas.

You can move and not lose, so move away

A well-managed and carefully executed move from one domain to another, or in this case from one domain onto another, can clearly work well.

This is super-important, because honestly, when I brought this up with most people prior to this venture they were very very dubious as to whether this could be pulled off without some serious collateral. When Google says that you can retain your ranking, it’s true, you can. And then some.

This may be a successful tactic to escape domain toxicity

The lack of any positive organic bump in the root domain we moved to as /blog/ could indicate that the success of this domain move was not due to the amalgamation of link juice between the two sites, but could in fact be due to the content having escaped some negative/toxic algo penalties that wpmu.org had accrued as a root domain.

However, Google is not stupid. You would expect that they would happily pass along the bad with the good on a 301, and it’s often recommended you don’t redirect (another thing that was making me nervous).

Branding could be the single most important factor

You don’t need to be a multinational; having a relatively established brand like WPMU DEV is enough.

Sure, we’re no Moz, let alone a Pfizer, but it could be that moving content from a well-established site (but not brand) to our more-established position is literally worth a 40% bump.

If so, the importance of building and managing a brand alongside your content strategies could well be top of your agenda. At least that’s my takeaway… what’s yours?

About WPMU DEV — If you’re looking for some serious WordPress goodness, you’ve found it at WPMU DEV. With over 140 premium WordPress plugins, with new ones coming all the time, along with cracking Multisite and BuddyPress themes we’ve got all your requirements covered. Along with the best WordPress support you are every gonna find, anywhere, period.

The IdeaGraph – Whiteboard Friday

Posted on December 27, 2013 by

There can be important links between topics that seem completely unrelated at first glance. These random affinities are factoring into search results more and more, and in today’s Whiteboard Friday, Ian Lurie of Portent, Inc. shows us how we can find and benefit from those otherwise-hidden links.

Howdy Moz fans. Today we’re going to talk about the IdeaGraph. My name’s Ian Lurie, and I want to talk about some critical evolution that’s happening in the world of search right now.

Google and other search engines have existed in a world of words and links. Words establish relevance. Links establish connections and authority. The problem with that is Google takes a look at this world of links and words and has a very hard time with what I call random affinities.

Let’s say all cyclists like eggplant, or some cyclists like eggplant. Google can’t figure that out. There is no way to make that connection. Maybe if every eggplant site on the planet linked to every cycling site on the planet, there would be something there for them, but there really isn’t.

So Google exists purely on words and links, which means there’s a lot of things that it doesn’t pick up on. The things it doesn’t pick up on are what I call the IdeaGraph.

The IdeaGraph is something that’s always existed. It’s not something new. It’s this thing that creates these connections that are formed only by people. So things that are totally unrelated, like eggplant and cyclists, and by the way that’s not true as far as I know. I’m a cyclist and I hate eggplant. But all these things that randomly connect are part of the IdeaGraph.

The IdeaGraph has been used by marketers for years and years and years. If you walk into a grocery store, and you’re going from one aisle to the next and you see these products in semi-random order, there’s some research there where they test different configurations and see, if someone’s walking to the dairy section way at the back of the store, what products can we put along their walk that they’re most likely to pick up? Those products, even if the marketers don’t know it, are part of the IdeaGraph, because you could put chocolate there, and maybe the chocolate is what people want, but maybe you should put cleaning supplies there and nobody wants it, because the IdeaGraph doesn’t connect them tightly enough.

The other place that you run into issues with the IdeaGraph on search and on the Internet is with authorship and credibility and authority.

Right now, if you write an article, and it gets posted on a third-party site, like The New York Times, and it’s a huge hit, and it gets thousands and thousands and thousands of links, you might get a little authority sent back to your site, and your site is sad. See? Sad face website. Because it’s not getting all the authority it could. Your post is getting tons. It’s happy. But your site is not.

With the IdeaGraph it will be easier because the thing that connects your site to your article is you. So just like you can connect widely varying ideas and concepts, you can also connect everything you contribute to a single central source, which then redistributes that authority.

Now Google is starting to work on this. They’re starting to work on how to make this work for them in search results. What they’ve started to do is build these random affinities. So if you take cyclists and eggplant, theoretically some of the things Google is doing could eventually create this place, this space, where you would be able to tell from Google, and Google would be able to tell you that there is this overlap.

The place that they’re starting to do it, I think, remember Google doesn’t come and tell us these things, but I think it’s Google+. With authorship and publisher, rel=author and rel=publisher, they’re actually tying these different things together into a single receptacle into your Google+ profile. Remember, anyone who has Gmail, has a Google+ profile. They may not know it, but they do. Now Google’s gathering all sorts of demographic data with that as well.

So what they’re doing is, let’s say you’re using rel=author and you publish posts all over the Internet, good posts. If you’re just doing crappy guest blogging, this probably won’t work. You’ll just send yourself all the lousy credit. You want the good credit. So you write all these posts, and you have the rel=author on the post, and they link back to your Google+ profile.

So your Google+ profile gets more and more authoritative. As it gets more and more authoritative, it redistributes that authority, that connection to all the places you publish. What you end up with is a much more robust way of connecting content to people and ideas to people, and ideas to each other. If you write about cycling on one site and eggplant on another, and they both link back to your Google+ profile, and a lot of other people do that, Google can start to say, “Huh, there might be a connection here. Maybe, with my new enhanced query results, I should think about how I can put these two pieces of information together to provide better search results.” And your site ends up happier. See? Happy site. Total limit of my artistic ability.

So that becomes a very powerful tool for creating exactly the right kind of results that we, as human beings, really want, because people create the IdeaGraph. Search engines create the world of words and links, and that’s why some people have so much trouble with queries, because they’re having to convert their thinking from just ideas to words and links.

So what powers the IdeaGraph is this concept of random affinities. You, as a marketer, can take advantage of that, because as Google figures this out through Google+, you’re going to be able to find these affinities, and just like all those aisles in the grocery store, or when you walk into a Starbucks and there’s a CD there—you’re buying coffee and there’s a CD? How do those relate? When you find those random affinities, you can capitalize on them and make your marketing message that much more compelling, because you can find where to put that message in places you might never expect.

An example I like is I went on Amazon once and I searched for “lonely planet,” and in the “people who bought this also bought,” I found a book on making really great smoothies, which tells me there’s this random affinity between people who travel lonely planet style and people who like smoothies. It might be a tiny attachment. It might be a tiny relationship, but it’s a great place to do some cross marketing and to target content.

So if you take a look here, if you want to find random affinities and build on them, take a look at the Facebook Ad Planner. When you’re building a Facebook ad, you can put in a precise interest, and it’ll show you other related precise interests. Those relationships are built almost purely on the people who have them in common. So sometimes there is no match, there’s no relationship between those two different concepts or interests, other than the fact that lots of people like them both. So that’s a good place to start.

Any site that uses collaborative filtering. So, Amazon, for example. Any site that has “people who bought this also bought that” is a great place to go try this. Go on Amazon and try it and look at “people who bought also bought.” You’ll find all sorts of cool relationships.

Followerwonk is a fantastic tool for this. This one takes a little more work, but the data you can find is incredible. Let’s say you know that Rand is one of your customers. He’s a perfect customer, and he’s typical of your perfect customer. You can go on Followerwonk and find all the people who follow him and then pull all of their bios, do a little research into the bios and find what other interests those people express.

So they’re following Randfish, but maybe a whole bunch of them express an interest in comic books, and it’s more than just one or two. It’s a big number of them. You just found a random affinity. People who like Rand also like comic books. You can then find this area, and it’s always easier to sell and get interest in this area.

Again, you can use that to drive content strategy. You can use that to drive keyword selection in a world where we don’t really know what keywords are driving traffic anymore, but we can find out what ideas are. You can use it to target specific messages to people.

The ways you capitalize on this, on your own site you want to make sure that you have rel=author and publisher set up, because that’s the most obvious IdeaGraph implementation we have right now, is rel=author and publisher.

Make sure you’re using schemas from Schema.org whenever you can. For example, make sure you use the article mark-up on your site because Google’s enhanced articles, results that are showing up at the bottom of search results right now, those are powered, in part, by pages that have the article mark-up, or at least there’s a very high correlation between them. We don’t know if it’s causal, but it seems to be.

Use product mark-up and review mark-up. I’ve seen a few instances and some of my colleagues have seen instances where schema mark-up on a page allows content to show up in search results attributed to that page, even if they’re being populated to the page by JavaScript or something else.

Get yourself set up with Google Analytics Demographics, as Google rolls it out. You’ll be able to get demographic data and categorical data in Google Analytics based on visitors to your site. Then again, if you have a demographic profile, you can look at the things that that demographic profile is interested in and find those random affinities.

So just to summarize all of this, links and words have worked for a long time, but we’re starting to see the limitations of it, particularly with mobile devices and other kinds of search. Google has been trying to find a way to fix this, as has Bing, and they’re both working very hard at this. They’re trying to build on this thing that has always existed that I call the IdeaGraph, and they’re building on it using random affinities. Selling to random affinities is much, much easier. You can find them using lots of tools out on the web like collaborative filtering, Facebook, and Followerwonk. You can take advantage and position your site for it by just making sure that you have these basic mark-up elements in place, and you’re already collecting data.

I hope that was helpful to all Moz fans out there, and I look forward to talking to you online. Thanks.

Mission ImposSERPble 2: User Intent and Click Through Rates

Posted on December 26, 2013 by

It’s been quite a while since I first read (and bookmarked) Slingshot SEO’s YouMoz blog post, Mission ImposSERPble: Establishing Click-through Rates, which showcased their study examining organic click-through rates (CTR) across search engine result pages. The Slingshot study is an excellent example of how one can use data to uncover trends and insights. However, that study is over two and a half years old now, and the Google search results have evolved significantly since then.

Using the Slingshot CTR study (and a few others) as inspiration, Catalyst thought it would be beneficial to take a fresh look at some of our own click-through rate data and dive into the mindset of searchers and their proclivity for clicking on the different types of modern organic Google search results.

Swing on over to Catalyst’s website and download the free Google CTR Study: How User Intent Impacts Google Click-Through Rates

**TANGENT: I’m really hoping that the Moz community’s reception of this ‘sequel’ post follows the path of some of the all-time great movie sequels (think Terminator 2, The Godfather: Part II) and not that of Jaws 2.

How is the 2013 Catalyst CTR study unique?

RECENT DATA: This CTR study is the most current large-scale US study available. It contains data ranging from Oct. 2012 – June 2013. Google is constantly tweaking its SERP UI, which can influence organic CTR behavior.
MORE DATA: This study contains more keyword data, too. The keyword set for this study spans 17,500 unique queries across 59 different websites. More data can lead to more accurate representations of the true population.
MORE SEGMENTS: This study segments queries into categories not covered in previous studies which allows us to compared CTR behavior attributed to different keyword types. For example, branded v. unbranded queries, and question v. non-question based queries.

How have organic CTRs changed over time?

The most significant changes since the 2011 Slingshot study is the higher CTRs for positions 3, 4, and 5.

Ranking on the first page of search results is great for achieving visibility; however, the search result for your website must be compelling enough to make searchers want to click through to your website. In fact, this study shows that having the most compelling listing in the SERPs could be more important than “ranking #1” (provided you are still ranking within the top five listings, anyway).

Read on to learn more.

Catalyst 2013 CTRs vs. Slingshot SEO 2011 CTRs

data table of Catalyst CTRs compared to Slingshot SEO CTRs

Since Slingshot’s 2011 study, click-through rates have not dramatically shifted, with the total average CTR for first page organic results dropping by just 4%.

While seemingly minor, these downward shifts could be a result of Google’s ever-evolving user interface. For example, with elements such as Product Listing Ads, Knowledge Graph information, G+ authorship snippets, and other microdata becoming more and more common in a Google SERP, users’ eyes may tend to stray further from the historical “F shape” pattern, impacting the CTR by ranking position.

Positions 3-5 showed slightly higher average CTRs than what Slingshot presented in 2011. A possible explanation for this shift is that users could be more aware of Paid Search listing located at the top of the results page, so in an attempt to “bypass” these results, they may have modified their browsing behavior to quickly scan/wheel-scroll past a few listings down the page.

What is the distribution of clicks across a Google SERP?

example Google search engine result page click distributions

Business owners need to understand that even if your website ranks in the first organic position for your target keyword, your site will almost certainly never receive traffic from every one of those users/searchers.

On average, the top organic SERP listing (#1) drives visits from around 17% of Google searches.

The top four positions, or typical rankings “above the fold” for many desktop users, receive 83% of first page organic clicks.

The Catalyst data also reveals that only 48% of Google searches result in a page one organic click (meaning any click on listings ranging 1-10). So what is the other 52% doing? Two things, the user either clicks on a Paid Search listing, or they “abandon” the search, which we define as:

Query Refinement – based on the displayed results, the user alters their search
Instant Satisfaction – based on the displayed results, the user gets the answer they were interested in without having to click
2nd Page Organic SERP – the user navigates to other SERPs
Leave Search Engine – the user exits the Google search engine

How do branded query CTRs differ from unbranded queries?

Branded CTRs for top ranking terms are lower than unbranded CTRs, likely due to both user intent and the way Google presents results.

branded query CTRs vs. unbranded query CTRs

data table of branded and unbranded organic CTRs

These numbers shocked us a bit. At the surface, you might assume that listings with top rankings for branded queries would have higher CTRs than unbranded queries. But, when you take a closer look at the current Google UI and place yourself in the mindset of a searcher, our data actually seems more likely.

Consumers who search unbranded queries are often times higher in the purchasing funnel: looking for information, without a specific answer or action in mind. As a result, they may be more likely to click on the first result, particularly when the listing belongs to a strong brand that they trust.

Additionally, take a look at the example below, notice how many organic results are presented “above the fold” for a unbranded query compared to an branded query (note: these SERP screenshots were taken from 1366×768 screen resolution). There are far fewer potential organic click paths for a user to take when presented with the branded query’s result page (1 organic result v. 4.5 results). It really boils down to ‘transactional’ v. ‘informational’ queries. Typically, keywords that are more transactional (e.g. purchase intent) and/or drive higher ROI are more competitive in the PPC space and as a result will have more paid search ads encroaching on valuable SERP real estate.

example branded search query v. unbranded search query result page

We all know the makeup of every search result page is different and the number of organic results above the fold can be influenced by a number of factors, including, device type, screen size/resolution, paid search competiveness, and so on.

You can use your website analytics platform to see what screen resolutions your visitors are using and predict how many organic listings your target audience would typically see for different search types and devices. In our example, you can see that my desktop visitors most commonly use screen resolutions higher than 1280×800, so I can be fairly certain that my current audience typically sees up to 5 organic results from a desktop Google search.

Google Analytics screen resolution of my audience

Does query length/word count impact organic CTR?

As a user’s query length approaches the long tail, the average CTR for page one rankings increases.

head vs long tail organic ctr

The organic click percentage totals represented in this graph suggest that as a user’s query becomes more refined they are more likely to click on a first page organic result (~56% for four+ word queries v. ~30% for one-word queries).

Furthermore, as a query approaches the long tail, click distributions across the top ten results begin to spread more evenly down the fold. Meaning, when a consumer’s search becomes more refined/specific, they likely spend more time scanning the SERPs looking for the best possible listing to answer their search inquiry. This is where compelling calls-to-action and eye-catching page titles/meta descriptions can really make or break your organic click through rates.

As previously stated, only about 30% of one-word queries result in a first page organic click. Why so low? Well, one potential reason for this is that searchers use one-word queries simply to refine their search based on their initial impression of the SERP. This means that the single word query would become a multiple word query. If the user does not find what they are looking for within the first result, they modify their search to be more specific, often resulting in the query to contain multiple words.

Additionally, one-word queries resulted in 60% of the total first page organic clicks (17.68%) being attributed to the first ranking. Maybe, by nature, one-word queries are very similar to navigational queries (as the keywords are oftentimes very broad or a specific brand name).

Potential business uses

Leveraging click-through rate data enables us to further understand user behavior on a search result and how it can differ depending on search intent. These learnings can play an integral role in defining a company’s digital strategy, as well as forecasting website traffic and even ROI. For instance:

Forecasting Website Performance and Traffic Given a keyword’s monthly search volume, we can predict the number of visits a website could expect to receive by each ranking position. This becomes increasingly valuable when we have conversion rate data attributed to specific keywords.
Identifying Search Keyword Targets With Google Webmaster Tools’ CTR/search query data we can easily determine the keywords that are “low-hanging fruit”. We consider low hanging fruit to be keywords that a brand ranks fairly well on, but are just outside of achieving high visibility/high organic traffic because the site currently ranks “below the fold” on page 1 of the SERPs or rank somewhere within pages 2-3 of the results.). Once targeted and integrated into the brand’s keyphrase strategy, SEOs can then work to improve the site’s rankings for that particular query.
Identifying Under-performing Top Visible Keywords
By comparing a brand’s specific search query CTR against the industry average as identified in this report, we can identify under-performing keyphrases. Next, an SEO can perform an audit to determine if the low CTR is due to factors within the brand’s control, or if it is caused by external factors.

Data set, criteria, and methodology

Some information about our data set and methodology. If you’re like me, and want to follow along using your own data, you can review our complete process in our whitepaper. All websites included in the study are Consumer Packaged Goods (CPG) brands. As such, the associated CTRs, and hypothesized user behaviors reflect only those brands and users.

Data was collected via each brand’s respective Google Webmaster Tools account, which was then processed and analyzed using a powerful BI and data visualization tool.

Catalyst analyzed close to 17,500 unique search queries (with an average ranking between 1–10, and a minimum of 50 search impressions per month) across 59 unique brands over a 9 month timeframe (Oct. 2012 – Jun 2013).

Here are a few definitions so we’re all on the same page (we mirrored definitions as provided by Google for their Google Webmaster Tools)…

Click-Through Rate (CTR) – the percentage of impressions that resulted in a click for a website.
Average Position – the average top position of a website on the search results page for that query. To calculate average position, Google takes into account the top ranking URL from the website for a particular query.

Final word

I have learned a great deal from the studies and blog posts shared by Moz and other industry experts throughout my career, and I felt I had an opportunity to meaningfully contribute back to the SEO community by providing an updated, more in-depth Google CTR study for SEOs to use as a resource when benchmarking and measuring their campaigns and progress.

For more data and analysis relating to coupon-based queries, question based queries, desktop v. mobile user devices, and more download our complete CTR study.

Have any questions or comments on our study? Did anyone actually enjoy Jaws 2? Please let us know and join the discussion below!

Moz Holiday Traditions

Posted on December 24, 2013 by

Gianluca Fiorelli

Merry Christmas (or whatever you celebrates) to all of you!

My Christmas traditions… I think they are quite common: all family preparing the dinner, running in afternoon for the obviously forgotten gift (this year is the Han Solo action figure mission for my elder son), putting the kids sleeping just after dinner or Santa won’t pass (but before we must leave few cookies and a cup of milk for him close to the tree)… And tomorrow, after all the celebration, going to the movies.

But let’s return to Moz traditions: I imagine that also this year we will Rand Santa doing the Whiteboard Friday: if not… I’ll snap your face 😀

Machine Learning for SEOs

Posted on December 23, 2013 by

The author’s posts are entirely his or her own (excluding the unlikely event of hypnosis) and may not always reflect the views of Moz.

Since the Panda and Penguin updates, the SEO community has been talking more and more about machine learning, and yet often the term still isn’t well understood. We know that it is the “magic” behind Panda and Penguin, but how does it work? Why didn’t they use it earlier? What does it have to do with the periodic “data refreshes” we see for both of these algorithms?

I think that machine learning is going to be playing a bigger and bigger role in SEO, and so I think it is important that we have a basic understanding of how it works.

Disclaimer: Firstly, I’m no expert on machine learning. Secondly, I’m going to intentionally simplify aspects in places and brush over certain details that I don’t feel are necessary. The goal of this post is not to give you a full or detailed understanding of machine learning, but instead to give you a high-level understanding that allows you to answer the questions in my opening paragraph should a client ask you about them. Lastly, Google is a black box, so obviously it is impossible to know for sure exactly how they are going about things, but this is my interpretation of the clues the SEO community has stitched together over time.

Watermelon farming

Machine learning is appropriate to use when there is a problem that does not have an exact answer (i.e. there isn’t a right or wrong answer) and/or one that does not have a method of solution that we can fully describe.

Examples where machine learning is not appropriate would be a computer program that counts the words in a document, simply adds some numbers together, or counts the hyperlinks on a page.

Examples where machine learning would be appropriate are optical character recognition, determining whether an email is spam, or identifying a face in a photo. In all of these cases it is almost impossible for a human (who is most likely extremely good at these tasks) to write an exact set of rules for how to go about doing these things that they can feed into a computer program. Furthermore, there isn’t always a right answer; one man’s spam is another man’s informative newsletter.

Explaining Machine Learning with Will Critchlow at SearchLove 2013 in London. I like watermelons.

The example I am going to use in this post is that of picking watermelons. Watermelons do not continue to ripen once they are picked, so it is important to pick them when they are perfectly ripe. Anyone who has been picking watermelons for years can look at a watermelon, give it a feel with their hands, and from its size, colour and from how firm it feels they can determine whether it is under-ripe, over-ripe or just right. They can do this with a high degree of accuracy. However, if you asked them to write down a list of rules or a flow chart that you or I could use to determined whether a specific watermelon was ripe, then they would almost certainly fail – the problem doesn’t have a clean cut answer you can write into rules. Also note that there isn’t necessarily a right or wrong answer – there may even be disagreement among the farmers.

You can imagine that the same is true about how to identify whether a webpage is spammy or not; it is hard or impossible to write an exact set of rules that work well, and there is room for disagreement.

Robo-farmers

However, this doesn’t mean that it is impossible to teach a computer to find ripe watermelons; it is absolutely possible. We simply need a method that is more akin to how humans would learn this skill: learning by experience. This is where machine learning comes in.

Supervised learning

We can set up a computer (there are various methods, we don’t need to know the details at this point, but the method you’ve likely heard of is artificial neural networks) such that we can feed it information about one melon after another (size, firmness, color, etc.), and we also tell the computer whether that melon is ripe or not. This collection of melons is our “training set,” and depending the complexity of what is being learnt it needs to have a lot of “melons” (or webpages or whatever) in it.

Over time, the computer will begin to construct a model of how it thinks the various attributes of the melon play into it being ripe or not. Machine learning can handle situations where these interactions could be relatively complex (e.g. the firmness of a ripe melon may change depending on the melon’s color and the ambient temperature). We show each melon in the training set many times in a round robin fashion (imagine this was you; now that you’ve noticed something you didn’t before you can go back to previous melons and learn even more from them).

Once we’re feeling confident that the computer is getting the hang of it, then we can give it a test by showing it melons from another collection it has not yet seen (we call this set of melons the “validation set”), but we don’t share whether these melons are ripe or not. Now the computer tries to apply what it has learnt and predict whether the melons are ripe or not (or even how ripe they may or may not be). We can see from how many melons the computer accurately identifies how well it has learnt. If it didn’t learn well we may need to show it more melons or we may need to tweak the algorithm (the “brain”) behind the scenes and start again.

This type of approach is called supervised learning, where we supply the learning algorithm with the details about whether the original melons are ripe or not. There do exist alternative methods, but supervised learning is the best starting point and likely covers a fair bit of what Google is doing.

One thing to note here is that even after you’ve trained the computer to identify ripe melons well, it cannot write that exhaustive set of rules we wanted from the farmer any more than the farmer could.

Caffeine infrastructure update

So how does all this fit with search?

First we need to rewind to 2010 and the rollout of the Caffeine infrastructure update. Little did we know it at the time, but Caffeine was the forefather of Panda and Penguin. It was Caffeine that allowed Panda and Penguin to come into existence.

Caffeine allowed Google to update its index far faster than ever before, and update PageRank for parts of the web’s link graph independently of the rest of the graph. Previously, you had to recalculate PageRank for all pages on the web at once; you couldn’t do just one webpage. With Caffeine, we believe that changed and they could estimate, with good accuracy, updated PageRank for parts of the web (sub-graphs) to account for new (or removed) links.

This meant a “live index” that is constantly updating, rather than having periodic updates.

So, how does this tie in with machine learning, and how does it set the stage for Panda and Penguin? Lets put it all together…

Panda and Penguin

Caffeine allowed Google to update PageRank extremely quickly, far faster than ever before, and this is likely the step that allowed them finally apply machine learning at scale as a major part of the algorithm.

The problem that Panda set out to solve is very similar to the problem of determining whether a water melon is ripe. Anyone reading this blog post could take a short look at a webpage, and in most cases tell me how spammy that page is with a high degree of accuracy. However, very few people could write me an exact list of rules to judge that characteristic for pages you’ve not yet seen (“if there are more than x links, and there are y ads taking up z% of the screen above the fold…”). You could give some broad rules, but nothing that would be effective for all the pages where it matters. Consider also that if you (or Google) could construct such a list of strict rules, it would become easier to circumvent them.

So, Google couldn’t write specific sets of rules to judge these spammy pages, which is why for years many of us would groan when we looked at a page that was clearly (in our minds) spammy but which was ranking well in the Google SERPs.

The exact same logic applies for Penguin.

The problems Google was facing were similar to the problem of watermelon farming. So why weren’t they using machine learning from day one?

Training

Google likely created a training set by having their teams of human quality assessors give webpages a score for how spammy that page was. They would have had hundreds or thousands of assessors all review hundreds or thousands of pages to produce a huge list of webpages with associated spam scores (averaged from multiple assessors). I’m not 100% sure on exactly what format this process would have taken, but we can get a general understanding using the above explanation.

Now, recall that to learn how ripe the watermelons are we have to have a lot of melons and we have to look at each of them multiple times. This is a lot of work and takes time, especially given that we have to learn and update our understanding (we call that the “model”) of how to determine ripeness. After that step we need to try our model out on the validation set (the melons we’ve not seen before) to assess whether it is working well or not.

In Google’s case, this process is taking place across its whole index of the web. I’m not clear on the exact approach they would be using here, of course, but it seems clear that applying the above “learn and test” approach across the whole index is immensely resource intensive. The types of breakthroughs that Caffeine enabled with a live index and faster computation on just parts of the graph are what made Machine Learning finally viable. You can imagine that previously if it took hours (or even minutes) to recompute values (be it PageRank or a spam metric) then doing this the thousands of times necessary to apply Machine Learning simply was not possible. Once Caffeine allowed them to begin, the timeline to Panda and subsequently Penguin was pretty quick, demonstrating that once they were able they were keen to utilise machine learning as part of the algorithm (and it is clear why).

What next?

Each “roll out” of subsequent Panda and Penguin updates was when a new (and presumably improved) model had been calculated, tested, and could now be applied as a signal to the live index. Then, earlier this year, it was announced that Panda would be continuously updating and rolling out over periods of around 10 days, so the signs indicate that they are improving the speed and efficiency with which they can apply Machine Learning to the index.

Hummingbird seems to be setting the stage for additional updates.

I fully expect we will see more machine learning being applied to all areas of Google over the coming year. In fact, I think we are already seeing the next iterations of it with Hummingbird, and at Distilled we are viewing the Hummingbird update in a similar fashion to Caffeine. Whilst Hummingbird was an algorithm update rather than an infrastructure update, we can’t shake the feeling that it is setting the foundations for something yet to come.

Wrap-up

I’m excited by the possibilities of machine learning being applied at this sort of scale, and I think we’re going to see a lot more of it. This post set out to give a basic understanding of what is involved, but I’m afraid to tell you I’m not sure the watermelon science is 100% accurate. However, I think understanding the concept of Machine Learning can really help when trying to comprehend algorithms such as Panda and Penguin.

Building SEO-Focused Pages to Serve Topics & People Rather than Keywords & Rankings – Whiteboard Friday

Posted on December 20, 2013 by

With updates like Hummingbird, Google is getting better and better at determining what’s relevant to you and what you’re looking for. This can actually help our work in SEO, as it means we don’t have to focus quite so intently on specific keywords.

In today’s Whiteboard Friday, Rand explains how focusing on specific kinds of people and the topics they’re interested in can be even more effective in driving valuable traffic than ranking for specific keywords.

Howdy, Moz fans and welcome to another edition of “Whiteboard Friday.” This week, I want to talk to you a little bit about the classic technique of building SEO pages for keywords and rankings versus the more modern technique of trying to do this with people and topics in mind. So, let me walk you through the classic model and show you why we’ve needed to evolve.

So, historically, SEO has really been about keyword rankings. It’s “I want to rank well for this keyword because that particular keyword sends me traffic that is of high quality. The value of the people visiting my site from that is high.” The problem is, this doesn’t account for other types of traffic, channels, and sources, right? We’re just focused on SEO.

This can be a little bit problematic because it can mean that we ignore things like social and content marketing opportunities and email marketing opportunities. But, okay. Let’s stick with it. In order to do this, we do some keyword research. We figure out which terms and phrases are popular, which ones are high and low competition, which ones we expect to drive high-quality traffic.

We create some landing pages for each of these terms and phrases. We get links. And we optimize that content so that hopefully, it performs well in the search engines. And then we measure the success of this process based on both the ranking itself. But also, the keywords that drive traffic to those pages. And whether people who visit coming from those keywords are high-quality visitors.

And then we decide “Yeah, I’m not ranking so well for this keyword. But gosh, it’s sending great traffic. Let me focus more on this one.” Or “Oh, I am ranking well for this. But the keyword is not sending me high-quality traffic. So, it doesn’t matter that much. I’m going to ignore it because of the problems.”

So, a lot of times, creating these landing pages with each particular term and phrase is doing a lot of unnecessary overlapping work, right? Even if you’re not doing this sort of hyper, slight modifications of each phrase. “Brown bowling shoes,” “red bowling shoes,” “blue bowling shoes.” Maybe you could just have a bowling shoes page and then have a list of colors to choose from. Okay.

But even still, you might have “bowling shoes” and “shoes for going bowling.” And “shoes for indoor sports,” all of these different kinds of things that could have a considerable amount of overlap. And many different topic areas do this.

The problem with getting links and optimizing these individual pages is that you’re only getting a page to rank for one particular term or maybe a couple of different terms, versus a group of keywords in a topic that might all be very well-served by the same content, by the same landing page.

And by the way, because you’re doing this, you’re not putting in the same level of effort, energy, quality and improvement, right? Because it’s an improvement into making this content better and better. You’re just trying to churn out landing page after landing page.

And then, if you’re measuring success based on the traffic that the keyword is sending, this isn’t even possible anymore. Because Google has taken away keyword referral data and given us (not provided) instead.

And this is why we’re seeing this big shift to this new model, this more modern model, where SEO is really about the broad performance of search traffic across a website, and about the broad performance of the pages receiving search visits. So, this means that I look at a given set of pages, I look at a section of my site, I look at content areas that I’m investing in, and I say “Gosh, the visits that come from Google, that come from Bing, that come from Image Search, whatever they are, these are performing at a high quality, therefore, I want to invest more in SEO.” Not necessarily “Oh, look. This keyword sent me this good traffic.”

I’m still doing keyword research. I’m still using that same process, right? Where I go and I try to figure out “Okay, how many people are searching for this term? Do I think they’re going to be high-quality visitors? And is the competition low enough to where I think my website can compete?”

I’m going to then define groups of terms and phrases that can be well-served by that content. This is very different. Instead of saying “Blue bowling shoes” and “Brown bowling shoes,” I’m saying, “I think I can have one great page around bowling shoes, in general, that’s going to serve me really well. I’m going to have all different kinds, custom bowling shoes and all these different things.”

And maybe some of them deserve their own individual landing pages, but together, this group of keywords can be served by this page. And then these individual ones have their own targeted pages.

From there, I’m going to optimize for two things that are a little bit different than what I’ve done in the past. Both keyword targeting and being able to earn some links. But also, an opportunity for amplification.

That amplification can come from links. It could come from email marketing, it could come from social media. It could come from word-of-mouth. But, regardless, this is the new fantastic way to earn those signals that seem to correlate with things ranking well.

Links are certainly one of them. But we don’t need the same types of direct anchor text that we used to need. Broad links to a website can now help increase our domain authority, meaning that all of our content ranks well.

Google certainly seems to be getting very good at recognizing relevancy of particular websites around topic areas. Meaning that if I’ve done a good job in the past of showing Google that I’m relevant for a particular topic like bowling shoes. When I put together custom, graphic-printed, leather bowling shoes pages, that page might rank right away. Even if I haven’t done very much work to specifically earn links to it and get anchor text and those kinds of things, because of the relevancy signals I’ve built up in the past. And that’s what this process does.

And now, I can measure success based on how the search traffic to given landing pages is performing. Let me show you an example of this.

And here, I’ve got my example. So, I’m focusing beyond bowling shoes. I’m going to go with “Comparing mobile phone plans,” right? So, let’s say that you’re putting together a site and you want to try and help consumers who are looking at different mobile phone plans, figure out which one they should go with, great.

So, “Compare mobile phone plans” is where you’re starting. And you’re also thinking about ‘Well, okay. Let me expand beyond that. I want to get broad performance.” And so, I’m trying to get this broad audience to target. Everyone who is interested in this topic. All these consumers.

And so, what are things that they also might be interested in? And I’ll do some keyword research and some subject matter research. Maybe I’ll talk to some experts, I’ll talk to some consumers. And I’ll see providers, they’re looking for different phone providers. They might use synonyms of these different terms. They might have some concept expansion that I go through as I’m doing my keyword research.

Maybe I’m looking for queries that people search for before and after. So, after they make the determination if they like this particular provider, then they go look at phones. Or after they determine they like this phone, they want to see which provider offers that phone. Fine, fair.

So, now, I’m going to do this definition of the groups of keywords that I care about. I have comparison in my providers. Verizon, T-Mobile, Sprint, AT&T. Comparison of phones, the Galaxy, iPhone, Nexus, by price or features. What about people who are really heavy into international calling or family plans or travel a lot? Need data-heavy stuff or doing lots of tethering to their laptops.

So, this type of thing is what’s defining the pages that I might build by the searcher’s intent. When they search for keywords around these topics, I’m not necessarily sure that I’m going to be able to capture all of the keywords that they might search for and that’s okay.

I’m going to take these specific phrases that I do put in my keyword research. And then, I’m going to expand out to, “All right, I want to try and have a page that reaches all the people who are looking for stuff like this.” And Google’s actually really helping you with search algorithms like Hummingbird, where they’re expanding the definition of what keyword relevancy and keyword matching is really meaning.

So, now, I’m going to go and I’m going to try and build out these pages. So, I’ve got my phone plans compared. Verizon versus T-Mobile versus AT&T versus Sprint. The showdown.

And that page is going to feature things like “I want to show the price of the services relative to time over time. I want to show which phones they have available.” And maybe pull in some expert ratings and reviews for those particular phones. Maybe I’ll toss in CNET’s rating on each of the phones and link over to that.

What add-ons do they have? What included services? Do I maybe want to link out to some expert reviews? Can I have sorting so that I can say “Oh, I only want this particular phone. So, show me only the providers that have got that phone” or those types of things.

And then, I’m going to take this and I’m going to launch it. All this stuff, all these features are not just there to help be relevant to the search query. They’re to help the searcher and to make this worthy of amplification.

And then, I can use the performance of all the search traffic that lands on any version of this page. So, this page might have lots of different URLs based on the sorting or what features I select or whatever that is. Maybe I rel canonical them or maybe I don’t, because I think it can be expanded out and serve a lot of these different needs. And that’s fine, too.

But this, this is a great way to effectively determine the ROI that I’ve gotten from producing this content, targeting these searchers. And then, I can look at the value from other channels in how search impacts social and social impacts search by looking at multi-channel and multi-touch. It’s really, really cool.

So, yes. SEO has gotten more complex. It’s gotten harder. There’s a little bit of disassociation away from just the keyword and the ranking. But this process still really works and it’s still very powerful. And I think SEOs are going to be using this for a long time to come. We just have to have a switch in our mentality.

All right, everyone. I look forward to the comments. And we’ll see you again next week for another edition of “Whiteboard Friday.” Take care.

I Am an Entity: Hacking the Knowledge Graph

Posted on December 19, 2013 by

For a long time Google has algorithmically led users towards web pages based on search strings, yet over the past few years, we’ve seen many changes which are leading to a more data-driven model of semantic search.

In 2010 Google hit a milestone with its acquisition of Metaweb and its semantic database now known as Freebase. This database helps to make up the Knowledge Graph; an archive of over 570 million of the most searched-for people, places and things (entities), including around 18 billion cross-references. A truly impressive demonstration of what a semantic search engine with structured data can bring to the everyday user.

What has changed?

The surge of Knowledge Graph entries picked up by Dr Pete a few weeks ago indicates a huge change in the algorithm. Google has been attempting to establish a deep associative context around the entities to try and understand the query rather than just regurgitate what it believes is the closest result for some time, but this has been focused on a very tight dataset reserved for high profile people, places and things.

It seems that has changed.

Over the past few weeks, while looking into how the Knowledge Graph pulls data for certain sources, I have made a few general observations and have been tracking what, if any, impact certain practices have on the display of information panels.

If I’m being brutally honest, this experiment was to scratch a personal “itch.” I was interested in the constructs of the Knowledge Graph over anything else, which is why I was so surprised that a few weeks ago I began to see this:

Google Search for "Andrew Isidoro's Age"

It seems that anyone now wishing to find out “Andrew Isidoro’s Age” could now be greeted with not only my age but also my date of birth in an information panel. After a few well-planned boasts to my girlfriend about my new found fame (all of which were dismissed as “slightly sad and geeky”), I began to probe further and found that this was by no means the only piece of information that Google could supply users about me.

It also displayed data such as my place of birth and my Job. It could even answer natural language queries and connect me to other entities like in queries such as: “Where did Andrew Isidoro go to school?“

and somewhat creepily, “Who are Andrew Isidoro’s parents?“.

Many of you may now be a little scared about your own personal privacy, but I have a confession to make. Though I am by no means a celebrity, I do have a Freebase profile. The information that I have inputted into this is now available for all to see as a part of Google’s search product.

I’ve already written about the implications of privacy so I’ll gloss over the ethics for a moment and get right into the mechanics.

How are entities born?

Disclaimer: I’m a long-time user of and contributor to Freebase, I’ve written about its potential uses in search many times and the below represents my opinion based on externally-visible interactions with Freebase and other Google products.

After taking some time to study the subject, there seems to be a structure around how entities are initiated within the Knowledge Graph:

Affinity

As anyone who works with external data will tell you, one of the most challenging tasks is identifying the levels of trust within a data-set. Google is not different here; to be able to offer a definitive answer to a query, they must be confident of its reliability.

After a few experiments with Freebase data, it seems clear that Google are pretty damn sure the string “Andrew Isidoro” is me. There are a few potential reasons for this:

To take a definition from W3C:

“Provenance is information about entities, activities, and people involved in producing a piece of data or thing, which can be used to form assessments about its quality, reliability or trustworthiness.”

In summary, provenance is the ‘who’. It’s about finding the original author, editor and maintainer of data; and through that information Google can begin to make judgements about their data’s credibility.

Google has been very smart with their structuring of Freebase user accounts. To login to your account you are asked to sign in via Google; which of course gives the search giant access to your personal details, and may offer a source of data provenance from a user’s Google+ profile.

Freebase Topic pages also allow us to link a Freebase user profile through the “Users Who Say They Are This Person” property. This begins to add provenance to the inputted data and, depending on the source, could add further trust.

Recently an area of tremendous growth in material for SEOs has been structured data. Understanding the schema.org vocabulary has become a big part of our roles within search but there is still much that isn’t being experimented with.

Once Google crawls web pages with structured markup, it can easily extract and understand structured data based on the markup tags and add it to the Knowledge Graph.

No property has been more overlooked in the last few months than the sameAs relationship. Google has long used two-way verification to authenticate web properties, and even explicitly recommends using sameAs with Freebase within its documentation; so why wouldn’t I try and link my personal webpage (complete with person and location markup) to my Freebase profile? I used a simple itemprop to exhibit the relationship on my personal blog:

<link itemprop="sameAs" href="<a href="http://www.freebase.com/m/0py84hb" >http://www.freebase.com/m/0py84hb</a>">Andrew Isidoro</a>

Finally, my name is by no means common; according to howmanyofme.com there are just 2 people in the U.S. named Andrew Isidoro. What’s more, I am the only person with my name in the Freebase database, which massively reduces the amount of noise when looking for an entity related to a query for my name.

Data sources

Over the past few months, I have written many times about the Knowledge Graph and have had conversations with some fantastic people around how Google decides which queries to show information panels for.

Google uses a number of data sources and it seems that each panel template requires a number of separate data sources to initiate. However, I believe that it is less an information retrieval exercise and more of a verification of data.

Take my age panel example; this information is in the Freebase database yet in order to have the necessary trust in the result, Google must verify it against a secondary source. In their patent for the Knowledge Graph, they constantly make reference to multiple sources of panel data:

“Content including at least one content item obtained from a first resource and at least one second content item obtained from a second resource different than the first resource”

These resources could include any entity provided to Google’s crawlers as structured data, including code marked up with microformats, microdata or RDFa; all of which, when used to their full potential, are particularly good at making relationships between themselves and other resources.

The Knowledge Graph panels access several databases dynamically to identify content items, and it is important to understand that I have only been looking at initiating the Knowledge Graph for a person, not for any other type of panel template. As always, correlation ≠ causation; however it does seem that Freebase is a major player in a number of trusted sources that Google uses to form Knowledge Graph panels.

Search behaviour

As for influencing what might appear in a knowledge panel, there are a lot of different potential sources that information might come from that go beyond just what we might think of when we think of knowledge bases.

Bill Slawski has written on what may affect data within panels; most notably that Google query and click logs are likely being used to see what people are interested in when they perform searches related to an entity. Google search results might also be used to unveil aspects and attributes that might be related to an entity as well.

For example, search for “David Beckham”, and scan through the titles and descriptions for the top 100 search results, and you may see certain terms and phrases appearing frequently. It’s probably not a coincidence that his salary is shown within the Knowledge Graph panel when “David Beckham Net Worth” is the top auto suggest result for his name.

Why now?

Dr Pete wrote a fantastic post a few weeks ago on “The Day the Knowledge Graph Exploded” which highlights what I am beginning to believe was a major turning point in the way Google displays data within panels.

However, where Dr Pete’s “gut feeling is that Google has bumped up the volume on the Knowledge Graph, letting KG entries appear more frequently,” I believe that there was a change in the way they determine the quality of their data. A reduction in affinity threshold needed to display information.

For example, not only did we see an increase in the number of panels displayed but we began to see a few errors in the data:

This error can be traced back to a rogue Freebase entry added in December 2012 (almost a year ago) that sat unnoticed until this “update” put it into the public domain. This suggests that some sort of editorial control was relaxed to allow this information to show, and that Freebase can be used as a single source of data.

For person-based panels, my inclusion seems to show a new era of Knowledge Graph that Dr Pete reported a few weeks ago. We can see that new “things” are being discovered as strings then, using data, free text extraction and natural language processing tools, Google is able to aggregate, clean, normalize and structure information from Freebase and the search index, with the appropriate schema and relational graphs, to create entities.

Despite the brash headline, this post is a single experiment and should not be treated as gospel. Instead, let’s use this as a chance to generate discussion around the changes to the Knowledge Graph, for us to start thinking about our own hypotheses and begin to test them. Please leave any thoughts or comments below.

Entries sought for European Search Awards

Posted on December 19, 2013 by

Entries sought for European Search Awards

The search is on for Europe’s top search and digital talent, as the European Search Awards 2014open for entries.

Now in its third year, the European Search Awards attracts hundreds of entries from the leading search and digital agencies and professionals throughout Europe. Categories include Best Use of Search, Best Pan-European Campaign, Best Mobile Campaign and Best Agency. The awards, which are organised by Greater Manchester based events agency Don’t Panic, are open to companies based worldwide who are delivering work in Europe.

The deadline for entries is 17 January 2014, and the shortlists will be published on 14 February.The judging panel includes:

• Jose Truchado, Director of SEO, Expedia

• Danny Goodwin, Associate Editor of Search Engine Watch

• Gianluca Fiorelli, Founder of LoveSEO

• Kaspar Szymanski, SEO Consultant

• Bas van den Beld, Founder, State of Digital

• Bastian Grimm, Managing Partner, Grimm Digital

• Fernando Maciá Domene, CEO, Human Level Communications

The winners will be announced at an awards ceremony in Reykjavik, Iceland, on Friday 28 March 2014.

Don’t Panic launched the UK Search Awards in 2011 and the European Search Awards in 2012. This year saw the inaugural US Search Awards take place in Las Vegas.

The European Search Awards are delivered in partnership with several sponsors, including Reykjavik Internet Marketing Conference(RIMC) and SEMPO, among others.

For more information about the European Search Awards, please visit http://www.europeansearchawards.com/

The US Search Awards held alongside Pubcon this past October were a big success, with lots of effort by the judges to narrow very competitive fields. Bing was asked to participate in the judging for the US Awards and it was exciting stuff. Lots of work, but thoroughly worthwhile. The European Awards have been around a bit longer and competition is sure to be tough this year.

Bottom line: Review the criteria and if you think someone is worthy of a shout out, submit them! With over 20 categories, you’re bound to know someone who qualifies!

Duane Forrester
Sr. Product Manager
Bing

Top posts by unique pageviews

Top YouMoz posts by unique pageviews

Top Moz Blog posts by number of thumbs up

Top Moz Blog posts by number of comments

Top Moz Blog posts by number of linking root domains

Top comments by number of thumbs up

Most active users by number of comments

Foursquare: the data aggregator

Foursquare: The Data Aggregator?

What’s next for Foursquare?

TL;DR

Why on earth would you kill such a well-known site?

Latent penguin / penalty Issues

Dilution to concentration, juice-wise

Brand, brand, brand

Technical time: merging two domains into one

So, how was it for us?

In conclusion

You can move and not lose, so move away

This may be a successful tactic to escape domain toxicity

Branding could be the single most important factor

How is the 2013 Catalyst CTR study unique?

How have organic CTRs changed over time?

What is the distribution of clicks across a Google SERP?

How do branded query CTRs differ from unbranded queries?

Does query length/word count impact organic CTR?

Potential business uses

Data set, criteria, and methodology

Final word

Watermelon farming

Robo-farmers

Supervised learning

Caffeine infrastructure update

Panda and Penguin

Training

What next?

Wrap-up

What has changed?

How are entities born?

Affinity

Data sources

Search behaviour

Why now?

Entries sought for European Search Awards