Indexing: A tale of two search engines

Messy SEO is a column covering the nitty-gritty, unpolished tasks involved in the auditing, planning, and optimization of websites, using MarTech’s new domain as a case study.

This installment of “Messy SEO” details my process of working with our team to analyze indexing patterns for MarTech’s pages. In Part 6, we discussed the necessity of creating pillar pages to establish a better site hierarchy and rank for our most relevant topics.

RELATED: Google lets you report an indexing issue

MarTech.org has had many indexing issues since its creation last year. The most pressing one lately is that Google seems to be prioritizing outdated content in the SERPs, meaning many of the (now redirected) Marketing Land and MarTech Today URLs are still populating the index. As a result, the majority of MarTech’s top-performing pages are irrelevant to our brand as it exists today. 

One of the ways we’re addressing this issue is by creating pillar pages that center on the main industry topics we cover at MarTech. This will help us establish a hierarchy of relevant topics.

We’ve primarily focused on Google’s indexation throughout this process, neglecting to review the ways other search engines have treated our content. So, we decided to compare the MarTech, Marketing Land, and MarTech Today data from Google with that from Microsoft Bing – and the discrepancies were telling.

Indexing status almost a year after migration and consolidation

There have been a lot of changes to MarTech’s indexing since the migration, most notably the title change issues. Thankfully, these were largely resolved, but there are some other issues we found when comparing the content indexed on Google and that on Bing.

Google’s indexing

Despite many lingering indexing issues, Google has made some adjustments to MarTech’s indexation over the past year. The search engine removed virtually all of our duplicate URLs after we set up our redirects, and a good portion of Marketing Land and MarTech Today pages have been removed as well. However, we’ve recently noticed some interesting performance and indexing trends.

Performance. The majority of the top pages from the past three months in terms of interaction are legacy pages that have little to no relevance to our MarTech brand. Aside from the homepage, the “What is MarTech” page, and our CDP platform page, the top URLs are largely irrelevant to our target audience.

Granted, these articles have been live for years, building up authority on the Marketing Land and MarTech Today domains. But, after almost a year of MarTech being live, it’s odd that there are so many old, less relevant pages sitting at the top of our performance lists – especially when our team has published so much good content since then.

Indexed pages. Google has roughly 29,000 MarTech URLs in its index. The majority of these are relevant links we’ve placed in our sitemaps. However, there are over 7,000 URLs in the “Indexed, not submitted in sitemap” category. Many of these URLs are irrelevant — a disconcerting number have parameters that look like either tracking code or, in some cases, spam.

The prevalence of URL parameters isn’t surprising, but it’s not clear why Google is including so many of these in the index. The more alarming trend, however, is the number of Marketing Land and MarTech Today URLs that are still in Google’s index as well.

We know that there are plenty of Marketing Land and MarTech Today URLs online, both in our older pieces of content and on other websites. But it’s strange to see so many still in Google’s index.

Bing’s indexing

Bing’s indexing tells a different story. Though there are still plenty of irrelevant content pieces, they’re much less prominent in the SERPs.

Performance. MarTech’s top-performing pages on Bing look somewhat similar to those on Google. The homepage, “What is MarTech” page, and legacy pages are still there, but we also found one of our more recent news articles in the mix. The importance of the piece to our industry undoubtedly helped bring it to the forefront, but it’s peculiar that Google didn’t treat it the same way.

This newer article’s numbers are encouraging, but, just like the results on Google, our more relevant topic pages are failing to perform well.

Indexed pages. Bing has indexed fewer of our MarTech pages (roughly 17,000 URLs), which isn’t surprising, given how much smaller it is than Google. However, after analyzing these URLs, we found the ratio of relevant content to irrelevant content to be much lower. We’re not seeing a huge number of indexed URLs with parameters.

The most glaring difference between the two search engines is their indexing of our old domain pages. While Google still retains over 2,000 URLs from Marketing Land and MarTech Today, there are only 143 of these URLs left in Bing’s index.

Yes, Bing had fewer of these pages to begin with, but the inconsistency is still shocking.

A discrepancy between Google and Bing’s indexing

Of the two search engines, it seems Bing is doing a better job of crawling our old URLs and adjusting its index accordingly. This makes sense — there are fewer pages indexed on Bing, so the search engine has less to clean up.

But why is Google holding on to so many of these old URLs? One possible explanation is that it simply hasn’t crawled all of the old URLs yet. This would mean it hasn’t found the 301 redirects we put in place, believing the old sites are still live.

This seems unlikely, however, as we migrated the site almost a year ago. Google has had plenty of time to crawl our pages. Yet, we’re still open to this possibility.

Another explanation could be that there’s a structural issue on the MarTech site that is somehow telling Google the old domains are still live. We’re conducting some deep technical audits at the moment to determine if this is true. Until we know more, we’re going to continue to create good content and do all we can to help it rank higher than the less relevant pages.

Have you noticed discrepancies in indexing between Google and Bing? How are you addressing the issue? Email me at [email protected] with the subject line “Messy SEO Part 7” to let me know.

More Messy SEO

Read more about our new MarTech domain’s SEO case study.

• Part 1: Navigating a site consolidation migration
• Part 2: The importance of canonicalization
• Part 3: How to find cached images and improve user experience
• Part 4: When Google’s title change goes wrong
• Part 5: Fixing site structure while a Google title change sinks clickthroughs
• Part 6: Pillar pages and topic clusters