Help Google help you.
Learn how to identify common crawling, rendering and indexing problems and how to fix them.
Why technical SEO and content marketing should come before link-building.
For most websites, link-building should be the last thing you should do. I always tell clients that links are like a steroid injection. When done poorly, link-building can hurt your website.
Your top priority should be building out your website’s topical relevance. This means mapping out your services and/or products, identifying all possible questions that a person may have before they become a customer, and authoring content that solves their questions.
But all your valuable content marketing won’t benefit you if search engines cannot find and understand the information on your pages. This is where technical SEO comes in.
Copy my 170+ point technical SEO checklist that I use to audit Wordpress websites.
Let’s get started.
- What is crawling?
- Is Google crawling your website?
- Does crawl budget affect your website?
- What is a robots.txt file?
- How to understand your robots.txt file
- Common robots.txt file mistakes you should avoid
- Robots.txt for Shopify – best practices
- Robots.txt for Wordpress – best practices
- Robots.txt for ecommerce sites – best practices
- What is a sitemap?
- Should your sitemap be compressed?
- Are sitemaps necessary?
- Can you have more than one sitemap?
- How to use Screaming Frog to identify common sitemap issues
- How to use Sitebulb to identify common sitemap issues
- For publishers: pagination in the context of crawling
- For ecommerce sites: how to execute pagination for optimal crawling
- What is rendering?
- 3 tools to help you diagnose whether Google can render your page’s content
- Common rendering problems and how you can fix them
- What is indexing?
- How to know if Google has indexed the pages on your website?
- What are the common problems that prevent Google from indexing the pages on your website?
- Are noindex tags bad?
- When should you use a noindex tag?
- Should you noindex a page or apply a 301 redirect or canonicalize it to another URL?
- Should you 301, 404 or 410 a URL?
- What is pagination?
- Doing ecommerce pagination right: a framework you can follow
- What is a canonical URL?
- What is duplicate content?
- How duplicate content on your website can hurt your SEO
- What should you do with duplicate content?
- Wordpress: how to fix a trailing slash or non-trailing slash issue
- What is thin content, why thin content can hurt your SEO, and what steps should you take?
- How to identify thin content URLs using Screaming Frog
- How to identify thin content URLs using Sitebulb
- What are redirect chains, why are they bad, and how you can find redirect chains and fix them using Screaming Frog and Sitebulb
- Should you worry about URLs marked as “indexed, not submitted in sitemap?”
- How to make sense of “excluded” URLs in Google Search Console coverage
- How to make sense of error messages in Google Search Console coverage
- What is a site migration and why should you consider doing one
- Common site migration mistakes
- Pre-launch checklist
- Staging checklist
- Launch-day checklist
- Post-launch checklist
- Schema for local businesses with one or more locations
- Have you put SameAs schema on your about page?
- What is FAQ schema markup and where should it go on a page?
- Should you put FAQPage markup on a product page or on a category page?
- How to add JSON-LD to a Wordpress website using a plugin
- How to add FAQPage JSON-LD to a Squarespace website
- How to add FAQPage markup on a Shopify collections page
- How to add clickable hyperlinks inside your FAQPage rich result
- How to verify that your FAQ schema markup is valid
- Why geo-redirects are a bad idea
- Hreflang vs sitemap
- Expanding internationally? How to add international SEO to a website
Things you should know.
What is crawling?
Crawling refers to the discovery process that a search engine must take in order to find all the URLs on a website. Search engines such as Google must be able to crawl a website before it can consider adding URLs to its index.
Therefore, in the context of technical SEO, you want to reduce all possible barriers that are making it difficult for search engines to discover pages, images, and other files that you want shown on search engine result pages.
What is rendering?
Rendering is the process where a search engine visits a URL and sees what information is found on the page. To do this, it will run the code on the page and spend a few seconds to see what is displayed.
However, many things can prevent search engines from efficiently seeing the content on your webpages and the goal of technical SEO is to reduce/remove these as much as possible.
What is indexing?
Indexing is the process how and where a search engine places a URL on its result pages. This is often referred to as ‘ranking’.
Most search engines do not publicly disclose how their indexing system works and Google is frequently updating its indexing algorithm to provide better search results for their users.
Unlike crawling and rendering, indexing is not something you can directly influence with ease.
What is a robots.txt file?
A robots.txt file tells search engine crawlers which URLs the crawler can access on your site (source).
Should you use a 404, 301 or 410?
When you start removing discontinued products or irrelevant pages on your website, you will come across a 404, 301 and 410 decision tree.
Each decision and the correct course of action will depend on the context of what you’re trying to achieve.
A 301 redirect should be used to point users to a better experience on another URL. For example, a CMS-generated category or tag archive page tends to offer very little value to a web visitor. In these instances, you can create a custom page for each category or tag taxonomy and 301 these to the custom URLs. I do this a lot for WooCommerce websites so that category pages are keyword targeted and offer useful information to the user when they land on it from a Google search.
A common mistake people make is 301 redirecting a deleted page to their homepage of category page. This should only be done if the page has inbound backlinks from noteworthy websites.
In most cases, an outdated page with no backlinks should serve a 404 ‘page not found’ status.
If you’re updating a website’s site structure (e.g., introducing a physical subfolder for a category), you should use a 301 to point users and search engines to the new location of the page within your website.
For discontinued products, I recommend creating a custom page that communicates to the user that the item is no longer available and provide them with suitable alternatives.
Most CMS platforms will serve a 404 by default when you delete a page on your website. After a certain period of time, search engines will stop showing the URL in its SERPs as it understands that the page no longer exists.
When deleting a page, ensure that all internal links to the deleted page are removed. There is nothing more frustrating for a user than clicking on a link and landing on a ‘page not found’ greeting.
There aren’t many reasons to use a 410 these days. You’re better served by using a 404 and using the URL removal tool in Google Search Console so that the URL is no longer served on Google SERPs.
What is a canonical URL?
A canonical URL is the URL of the page that a search engine thinks is most representative from a set of duplicate pages on your site.
What is pagination?
Pagination is the process of dividing a document into discrete pages, either electronic pages or printed pages – (source).
It is common for ecommerce sites to have hundreds if not thousands of product URLs. Similarly, publishers, comparison websites and sites that are content marketing focused will have hundreds of URLs within a category/archive page.
To address this, pagination was introduced to show an archive of all these URLs where recent products/articles are shown earlier in the pagination series.
For most ecommerce websites, pagination is used for splitting up product category pages into easy-to-consume pages where a maximum of 10-25 products are shown on each URL. Subsequent products are then shown in a paginated series of pages.
For publishers, pagination is used to categorise related URLs and show users and search engines all the pages that are relevant to a given topic.