Optimising goes beyond content, your website also deserves a once-over.
There are many technical SEO factors that impact your website’s ranking. But before we get into those, we first need to understand the difference between crawling and indexing.
Crawling vs Indexing
Even though these terms are often confused, they are 2 very different things.
A crawler, also known as a spider, is an automatic computer programme that visits your website and its pages. This is called crawling.
Indexing on the other hand, happens when your pages are added to search engine’s result pages.
While crawling needs to happen in order for indexing to happen, search engines can still decide against indexing your page or website after crawling. The goal of technical SEO is to make crawling, and subsequent indexing, as easy as possible for Googlebot and other crawlers. If all of your pages can be crawled in one go, you’ll have better chances of achieving high rankings for relevant keywords.
Technical SEO Factors Impacting Crawling
Robots.txt
Robots.txt is a text-file planted in the root of your website. The purpose of this file is to tell Googlebot what parts of your site can be crawled (and thus indexed). Clear examples of pages you’d best exclude from crawling are shopping baskets in online shops, profile pages, or search result pages.
Sitemap.xml
For Googlebot to be able to easily crawl your pages, you need a clear sitemap. This is exactly why the sitemap.xml-file was introduced, another text-file in the root of your website. Sitemap.xml provides a clear overview of your site’s pages and their hierarchy.
Google has a tool called Google Search Console, where you can submit your sitemap. During future visits, Googlebot will follow your sitemap when crawling your website. Important to note here is that some tools will automatically generate a sitemap when you use them, make sure to always check whether or not the included pages are all relevant.
HTTP Status Code
When servers respond to requests for webpages, there will always be a HTTP status code included. You want that to be a 2xx status code, which means the request was processed successfully. Pages returning 4xx or 5xx status codes cannot be crawled, these signify client errors and server errors respectively.
SEO-Friendly Site Architecture
You can help Googlebot efficiently crawl your website by implementing a clear page structure — based on your keywords — and internal links that show the pages’ hierarchy. Bread crumbs are a good addition here to further clarify structure and linking.
Technical SEO Factors Impacting Indexing
Robots Tag
The robots tag, not to be confused with robots.txt, indicates whether or not a page can be included in indexing. You can also decide whether or not Googlebot can follow links on that page.
In this first example, Google will both index the page and follow links. In the second example, Google will not index the page, but will follow links.
<meta name="robots" content="index, follow"><meta name="robots" content="noindex, follow">The first robots tag, with content ‘index, follow’, is the standard tag in websites.
Canonical Tags
Duplicate content is a well-known phenomenon in online marketing and SEO. We call something duplicate content when two or more pages on your website contain near-identical text, or when a page on your website is very similar to another website’s page.
You can sometimes deliberately add duplicate content to your website, in which case you can easily tell search engines which page is the original by including canonical tags.
However, duplicate content can also be generated because of issues in your website architecture. URLs containing parameters is an example we see regularly. In cases like this, canonical tags alone will not solve the problem, you should let an expert give your website a detailed once-over.
Page Speed
Time is money. And you can take that literally, because page speed is one of the most important factors in user experience. And user experience is, in its turn, an important ranking factor in search engines. If your pages’ speed is not up to par, Google and other search engines might exclude your website from their SERP.
SEO Best Practices
Now that Googlebot has crawled your pages, and they have been indexed, the real work can begin. Obviously, you won’t be happy with positions on the 6th page in SERP, you’ll want to end up on the first page, alongside your competitors.
But how do you do that? There are roughly 250 factors that contribute to your website’s ranking in search results. These factors can be split into what happens ‘on’ your website (content, technical SEO, keywords, and UX), and what happens ‘around’ your website (backlinks, social media, and brand mentions).
Websites continuously scoring on the first page of search results use a combination of a strong content strategy — based on relevant keywords — with a clear structure, and a strong backlink profile. These sites’ UX and technical details have all been fine-tuned.
Recent Publications
