Technical SEO Audit: A Practical Screaming Frog Guide

You can publish excellent content, invest in design and pay for ads, and still be nowhere to be found on Google. Very often the reason is not in the copy at all but in the technical layer underneath it: pages accidentally blocked from indexing, links that lead to errors, duplicate titles, or two versions of the same page competing with each other.

A technical SEO audit finds these problems before they cost you traffic and leads. The most widely used tool for the job is Screaming Frog SEO Spider: a crawler that moves through your website the way a search engine bot does and collects everything it finds into clear, filterable tables.

In this guide we cover:

• what a technical SEO audit is and why it matters;
• how to get started with Screaming Frog and what the free version includes;
• how to check pages, response codes and broken links;
• what to look for in URLs, page titles and meta descriptions;
• how to deal with duplicate pages using canonical tags;
• how to build a structured action plan and which mistakes to avoid.

What a technical SEO audit is and why it matters

An SEO audit is a review of your website against what search engines need from it. Its goal is simple: find whatever stops Google from crawling your site, understanding it and showing it in search results.

An audit has several layers. A content audit looks at the copy and how well it matches search intent. A backlink audit looks at who links to you. A technical audit focuses on the infrastructure: are all important pages reachable for bots, does the server return the right status codes, are there duplicate pages, and are the meta tags set up properly?

The technical layer is the foundation for everything else. If a service page is blocked from indexing, even the best copy on it will not help. If half of your internal links point to redirects or errors, the bot wastes its crawl resources and visitors get frustrated.

When it makes sense to run a technical audit:

• Before launch of a new website, so you do not go live with test pages in the index;
• After a migration or redesign, when URLs change and redirects are easy to lose;
• When organic traffic drops for no obvious reason;
• On a regular schedule: once a quarter for a small website, and once a month for a large site or an online store.

What Screaming Frog SEO Spider is and how to get started

Screaming Frog SEO Spider is a desktop website crawler built by Screaming Frog, a UK-based SEO agency. You install it on your computer, and it runs on Windows, macOS and Linux. SEO specialists, developers and website owners use it to see a site the way a search engine bot sees it.

The crawler starts from the homepage, follows every link it finds and collects data about each URL: the response code, title, meta description, H1 and H2 headings, inbound and outbound links, indexing directives, the canonical tag and much more. Every report can be exported to CSV or Excel, or sent straight to Google Sheets.

Free or paid version

The tool has a free version that crawls up to 500 URLs per crawl. For a brochure website, a landing page or a small blog that is usually plenty, and it covers the essential checks: response codes, broken links, redirects, page titles and meta descriptions, and duplicate pages.

The paid licence is sold as an annual subscription. It removes the URL limit and unlocks advanced features: saving crawls and comparing them, JavaScript rendering for sites whose content loads in the browser, custom data extraction, scheduled crawls, and integrations with Google Search Console, Google Analytics and PageSpeed Insights. Check the vendor's website for current pricing.

Your first crawl

After downloading and installing the tool, enter your website address in the bar at the top and click Start. The crawler begins collecting data, and an overview panel on the right side of the screen shows how many issues it has found in each category. Clicking a category filters the main table.

Settings to check before you crawl

• Subdomains. By default the tool treats subdomains as external sites. If your blog or shop lives on a subdomain, enable Crawl All Subdomains under Configuration › Spider › Crawl.
• robots.txt. The crawler respects your robots.txt file just as Google does. If you also want to check blocked pages, you can change this behaviour in the settings, but remember that this is not what Google sees.
• JavaScript. If your website is built as an app whose content loads in the browser, turn on JavaScript rendering (paid version). Otherwise the crawler will see almost empty pages.
• Staging site. If you are crawling a password-protected test version, you can enter the login details in the authentication settings.

Pages and response codes: the Internal tab

The Internal tab lists every URL found within your own domain. Click a URL and the lower panel shows everything about it: page size, the server's response code, word count, meta tags, inbound and outbound links and more. You can filter the table by content type, for example HTML only or images only.

The Indexable column

The first column worth checking is Indexable. It shows whether a page can be indexed, and if it cannot, the Indexability Status column explains why: a noindex directive, a canonical tag pointing to another URL, a redirect or an error.

This is how you find two kinds of problems. First, important pages that were blocked by mistake, such as a service page that got a noindex tag during development and kept it after launch. Second, technical pages that have no business being on Google, such as the shopping cart, checkout, internal search results or thank-you pages. Those should be kept out of the index with a noindex directive.

Response codes

Every request to the server gets a response code. In the Response Codes tab you can filter by group:

• 2xx – the page loads normally. This is what you want for every page you link to.
• 3xx – a redirect. A permanent 301 redirect is fine after a URL change, but internal links should point directly to the final URL rather than through a redirect. Chains of several redirects in a row slow down both loading and crawling.
• 4xx – the page does not exist or is blocked. Every internal link to a 404 needs fixing: update the link or redirect it to a relevant page. We cover what a good 404 page looks like in our guide how to build a 404 error page.
• 5xx – a server error. If it keeps happening, it is a hosting or code problem and needs attention right away.

External and broken links: the External tab

The External tab shows every link from your website to other sites. Say a blog post links to a study or to a tool's documentation. Over time that page gets deleted or moves to a new address, and the link breaks. A visitor who clicks it hits an error, and the article loses some of its value.

For each external link the crawler shows its response code and the pages on your site where it appears. To trace the source of a problem, select the broken URL and open the Inlinks tab in the lower panel: it lists every page that links to it.

To see all broken links in one place, both internal and external, export the ready-made 4xx report from the Bulk Export menu. You get a table with the linking page, the broken link and the anchor text, which makes it easy to split the fixes across your team.

Pay attention to internal links too. They distribute authority between pages and help Google understand what matters most on your website. A page that no other page links to will hardly get any traffic. We go deeper into building a sound link structure in our guide to internal linking for SEO.

URLs: what to check

In the URL tab the crawler flags addresses with problematic characteristics: uppercase letters, underscores, non-ASCII characters, parameters, overly long URLs and duplicates.

Non-Latin characters in URLs

Google can crawl and rank URLs written in Greek, Cyrillic, Arabic or any other script. The trouble starts when people share them: in WhatsApp, in an email or on social media the address turns into a long string of encoded characters such as %CE%B1%CF%81%CF%87%CE%B9%CE%BA%CE%AE, which looks suspicious and breaks easily. That is why, even on websites in other languages, we recommend short URLs in Latin characters, using English words or a transliteration.

Length and structure

A long URL with many nested folders often signals an overly complicated site structure. A good URL is short, readable and describes the content. Use hyphens rather than underscores between words, and avoid uppercase letters, because on some servers /About and /about are two different pages.

Parameters

URLs with parameters, such as sorting or filtering in an online store, can generate thousands of versions of the same page. If the crawler finds many of them, you need to decide which ones are allowed to appear on Google and which should carry a canonical tag pointing to the main page.

Page titles and meta descriptions

The title tag is the headline shown in search results and in the browser tab. The meta description is the short text Google may show beneath it. Both influence whether a searcher decides to click on your result.

In the Page Titles and Meta Description tabs the crawler flags:

• pages with no title or description at all;
• duplicate titles and descriptions, meaning the same text on several pages;
• text that is too long or too short;
• a title identical to the page's H1;
• pages with more than one title tag.

Duplicates are the most common problem. On websites built from a template, many pages end up with the same title as the homepage, and Google cannot tell what makes them different. In online stores the same thing happens on similar product pages.

There is no magic number for length. Google truncates titles by pixel width rather than character count, and it sometimes replaces your description with text taken from the page. You can set the limits that suit you under Configuration › Spider › Preferences, and the crawler will flag anything outside them. We explain how to write titles and descriptions that earn clicks in our guide to search result snippets.

While you are at it, check the H1 and H2 tabs as well: a page without an H1, with several H1s, or with the same H1 as other pages points to a problem in the template structure.

Canonical tags and duplicate pages: the Canonicals tab

Duplicate pages are pages with identical or nearly identical content at different URLs. They appear easily: a version with and without www, with http and https, with and without a trailing slash, with tracking parameters, or when the same blog is available both on the main domain and on a subdomain. When several versions get indexed, they compete with each other and split the page's authority.

There are two main ways to handle duplicates:

• A 301 redirect – when the duplicate version should not be reachable at all, for example the http version or an old URL after a migration.
• A canonical tag – when the version needs to stay available to visitors but you want to tell Google which URL is the main one. For example, a product page opened with filter parameters.

The Canonicals tab shows which pages have no canonical tag, which have several conflicting ones, and which point to a URL that does not return a 200 code or is blocked from indexing. The key rule: the canonical URL must be a live, accessible, indexable page. A canonical pointing to a redirect or an error is usually a sign of a configuration mistake.

On multilingual websites, for example English and Greek or English and Portuguese, check the Hreflang tab as well: each version should point to its counterpart in the other language, and should not mark it as canonical.

Action plan: a technical audit in 7 steps

The crawler will turn up a lot of findings, and not all of them are urgent. To make sure the audit leads to actual fixes rather than just a long spreadsheet, work through it in order.

Step 1: Start with Google Search Console

Before you crawl, open the Page indexing report in Google Search Console. It shows which pages Google has actually indexed and why it rejected others. It is the only official source on what Google sees, and the crawler complements it.

Step 2: Crawl the whole website

Run a full crawl and save it (paid version), so you can compare it with the next crawl after the fixes.

Step 3: Check indexing

Make sure every important page is indexable and technical pages are not. Compare the picture with your sitemap (sitemap.xml): it should list only live pages that return a 200 code.

Step 4: Fix errors and redirects

Fix internal links to 404 pages, update links that pass through redirects and break up redirect chains. After a migration this is the most critical step.

Step 5: Sort out duplicates

Choose one main URL for each piece of content, set up 301 redirects and canonical tags as needed, and make sure every version of the domain redirects to a single one.

Step 6: Improve meta tags and headings

Write unique titles and descriptions for the most important pages first: the homepage, service pages, categories and top products. Then move on to the rest.

Step 7: Check speed and follow up

Connect the crawler to PageSpeed Insights or check your key pages manually. We explain what affects speed and how to improve it in our guide to website speed optimization. Then put a date in the calendar for the next audit.

Other audit tools

Screaming Frog is not the only option. Sitebulb gives you more visual reports with explanations of each issue. Ahrefs Webmaster Tools offers a free site audit for verified website owners, and Semrush has a cloud-based Site Audit tool that can run on an automatic schedule. Whatever you choose, Google Search Console remains the first must-have.

Common audit mistakes

• Fixing everything at once. Deal with whatever blocks indexing and breaks links first, and only then with description lengths.
• Relying on the crawler without Search Console. The crawler shows what can be crawled, not what Google decided to index.
• Crawling the staging site instead of the live one, or the other way round, and fixing the wrong version.
• Not checking after launch. Every migration, redesign or change to the URL structure calls for a fresh crawl.
• Treating every finding as an error. Some flags are recommendations. Decide based on the context of your own website.

How RJL Studio runs technical SEO audits

At RJL Studio a technical audit is a standard part of every launch, migration and redesign. We crawl the website, cross-check the findings against Google Search Console and fix what we find ourselves: 301 redirects, canonical tags, meta tags, the sitemap, indexing settings, hreflang for multilingual sites and loading speed. On Webflow sites we also make sure the staging version stays out of the index and that every old URL gets a redirect after a platform switch.

Want to know what is holding your website back on Google? Read about our SEO services and Webflow development, or get in touch and we will get back to you.

Frequently asked questions

How long does a technical SEO audit take?

Crawling a small business website takes a few minutes, but the analysis and the fixes are where the real work is. For a site with a few dozen pages, an audit with a fix plan usually takes anywhere from one day to a few working days. An online store with thousands of products takes longer.

Is the free version of Screaming Frog enough?

For a website with up to 500 URLs, including images and files, it covers the essential checks: response codes, broken links, meta tags and duplicates. For large websites, JavaScript-heavy sites and comparing crawls over time, you need a paid licence.

What is the difference between Screaming Frog and Google Search Console?

Google Search Console shows what Google itself has crawled, indexed and displayed, including clicks and impressions. Screaming Frog crawls your website from the outside and shows what a bot can find on it. Use both together: the crawler finds the problems, and Search Console confirms their impact.

How often should you run a technical audit?

For a small website that does not change much, a check once a quarter is enough. For an online store, a site with an active blog, or after any major change such as a migration or redesign, crawl more often, and always right after launch.

Can you run a technical audit without coding skills?

You can learn to run a crawl and read the reports without any programming. Some fixes, such as titles and descriptions, are made in your content management system. Others, such as redirects, canonical tags or rendering issues, usually need a developer or an agency that knows the platform.