Screaming Frog: A Practical Crawl Setup Guide
Screaming Frog is a desktop crawler that visits every URL it can find on your site and records what each one returns, and a good screaming frog setup is mostly a handful of settings: storage, robots handling, scope, rendering and extraction. Get those right before you press Start and the crawl answers real audit questions. Leave the defaults and you get a long spreadsheet of noise.
This guide is for marketers and developers who have installed the tool and want a repeatable configuration, not a tour of every tab. It covers the settings that matter for an audit, the custom extractions worth writing, a test for AI crawler access, and the exports to keep. The wider audit method lives in how to do an SEO audit, and the issues the crawl surfaces are ranked in technical SEO issues.
On this page
- Is the free version of Screaming Frog enough?
- Which storage mode and memory setting should you use?
- Crawl settings to change before the first crawl
- Does Screaming Frog render JavaScript?
- Custom extraction and custom search for audits
- How do you check AI crawler access with Screaming Frog?
- Exports and reports worth keeping
- Putting your Screaming Frog setup to work
Is the free version of Screaming Frog enough?
The free version is enough for a small site and for learning the interface, because it is capped at 500 URLs per crawl. Screaming Frog’s SEO Spider page lists the paid licence at £199 per year, with the same feature list apart from that cap. The page’s own “About” text is less consistent: it says the free version also restricts crawl configuration, saved crawls, JavaScript rendering, custom extraction and API integrations.
Because the company’s pages disagree, treat the paid licence as the version this guide assumes. The user guide states that saving and opening crawls requires a licence, and saved crawls are the thing you want for before-and-after comparisons. If your site has fewer than 500 URLs, try the free build first and see which settings are greyed out.
The tool runs on Windows, macOS and Linux. Everything below is the same on each.
Which storage mode and memory setting should you use?
Use database storage if you have an SSD, and memory storage only if you do not. That is the recommendation in Screaming Frog’s guide to crawling large websites, which says memory storage is better suited to sites under 500,000 URLs and to machines without an SSD or with little disk space.
Database storage also saves each crawl automatically, so you can reopen it from File > Crawls later. Memory storage needs a manual File > Save.
The same guide gives rough memory sizing for database mode: about 4GB of allocated memory for 2 to 3 million URLs, 8GB for up to 5 million, and 16GB for 10 million. The default allocation is 2GB on a 64-bit install. Raise it under File > Settings > Memory Allocation, and leave at least 2GB free for the operating system.
Most business sites are nowhere near those numbers. The reason to set storage deliberately is not scale, it is that a crawl you can reopen next month is a crawl you can compare against.
Crawl settings to change before the first crawl
Five groups of settings decide whether a crawl is useful: robots handling, user-agent, scope, sitemaps and the respect rules. The diagram below shows the order to work through them.
Scroll the diagram sideways to see all of it.
Robots.txt. The default under Configuration > Robots.txt is to respect the file. Leave it on for a normal audit, because you want to see what a polite bot sees. Internal URLs blocked by robots.txt are shown with status code 0 by default. There is also a Custom Robots option that lets you edit and test a robots.txt for the crawl only, which overrides the live file without changing it. That is the safest way to test a robots.txt change before you deploy it.
User-agent. Under Configuration > User-Agent there are two separate settings: the HTTP request user-agent, which is the header sent with each request, and the robots user-agent, which decides which robots.txt group is followed. There are presets for Googlebot, Bingbot and browsers, and you can type a custom value. We use this for the AI crawler test further down.
Scope. Configuration > Spider > Limits sets a crawl total (the licensed default is 5 million URLs), depth, query-string and folder limits. Exclude and Include take regular expressions. Excluded URLs are not crawled at all, so use Exclude for faceted navigation and session parameters, for example \?price to skip any URL containing that query. Include narrows a crawl to matching URLs, but the start page must link to one or the crawl finds nothing. This is the practical way to apply the logic in crawl budget to a desktop audit.
Sitemaps. XML sitemaps are not crawled by default in regular Spider mode. You can auto-discover them from robots.txt or paste URLs under Crawl These Sitemaps. Do it, because a URL that is in your sitemap but not reachable by links is exactly the orphan-page finding you want. See XML sitemap for what a clean one looks like.
Respect rules. Respect Noindex and Respect Canonical live in the Spider > Advanced tab. When on, noindexed and canonicalized URLs are not reported, though they are still crawled and their outlinks followed. Turn both off for an audit, so those URLs stay visible in the data and you can judge them.
Does Screaming Frog render JavaScript?
Yes, but only if you ask it to. The default reads the raw HTML, and under Configuration > Spider > Rendering you can switch the mode from Text Only to JavaScript, which renders pages in headless Chrome. Rendered page screenshots are on by default in that mode, and the window size is 411 by 731 for mobile and 1024 by 768 for desktop.
The audit move is to crawl twice and compare. A text-only crawl shows what a bot that does not run scripts receives. A JavaScript crawl shows what a browser-like renderer receives. If your titles, main copy or internal links appear only in the second crawl, you have a rendering dependency. Whether a given AI crawler runs scripts is covered, with the fix, in JavaScript rendering and AI crawlers.
Keep resource crawling on if you use rendering. Screaming Frog’s large-site guide says to turn off crawling of images, CSS and JavaScript only when you are not using JavaScript rendering.
Custom extraction and custom search for audits
Custom extraction pulls any value from the page into a column, and custom search flags pages that contain or lack a string. Both are under Configuration > Custom. Extraction supports XPath, CSS Path and regex, with options to extract the HTML element, inner HTML or text. The limits are up to 100 extractors and up to 1,000 extractions across them. Custom Search supports up to 100 filters, each set to contains or does not contain, in text or regex.
Both run on raw HTML by default, so switch on JavaScript rendering if the markup you want is injected by scripts. The extractors below are the ones we reach for first. The expressions are standard XPath written by us, so test each one on a few URLs before trusting a full crawl.
| What you want | Method | Example expression | What a gap tells you |
|---|---|---|---|
| Canonical target | XPath, Extract Text | //link[@rel='canonical']/@href | Missing or self-contradicting canonicals |
| JSON-LD blocks | XPath, Extract Inner HTML | //script[@type='application/ld+json'] | Pages with no structured data |
| Author name | XPath, Extract Text | //meta[@name='author']/@content | Articles with no visible authorship signal |
| Modified date | XPath, Extract Text | //meta[@property='article:modified_time']/@content | Stale or undated content |
| Tracking tag present | Custom Search, contains | googletagmanager.com | Templates missing analytics |
The JSON-LD extraction is the one that pays off most. Export it, then check which templates carry which types, using JSON-LD as the reference for what should be there. Once you have extractions for canonicals and dates, you can review a whole content library in one sheet instead of opening pages one by one.
How do you check AI crawler access with Screaming Frog?
You run the same crawl several times with different settings and compare status codes. Screaming Frog cannot be GPTBot, but it can test the three things that most often block AI crawlers: robots.txt rules, user-agent rules in a firewall or CDN, and content that only exists after JavaScript runs.
First, know the bot names. OpenAI’s crawler documentation lists three: OAI-SearchBot, which surfaces sites in ChatGPT search; GPTBot, which collects content that may be used for training; and ChatGPT-User, used for user-initiated visits. Anthropic’s crawler help page lists ClaudeBot, Claude-User and Claude-SearchBot. The difference between the OpenAI agents is covered in GPTBot vs OAI-SearchBot, and Anthropic’s in ClaudeBot.
Then run four crawls of the same URL set, which works well in List mode (Mode > List) for a sample of 100 to 500 important URLs.
Scroll the diagram sideways to see all of it.
In crawl 2, set only the robots user-agent to the bot token (for example OAI-SearchBot) and respect robots.txt. URLs that robots.txt disallows for that token will appear with status code 0. In crawl 3, set the HTTP request user-agent to a string containing the bot name and let robots.txt be ignored, so the only variable is how your server reacts to the user-agent. In crawl 4, repeat the baseline with rendering switched.
Read the results carefully, because the test has limits. A custom user-agent does not come from OpenAI’s or Anthropic’s IP addresses, and many firewalls decide on IP, not on the header. Anthropic’s page also says blocking by IP address may not work reliably, and it publishes an IP list for verification. So a clean crawl 3 means your user-agent rules are fine. It does not prove the real bot gets through. For that, check your server logs.
One more caveat from OpenAI’s documentation: because ChatGPT-User actions are initiated by a person, robots.txt rules may not apply to it. And a change to robots.txt can take around 24 hours to take effect for OAI-SearchBot. Our post on robots.txt for AI crawlers covers what to allow and block by goal.
Reading the results
The second diagram told you what to change. This one tells you where each symptom is fixed, which matters because the owner is often not the SEO.
Scroll the diagram sideways to see all of it.
A robots block is the one most teams get wrong in the other direction. Google’s robots.txt introduction says robots.txt is not a mechanism for keeping a page out of Google, and that a disallowed page can still be indexed if other sites link to it. Use noindex for that, and make sure the page is crawlable so the tag can be read.
Exports and reports worth keeping
Keep four things from every crawl: the Internal tab export, the Issues export, the redirect and canonical reports, and your custom extraction. The user guide lists the full menus, and most of it you will never open.
| Export or report | Where | Use it for |
|---|---|---|
| Redirect Chains | Reports > Redirects | Chains and loops to flatten, which also feeds redirects and SEO |
| Canonical Chains, Non-Indexable Canonicals | Reports > Canonicals | Canonicals pointing at redirected or noindexed URLs |
| Orphan Pages Report | Reports | Pages with no internal links; blank unless Analytics, Search Console or sitemap data is connected |
| Structured Data validation | Reports > Structured Data | Errors and warnings by type |
| Custom Extraction | Bulk Export, tab-based | Your canonical, JSON-LD and date columns |
| Multi-Export | Bulk Export menu | One click for several tabs and reports, with saved presets |
| AI | Bulk Export, tab-based | Output from any AI prompts you ran |
Two habits help. First, save the crawl and name it with the date, so the next one is a real comparison. Second, filter the Issues export to indexable pages before you send it to anyone. A list that mixes parameter URLs, redirects and real pages gets ignored, and the point of a technical audit checklist is a short list a developer can act on.
Does Screaming Frog use AI?
It can, when you connect a model yourself. The product page lists custom AI prompts with OpenAI, Gemini, Ollama and Anthropic while crawling. The connections sit under Configuration > API Access > AI, and the configuration guide notes that prompts using page text or HTML need Store HTML turned on under Config > Spider > Extraction.
Useful prompts are narrow and checkable: classify each page’s search intent, summarize a page in one line, or flag copy that contradicts its title. Treat the output as a draft that a person reviews, since a model reading a page can be confidently wrong. Also remember that sending page text to a third-party API is a data decision. For client sites, a local model through Ollama keeps the text on your machine.
Do not confuse this with the audit itself. The crawl is deterministic, and the same site gives the same status codes every time. The AI column is an optional annotation on top.
Putting your Screaming Frog setup to work
A good setup is saved and repeatable, so build it once and reuse it. Configure storage, robots, scope, rendering and extractions, then save the configuration so the next audit starts from the same place. Screaming Frog supports saving configurations, and File > Scheduling lets you run a crawl on a schedule.
Run the baseline first, then the AI access tests and the text-only against JavaScript comparison, and keep the exports. The crawl tells you what is on the site and how it responds to a bot. It does not tell you whether AI assistants name you in answers, because that needs prompt testing, not a crawler.
If you want that second half measured alongside the technical crawl, our AI visibility audit combines both, and the Ambeltek AI SEO audit tool gives you a free first pass. Either way, start with the crawl: a clean one makes every later step cheaper.
Frequently asked questions
How much does a Screaming Frog license cost?
The SEO Spider licence is listed at £199 per year on Screaming Frog's site, and it must be renewed each year. The free version crawls up to 500 URLs per crawl. For anything beyond a small site, or if you want saved crawls and rendering, plan on the paid licence.
Does Screaming Frog use AI?
Yes, as an option. You can connect OpenAI, Gemini, Ollama or Anthropic under Configuration > API Access > AI and run your own prompts against each page while it crawls. The crawl itself is not AI. The AI features are prompts you write and review.
What is Screaming Frog good for?
It is a desktop crawler that shows you your site the way a bot walks it: status codes, redirects, titles, canonicals, directives, internal links and structured data for every URL. It is best for technical audits, migrations and bulk checks, not for keyword research or rank tracking.
Can Screaming Frog crawl a site as GPTBot or ClaudeBot?
It can send a custom user-agent in the request header and follow robots.txt as a custom robots user-agent, so you can test how your robots.txt and firewall treat a bot name. It cannot reproduce the bot's real IP addresses, so a pass is not proof the real crawler gets through.
Does Screaming Frog render JavaScript?
Yes. Under Configuration > Spider > Rendering you can switch from Text Only to JavaScript, which renders each page in headless Chrome. It is slower and heavier, so use it when you need to see content and links that only appear after scripts run.
How much memory does Screaming Frog need?
It depends on the storage mode. Screaming Frog's guide suggests roughly 4GB of allocated memory for 2 to 3 million URLs in database mode and 8GB for up to 5 million. For most business sites, the default 64-bit allocation of 2GB is enough for a first crawl.