Crawlability

Written by Backbone Tutorials Team

Last updated: June 2026 · 9 min read

Before a page can rank, or even be indexed, a crawler has to reach it and fetch everything it needs. That sounds trivial until you realise how many ways a site quietly blocks its own crawler: pages nothing links to, scripts disallowed by accident, soft 404s that look like content, and thousands of junk URLs that soak up the crawler's attention. Crawlability is the unglamorous layer underneath everything else in this cluster.

This guide covers how crawlers discover and reach your pages, and the handful of mistakes that keep them out. It extends how Google crawls JavaScript and supports the SEO for single page apps overview.

Discovery: links and sitemaps

A crawler can only fetch URLs it knows about, and it learns them in two ways: by following links and by reading your sitemap. A page with no internal links pointing at it, and not in the sitemap, is an orphan, effectively invisible no matter how good it is.

Sitemaps and internal links

Keep an accurate sitemap that lists your canonical URLs, and make sure every important page is reachable through ordinary links from pages that already get crawled. The two reinforce each other: the sitemap is a hint, internal links are the real path, and together they keep new and deep pages discoverable.

robots.txt: allow what must render

The robots.txt file decides what a crawler is allowed to fetch. The crucial subtlety, repeated from the crawling guide, is that it controls crawling, not indexing, and that blocking resources breaks rendering.

Disallow controls crawling, not indexing

User-agent: *
Allow: /
Disallow: /search?
# never block the JS/CSS needed to render
Allow: /assets/
Sitemap: https://example.com/sitemap.xml

Disallow genuinely useless paths like internal search results, but leave your scripts, styles, and content crawlable. And remember what Disallow does not do: it does not remove a page from the index, and a blocked URL can still appear as a bare link. To keep something out, you need a different tool, covered at the end.

Status codes the crawler trusts

Crawlers read HTTP status codes literally, so the status you return is a signal in its own right. The trouble starts when the status and the content disagree.

200, 301, 404, and soft 404s

Return 200 for real content, 301 for a permanent move, and a true 404 or 410 for something gone. The classic single page app bug is the soft 404: a missing item renders an empty "not found" view but still returns 200, so Google sees a thin page where it should see an error. Make genuine not-found states return a real 404, and keep redirect chains short.

Canonicalization and duplicates

Sites generate duplicates without meaning to: the same content on www and the bare domain, with and without a trailing slash, or under tracking parameters. Left alone, crawlers waste effort and split signals across the copies.

Pointing duplicates at one URL

Pick one canonical version and point the others at it with a canonical link and, where appropriate, 301 redirects. Settle on a single host, serve one form of each URL, and let the canonical tag consolidate parameter variants. This is the same one-canonical-host discipline a healthy site applies everywhere, and it keeps a crawler focused on the version you actually want ranked.

Crawl budget and efficiency

On a small site Google will happily crawl everything, but as URL counts climb, its attention becomes a budget. Spend it on junk and your real pages get crawled less often.

Not wasting crawl on junk URLs

Infinite calendars, faceted filters that multiply into millions of combinations, and session parameters are the usual culprits. Prune them: disallow the patterns that have no search value, avoid linking into infinite spaces, and keep the crawlable URL set close to the pages you actually want indexed. Most small sites never feel this, but large catalogs live or die by it.

noindex versus disallow

The most common crawlability mistake is reaching for the wrong tool to remove a page. Disallow and noindex sound interchangeable and do almost opposite things.

Use noindex to remove, not robots.txt

<meta name="robots" content="noindex, follow">

To keep a page out of the index, mark it noindex and leave it crawlable, so Google can fetch the page and actually see the directive. If you Disallow it instead, Google cannot crawl it, so it never sees the noindex, and the URL can still surface as a bare listing. Crawlable plus noindex removes a page; blocked does not.

Crawlability is the quiet prerequisite for all of it: discoverable through links and the sitemap, fetchable past robots.txt, honest in its status codes, consolidated to one canonical, efficient with crawl, and using noindex rather than a block to remove pages. Get this layer right and the renderer always has something real to work with. Next, the cluster turns those checks into a repeatable JavaScript SEO audit.

Read next: back to the JavaScript SEO hub, or revisit how Google crawls JavaScript for the stage this feeds.