Technical SEO Fundamentals: A Practical Guide to Crawlability, Indexation, Site Architecture, and Core Web Vitals

What is technical SEO?

Technical SEO is the work of making sure a search engine can find your pages, fetch them, process their content and links, understand what they are, choose whether to include them in its index, and show them appropriately. It sits underneath content and links. A brilliant page that cannot be fetched or processed has no chance to be shown, and technical SEO exists to remove that kind of avoidable barrier.

That matters because crawler software produces long lists of warnings, and many describe harmless or intentional behaviour. Fixing all of them costs developer time and can introduce new faults. Fixing the few that genuinely affect discovery, access, rendering, indexing or presentation is where the value is. Technical SEO also does not guarantee rankings: it makes eligibility possible, and relevance, quality and competition decide the rest.

The six principles of technical SEO

This guide organises technical SEO into six principles. They describe distinct jobs in search processing and give you a diagnostic order: when a page is missing from results, ask at which stage it stopped. This is a practical model, not a claim that every search engine handles every URL in one strict line. Stages overlap, repeat and get skipped, and some decisions are not publicly observable.

A useful habit: name the stage before proposing a fix. “The page is not indexed” is a symptom. “The page is crawled and rendered but treated as a duplicate” is a diagnosis at the indexing stage, and it points to different evidence and a different fix than “the page is not linked from anywhere”.

Discoverability: how search engines find URLs

A URL can exist on your server and in your CMS without a search engine knowing about it. Discovery is the step where the URL becomes known, usually by following a link from a page already known, by reading an XML sitemap, or through other signals such as links from other websites. I found no Google statement that lists every discovery signal, so treat any complete list with caution.

The main discovery paths

• Internal links. Google documents that it finds links in a elements with an href attribute, and that every page should have at least one link from another page.
• XML sitemaps. A sitemap tells search engines which URLs you consider important and can carry last modified dates. Listing a URL does not ensure it is crawled or indexed, and Google says sites with roughly 500 pages or fewer that are well linked internally may not need one.
• External links. Links from other sites can surface a URL, but you cannot rely on them for your own pages.
• Recently changed pages. Updated URLs are found again through recrawling of known pages, links and sitemap last modified dates, with no fixed schedule.

Where discovery fails

An orphan page has no meaningful internal links pointing to it. A common cause is a CMS page that is published but missing from navigation, category listings and contextual links, so only a sitemap or a direct address reveals it. Dynamic routes add another risk: filters, internal search and interaction-generated URLs can create huge numbers of discoverable URLs, many of them unwanted, while the pages you do want stay buried.

How to investigate a page missing from a crawl

Do not start by resubmitting the sitemap. Compare sources first, because each one answers a different question.

Then separate the causes: missing links (orphan), absent sitemap entry, a blocked route, an inaccessible page, or simply a delay on a new URL. Only the last needs patience. The others need a specific change, and resubmitting a sitemap fixes none of them. Scenario 1 below works through this.

Crawlability: can search engines access the website?

Crawlability is the ability of a crawler to request a URL and retrieve the resources needed to process it. Permission to request a URL is not the same as obtaining useful content: a server can answer with a 200 status and an empty shell, and a page can load while the scripts it depends on fail. Both look “crawlable” in a simple check and still fail the real job.

Different failures, different evidence (user-agent access)

Each piece of evidence shows something different. robots.txt shows permission. HTTP responses and headers show what the server returned. Server logs show what crawlers actually requested. Rendered output shows what was processed. No single one covers all four.

A browser and a search crawler are not interchangeable. A request with a Googlebot user agent string from your laptop is not Googlebot, and rules in a firewall or CDN may treat the two differently. Google documents a way to verify its crawlers, using a reverse DNS lookup confirmed by a forward lookup, or matching against its published IP ranges. The URL Inspection live test uses Google’s own fetcher, so it is better evidence than a spoofed user agent.

Robots.txt, status codes and redirects

robots.txt tells compliant crawlers which paths they may request, mainly to manage server load. Google documents that it is not a mechanism for keeping a page out of Google. Check the actual response, because the handling of the file itself is specific:

• A file applies to one exact host, protocol and port. A subdomain needs its own.
• 4xx responses other than 429 are treated as if no robots.txt exists. Do not use 401 or 403 to limit crawling.
• 5xx responses and unreachable files pause crawling for a period, then Google may use the last good cached copy for up to 30 days.
• Rule paths are case sensitive, * and $ are supported, and the most specific rule wins. When rules conflict, the least restrictive applies.

URL parameters, faceted navigation and crawl budget

Parameters, filters and internal search can create near-endless URL combinations, called crawl traps when a crawler can keep following new variants. They are covered in detail in the robots.txt section below. Crawl budget is the amount of crawling a search engine is willing and able to do on your site. Google says its guide is aimed at sites with roughly a million or more unique pages that change about weekly, sites with 10,000 or more pages changing daily, or sites with many URLs showing “Discovered – currently not indexed”, and calls the figures rough estimates. For a small site, keeping the sitemap current and checking the Page indexing report is usually enough. Crawl budget is rarely the first suspect for a few hundred pages.

A blocked resource is not automatically a defect

A crawler may report blocked scripts, styles or images. That matters only when the blocked resource prevents access to content or functionality needed to process the page, for example the script that renders the product list. A blocked analytics tag or chat widget usually has no SEO effect. The way to tell is to compare the rendered output with and without the resource, as the JavaScript section explains.

How to test a URL

curl -sIL https://www.example.com/old-page/
# Read each hop: status code, Location header, final URL,
# and any X-Robots-Tag or Link: rel="canonical" header.

curl -s https://www.example.com/robots.txt -o /dev/null -w "%{http_code}\n"
# Confirm the robots.txt status code directly.

Then confirm four things: the URL returns a successful response, crawling is permitted for the relevant crawler, any redirect ends at the intended page, and that final page is the one you expect to be indexed. Finish with a URL Inspection live test to see what Google’s own fetcher gets.

Rendering: what do search engines receive and process?

The server returns source HTML. A browser, and Google’s renderer, may then run JavaScript and build the rendered DOM, the page as it exists after scripts finish. Content and links in the rendered DOM are what a user sees, and Google indexes from the rendered HTML.

Google documents a sequence of crawling, then rendering of 200 responses in a render queue using an evergreen Chromium, then indexing, and notes that rendering may be skipped for non-200 pages and pages blocked by robots.txt. It still recommends server-side or pre-rendering, partly because not all bots run JavaScript. So JavaScript is processed, and it is not categorically a problem.

What can go wrong is specific. Content may appear only after a script that fails, links may be built without href, containers may stay empty when an API call errors, lazy-loaded content may never enter view, and a required script or stylesheet may be blocked. The difference between what a user sees and what a search engine processes comes from these gaps. The step-by-step comparison is in the JavaScript SEO section.

Interactions a crawler cannot reliably reproduce

Interactive interfaces are not inherently bad for SEO. The question is whether important URLs, links and content have a reliable path that does not depend on an action a crawler may not perform. Google says it will not load content that requires user interactions such as clicking, swiping or typing.

The general pattern: use real links for destinations, give every item a reachable URL, and treat scripts as an enhancement on top of a working path. Lazy loading is fine when content loads as it enters the viewport, and a crawler or tester should be able to see it without scrolling interactions.

Interpretation: can search engines understand the page?

After content is retrieved and rendered, a search engine still has to work out what the page is, which part is the main content, and how it relates to other pages. Interpretation draws on the rendered HTML, the order and prominence of content, headings, internal links and their anchor text, breadcrumbs, canonical annotations, structured data, language annotations and the consistency between mobile and desktop versions.

• Main content versus boilerplate. Navigation, footers and widgets repeat across pages. The page’s own content should be easy to tell apart and should not be buried in template code or pushed far down by repeated elements.
• Semantic HTML. Elements such as main, nav, article, headings, lists and tables communicate purpose to browsers, assistive technology and crawlers. I treat this as good practice for structure and accessibility, and not a ranking factor.
• Anchor text and links. Descriptive anchor text and real links tell readers and search engines what the destination is about.
• Breadcrumbs and hierarchy. Breadcrumb links show where a page sits and add a crawlable path to its parents.
• Structured data. Google says markup must represent content visible to readers and must not mislead.
• Language and region annotations. Hreflang tells search engines about language versions. See the hreflang section.
• Duplicate or ambiguous content. Near-identical pages make it harder to know which one should represent the topic. Canonicals and consistent links help, covered below.
• Mobile and desktop consistency. The mobile version should carry the same primary content, headings, metadata and structured data as desktop.

Headings: a coherent hierarchy, not a counting exercise

Multiple H1 or H2 elements are not an automatic indexing or ranking failure. I found no Google documentation setting a required number of headings. Google’s title link guidance does say the main title should be the most prominent heading, for example the first visible H1, and that several equally prominent headings can leave no clear main title. A coherent outline helps people scan, helps screen reader users navigate, and keeps the page purpose clear. That is a reason to review headings that are used for styling, not a reason to treat “two H1s” as a defect by itself.

How to inspect interpretation

Open the rendered output, from DevTools or the URL Inspection rendered HTML. Check that the main content appears in a sensible order, that headings describe sections in a logical outline, that link destinations are appropriate, that the canonical, language and structured data match the visible page, and that the mobile version carries the same content.

Related readingUnderstanding Semantic SEO: Beyond Keywords

Covers the content side of this stage.

Indexing: why some crawled pages are not indexed

Discovery, crawling and rendering do not guarantee indexing. A page can be fully processed and still not be included, because of a directive, a canonical choice, duplication, an error response, or a judgement about whether it is worth including. The index is also not fully observable: Google shows a status and some reasons, not its complete internal decision.

Directives and controls

The most common confusion is robots.txt versus noindex. Robots.txt controls crawling. noindex controls indexing, and the crawler must be able to fetch the page to see it. If robots.txt blocks a URL, Google never sees the noindex, and the URL can still appear in results through links from elsewhere.

Reading Search Console index states

Look for patterns. Group affected URLs by template, directory, parameter or content type. A status that affects one URL is an individual case. The same status across every page of one template points at the template or at a site-level quality judgement. Treat a status as a clue and not as proof of cause.

Declared versus selected canonical

The declared canonical is what your page and sitemap say. The Google-selected canonical is what Google chose among similar pages. URL Inspection shows both for the indexed version, but the live test cannot predict the selected canonical. When they differ, compare the content and your signals before changing anything.

Initial uptake and query coverage

Search Console impressions and queries show whether indexed content is appearing for relevant searches, which helps judge whether a newly indexed page is getting any visibility. They are not a direct or complete measure of Google’s index state. A page can be indexed with no impressions, and impressions do not list everything. I do not claim access to internal counts, and you should not invent them.

A diagnostic workflow for a page that is not indexed

1. Inspect the URL. Read the index status, last crawl, declared and selected canonical, and the rendered page.
2. Check the response. A 200, correct headers and no error on the final URL.
3. Check directives. robots.txt, meta robots and X-Robots-Tag, and whether robots.txt hides a noindex.
4. Check canonical signals. Declared canonical, internal links, sitemap entry and redirects, and whether they agree.
5. Check links and sitemap. Is the URL reachable and listed?
6. Check rendered content. Is the main content present after rendering?
7. Check duplication and purpose. Does this page differ meaningfully from similar URLs, and does it deserve its own index entry?
8. Compare patterns. Compare with indexed URLs of the same template before deciding the cause.

Steps 2 to 5 can confirm a technical restriction. If they come back clean, the remaining explanations are duplication, selection and quality, which need judgement and further investigation, and no tool will name the exact reason.

Serving: why indexed pages do not always appear as expected

Being indexed makes a page eligible. It does not guarantee a ranking for any query, nor that the result looks the way you wrote it. Serving depends on query relevance, search intent, content usefulness, broader ranking considerations and competition. Technical correctness cannot supply any of those.

Titles and snippets

Google generates title links automatically and may use other text, for example when a title is half empty, outdated, inaccurate, repeated across many pages, or mismatched with the page language. It states no length limit for the title element, and displays are truncated to fit the device width. Snippets come primarily from page content, and Google may use the meta description when it describes the page better. There is also no stated limit for meta descriptions. Do not assume a title or description will always show as written, and do not assume every one is rewritten either. A rewrite is information, not an automatic defect.

Investigating low impressions or low CTR

Only after those questions should you consider a technical change. Low CTR with strong position suggests presentation. Low position suggests relevance, quality, authority or competition. Both are separate from whether the page is crawlable and indexed. Google also says a page that is indexed and eligible to show a snippet can be a supporting link in AI features, with no special markup required.

Related reading

• How to Build a Content Strategy That Supports SEO: the content side of relevance and intent.
• Building Topical Authority: A Practical Framework for 2026: covering a subject in depth.
• What Is Answer Engine Optimization (AEO)?: how AI answer features relate to indexed pages.

URL architecture and internal linking

Site architecture is how your URLs and links are arranged. Good architecture gives important pages short, clear, crawlable paths and keeps URL variants under control. Google lists simple, descriptive URLs and consistent internal links among its basics, and tells you to link to the canonical URL.

Internal links, link counts and anchors

• Distribution and depth. Check how many clicks separate important pages from the homepage and how many unique internal links each receives.
• Link types. Navigation links repeat sitewide. Contextual links sit in content and show relevance. Breadcrumbs reflect hierarchy. Each serves a different purpose.
• Anchor text. Prefer descriptive, relevant anchors over “click here”. Google recommends descriptive anchor text.
• Bad destinations. Links to redirects, broken URLs, non-canonical variants or non-200 pages waste crawl requests and send mixed signals.
• Sitewide versus contextual. Repeating a link in the footer does not replace a relevant contextual link.

There is no ideal number of internal links per page. Judge whether each link is useful, accessible, relevant and proportionate to the page’s purpose. In a crawler, inspect inlinks, outlinks, link destinations, anchor text and crawl depth, then sort by importance: a key service page with two links needs attention, a utility page with two links does not.

URL variants: consistent handling

The concern is not URL length, case or trailing slashes as penalties. Google notes it treats URLs as case sensitive, so differently cased URLs are distinct URLs, and it advises removing parameters that do not change content. I found no Google statement on trailing slashes. The risk is that variants serve the same content at several addresses, creating duplication, extra redirects and conflicting signals. Pick one form per property and enforce it.

Internal links that pass through redirects

A redirect is often necessary and is not a defect: it protects users, bookmarks and external links after a move. The avoidable problem is your own links pointing at URLs that redirect. Causes include links to old URLs after a migration, HTTP links when HTTPS is canonical, the wrong hostname, inconsistent trailing slashes, and links to a URL that redirects to the canonical. Chains that add hops, and broken or looping destinations, make it worse. Find them in a crawl by listing 3xx internal links with the source page, then update each source link so it points straight at the final canonical destination. Keep the redirect itself in place. See Scenario 5.

Related readingBuilding Topical Authority: A Practical Framework for 2026

A related view of internal linking and topic structure.

Robots.txt, XML sitemaps and URL parameters

These three tools do different jobs and are often confused.

Blocking crawling and controlling indexing interact. If you want a URL out of the index, let it be crawled and use noindex, or remove it with a proper status code. Blocking first hides the instruction. Test your rules against real URLs, because matching depends on the exact path, case and wildcards.

User-agent: *
# Example patterns only. Test every rule against real URLs.
Disallow: /search/
Disallow: /*?sort=
Disallow: /*?sessionid=
Allow: /products/

Parameter patterns that often create duplicate or low-value URLs include sort orders, tracking parameters, session identifiers, internal search queries, and stacked filter combinations. Others, such as a filter that produces a distinctive and wanted category, can be genuine landing pages. Do not block all parameters. For each one, ask whether it changes the meaningful content, whether it only reorders or duplicates, and whether the resulting pages deserve to be found.

Google’s faceted navigation guidance says to disallow crawling of filter URLs you do not need in search, notes that URL fragments are not crawled for this purpose, that canonical and nofollow are less effective long-term, and that filter combinations with no results should return a 404. Blocking resources needed for rendering is a separate risk: a disallow rule that covers the scripts or styles that build the page can make the processed page empty. Base crawl management on actual site behaviour from crawls, Crawl Stats and logs, not on a copied template.

Dynamic routes versus static routes

A dynamically generated URL is not inherently an SEO problem. What matters is whether the route is useful, reachable and not endlessly duplicated. Dynamic routing covers product filters, faceted navigation, search results, sort parameters, pagination, variants, query-string content, infinite scroll and client-side navigation. The decision table below applies the approach to a small set of routes.

Managing auxiliary URLs: purpose first

Search pages, filter combinations, sort URLs, utility pages and duplicate parameter routes exist to support functionality. Decide per URL type whether to leave it as is, canonicalise it, redirect it, return a 404 or 410, block crawling, use noindex, or keep it out of the sitemap. Do not apply noindex to all paginated, filtered or utility pages: some are useful landing pages. And remember that a URL blocked in robots.txt cannot have its noindex read. Only include indexable, canonical URLs in your sitemap.

Canonical tags, redirects and HTTP status codes

A canonical URL is the preferred address for content that exists at more than one URL. Google says redirects and rel="canonical" are strong signals and sitemaps a weak one, that none is guaranteed, that you should use redirects to retire a duplicate and rel="canonical" when the duplicate must stay accessible, and that self-referencing canonicals with absolute URLs are recommended.

Common canonical mistakes

• Canonical pointing to a redirected, broken or non-200 URL.
• Conflicting canonicals declared by different methods for the same page.
• Canonical in conflict with internal links and sitemap entries.
• Using canonicals to combine unrelated pages, or as a way to hide every duplicate.
• Relying on robots.txt or noindex for canonicalisation. Google says not to.

A canonical consistency audit

If Google selects a different URL, that is information. Compare the pages, check each signal above, and decide whether the selected URL is acceptable. If it is, align your signals to it. If not, strengthen the consistency of your own signals, or differentiate or consolidate the pages. Google also notes it prefers HTTPS over HTTP in most cases.

JavaScript SEO and rendered HTML

Source HTML alone can mislead on JavaScript-dependent sites, because the content and links you need may not exist until scripts run. On a conventional server-rendered page, where the main content and links already appear in the source HTML, a simple source check is usually enough. The comparison matters when pages rely on scripts for content, navigation, product lists or pagination.

A source versus rendered comparison, step by step

1. Pick representative URLs. One per template: category, product, article, search results.
2. Read the source HTML. View the raw response with view-source or curl. Note the title, main content, links and any empty containers.
3. Read the rendered DOM. Use DevTools Elements, or the rendered HTML from the URL Inspection live test.
4. Compare the essentials. Is the main content present? Are navigation and pagination links present as a elements with href? Are product and contextual links present? Is metadata where you expect it?
5. Look for failures. Empty containers, error messages, missing images, and failed network requests for scripts or API calls.
6. Check resources. In URL Inspection, review any resources that could not load, then see whether they matter to content or links.
7. Compare URL groups. A JavaScript crawl of a template group shows whether the same gap affects every URL, or only some.

A page that contains JavaScript is not a problem by itself. The question is whether important content and links are present, accessible and processed reliably. Do not state that Google cannot process JavaScript. Google’s documentation says it can, and also recommends server-side or pre-rendering as an aid. Not every other crawler or tool renders pages, so important links being present in the source HTML is a safer design.

Hreflang and multilingual technical SEO

Hreflang annotations tell Google about alternate versions of a page for different languages or regions. It helps the right version be served to the right audience. It does not guarantee a higher ranking in a target country, and a single-language website aimed at one audience does not need it.

• Valid codes. Language uses ISO 639-1, with an optional ISO 3166-1 Alpha 2 region. A region alone is invalid, and reserved region values have no effect.
• Reciprocal links. Each version must link to every other version, and the links must be bidirectional or the annotations are ignored.
• Self reference. Each version must list itself.
• x-default. A fallback for users whose language is not listed, such as a selector page.
• Implementation. HTML link elements, HTTP headers or an XML sitemap. Google says these are equivalent, so choose one and keep it maintainable.
• URL and canonical consistency. Reference the preferred, 200 HTTPS URLs and keep canonicals consistent with the annotations. Avoid HTTP URLs in hreflang.

Common errors are missing return links, invalid or made-up codes, pointing at redirected or non-200 URLs, a canonical pointing at another language version, and clusters built from different sources of truth. Audit clusters with a crawler, check each pair, then fix the generator, not just one page. Localised pages that keep untranslated main content can still be treated as duplicates. See Scenario 13.

Page UX, accessibility and mobile experience

Google primarily uses the mobile version of a page for indexing. It tells you to keep the same primary content, headings, metadata and structured data across mobile and desktop, to allow crawling of mobile resources, and not to make primary content depend on user interaction. Tabs and accordions are fine when the content is equivalent.

• Content parity and order. Important content should not vanish on small screens, and should not be pushed far down by repeated elements.
• Layout stability. Images, embeds and injected content that shift the layout hurt users and CLS.
• Navigation and controls. Menus and controls should be accessible links and buttons, usable by keyboard and assistive technology.
• Intrusive overlays. Overlays that cover the main content block users, and may hide content from view on mobile.
• Interaction responsiveness. Slow interaction response is what INP measures.
• Responsive templates. Test templates on real small screens, since a template can render incorrectly while desktop looks fine.

Accessibility, usability, technical processing and ranking are four different things. Not every accessibility issue is a ranking factor. They overlap often: descriptive links, a logical outline, real buttons and links, and stable layouts help people, assistive technology and crawlers alike. Fix accessibility because users need it, and treat any search benefit as a possible side effect.

Server performance and robots.txt accessibility

Crawling depends on a server that answers reliably. Google says 5xx and 429 responses lower the crawl capacity and make it crawl less, persistent 5xx errors can lead to pages being dropped, and it increases crawling again once the server recovers. A slow response is not an automatic ranking penalty. Establish what is happening before acting.

• Frequency. One incident or a recurring pattern?
• Scope. All URLs, one template, one directory, or only some requests, for instance requests from certain agents or IP ranges?
• Severity. Timeouts, slow but successful responses, or hard 5xx errors?
• Impact. Do Crawl Stats, logs or the Page indexing report show reduced crawling or recrawl delays?

Check infrastructure that sits in front of the site, such as a CDN or firewall: aggressive rate limiting or bot rules can return errors to crawlers while your own browser works. Check robots.txt directly rather than inferring from a third-party warning. A 200 with the intended content is the normal state. A 404 is read as no restrictions. A 5xx or timeout can pause crawling and rely on the last cached copy for up to 30 days. Distinguish a temporary incident, which should recover, from a persistent configuration problem, which will not. Scenario 15 shows the method.

Core Web Vitals and CLS

Core Web Vitals are three metrics of real-user experience. According to web.dev (page last updated 31 October 2024), they are Largest Contentful Paint (LCP) for loading, Interaction to Next Paint (INP) for responsiveness, and Cumulative Layout Shift (CLS) for visual stability. A page passes when it meets the good level for all three at the 75th percentile of page loads, segmented by mobile and desktop. First Input Delay is not a current Core Web Vital: INP replaced it.

Lab data versus field data

Field data comes from real users, such as the Chrome UX Report, PageSpeed Insights field results, Search Console and the web-vitals library. Lab data comes from controlled tests such as Lighthouse and DevTools. Lab tools only capture shifts during load and, for INP, depend on the interactions you perform, so a good lab score does not prove users have good Core Web Vitals. Use field data to learn whether there is a problem, and lab tools to find and test causes.

What CLS measures

CLS measures the largest burst of unexpected layout shifts during a page’s life. Shifts are grouped in session windows when they occur less than one second apart, with a five-second maximum per window. Shifts that follow a click, tap or key press, within 500 ms, are excluded as expected. Scrolling and zooming do not count as input.

<img src="hero.webp" width="1200" height="630" alt="Product overview">

.ad-slot { min-height: 250px; }
.card-img { aspect-ratio: 4 / 3; width: 100%; height: auto; }

How to find, reproduce, prioritise and verify

1. Confirm the field problem. Check the Core Web Vitals report and PageSpeed Insights field data, split by mobile and desktop, and note which URL groups fail.
2. Reproduce. Test on a throttled mobile profile, scroll, and try different consent and ad states, since many shifts happen after load or in conditions a single lab run misses.
3. Identify the element. Use the DevTools Performance panel layout shift entries, or attribution from the web-vitals library.
4. Prioritise. Start with templates that carry the most important pages and the largest shifts.
5. Fix and verify. Re-run lab tests, then watch field data, which updates gradually as real visits accumulate.

Pay equal attention to LCP and INP. For LCP, find the main content element and what delays it: server response, render-blocking resources or a large image. For INP, look at long tasks and event handlers in interactions. DevTools, Lighthouse and PageSpeed Insights help diagnose, and field data confirms. See Scenario 10.

Technical SEO Audit Warnings: What Matters, What Does Not, and How to Decide

Crawler output is diagnostic evidence, not a verdict. The table evaluates ten common findings individually. “Not automatically” means the finding needs the stated evidence before anyone writes a ticket.

A. Pages with blocked resources

A crawler may flag scripts, styles or images blocked by robots.txt. Many are not needed to understand the page. Compare the rendered output with and without the resource: does content, a link, the layout or a function change? Prioritise from that evidence, not the warning. A blocked bundle that builds the product list matters. A blocked tracking script does not.

B. Pages containing JavaScript

JavaScript presence alone proves nothing. Compare source HTML with the rendered DOM for a sample per template, check that content and links are present, and compare URL groups rather than assuming every script-using URL carries the same risk.

C. Multiple H1 or H2 elements

Multiple headings are not automatically an indexing or ranking failure. Look at logical structure, meaningful headings and accessibility. Do not prescribe a single H1 as a universal technical requirement unless the page context justifies it, for example when no heading clearly states the main title.

D. Long or short page titles

Length alone says nothing. Google states no title length limit, and display varies by device. Check clarity, relevance, duplication, page purpose and how the title appears in results. Rewriting by Google varies and is not proof of a defect.

E. Long meta descriptions

Exceeding a common character guideline is not a serious problem. Google may truncate it or choose another snippet from the page content. Revise the description when it is vague, irrelevant or the displayed snippet suppresses clicks on important pages.

F. Long URLs

A long URL is not an inherent failure. Look instead for unnecessary parameters, inconsistent variants, unlimited dynamic combinations, poor link paths and duplication, which are the real conditions underneath.

G. Blocked external resources

Third-party resources are usually lower priority than your own required resources. The exception is a third-party script that supplies content or needed functionality, for example a hosted widget that renders reviews you want processed. Check what the resource contributes instead of assuming it never matters.

H. Crawl warnings generally

Validate each warning against actual implementation, search visibility, page purpose and intended URL behaviour, then classify it.

Severity depends on how many URLs are affected, how important they are, which processing stage is involved and whether there is evidence of impact. The raw number of warnings is the weakest signal.

I. Low or high word count

Word count does not establish technical quality. Google’s SEO Starter Guide lists word count targets among myths. Ask whether the page serves its purpose, is useful, has a reason to be indexed separately, and differs from similar URLs.

J. Redirects and non-200 responses

Redirects, 404s and 410s are often correct. Check that the response matches the URL’s purpose, that a redirect lands on a relevant destination, and that internal links go straight to the right URL. This avoids blanket remediation and unnecessary development work.

Fifteen diagnostic scenarios

Every scenario below is a hypothetical example with invented website details, and none is real client work or an experiment I ran. Each follows the same path: symptom, principle, evidence and tools, hypothesis, validation, action, monitoring, and why the obvious fix may be wrong.

Technical SEO tools and what each can prove

Server logs, where available, are the only source that shows actual crawler requests, though user agent strings can be spoofed, so verify crawlers as described earlier. Search Console’s Crawl Stats is for root-level properties and its counts may differ slightly from your logs.

A step-by-step technical SEO audit workflow

A repeatable audit moves from the foundation upward, because an access problem makes later checks misleading.

Priorities differ by site type. A small brochure site rarely faces crawl budget issues and should focus on clean URLs, internal links, indexing and basic performance. An ecommerce store needs control of facets, parameters, pagination and variants. A large publisher deals with archives, freshness and crawl efficiency. A JavaScript-heavy application starts with rendering, link markup and routing.

How to prioritise technical SEO fixes

Rank findings by combining these factors: impact on important pages, number of URLs affected, severity, confidence in the diagnosis, the processing stage affected, business importance, implementation effort, and risk of unintended consequences. The matrix is a planning aid that reflects judgement, not a validated score.

The number of warnings matters far less than the importance of the conditions found. A short list of confirmed, high-impact defects beats a long list of unexamined findings.

Technical SEO checklist

Each item can be checked and verified. Must validate items apply to nearly every site, Context items depend on your site and evidence, and Not always needed items are unnecessary for some sites.

Final takeaway

Evidence-led technical SEO asks one question of every finding: which stage does it affect, and what shows an impact? Discoverability, crawlability, rendering, interpretation, indexing and serving give you a way to answer it, and to leave harmless warnings alone.

No fix guarantees indexing, rankings, AI citations or rich results. A sound technical foundation removes avoidable barriers so that good content can be found, processed and shown appropriately.

Frequently asked questions

Sources and further reading

Checked on 20 October 2026. Official documentation is labelled by publisher. Pages I could not load, such as the Google pagination and lazy loading guides, are not cited, so the advice on infinite scroll and load-more patterns is practitioner advice and not a quoted Google rule.

• GoogleIntroduction to robots.txt, last updated 10 December 2025.
• GoogleHow Google interprets the robots.txt specification, last updated 31 August 2026.
• GoogleBlock Search indexing with noindex, last updated 10 December 2025.
• GoogleHow to specify a canonical URL, last updated 10 July 2026.
• GoogleJavaScript SEO basics, last updated 4 March 2026.
• GoogleLocalized versions of your pages (hreflang), last updated 21 September 2026.
• GoogleManaging crawl budget for large sites, last updated 22 July 2026.
• GoogleManaging crawling of faceted navigation URLs, last updated 18 December 2025.
• GoogleHTTP status codes, network and DNS errors and Redirects and Google Search, last updated 14 April 2026.
• GoogleURL structure best practices and Sitemaps overview.
• GoogleMobile-first indexing best practices, last updated 10 December 2025.
• GoogleInfluencing title links, last updated 10 December 2025, and Control snippets, last updated 20 April 2026.
• GoogleVerifying Googlebot and other Google crawlers, last updated 20 March 2026.
• GoogleGeneral structured data guidelines, last updated 10 July 2026.
• GooglePage indexing report, URL Inspection tool and Crawl Stats report, Search Console Help.
• GoogleSEO Starter Guide and Link best practices, last updated 10 December 2025.
• web.devWeb Vitals, last updated 31 October 2024, Cumulative Layout Shift, last updated 12 April 2023, and Interaction to Next Paint, last updated 2 September 2025.