Most sites don’t have a content problem. They have a plumbing problem.
I’ve spent the last four years auditing enterprise sites and local businesses, and the pattern repeats itself constantly: solid content, decent backlinks, and a plateau that never breaks. Nine times out of ten, the bottleneck sits in the technical layer—a crawl block that’s been sitting in robots.txt since 2019, a canonical tag pointing to a redirect, or a render-blocking script that’s been quietly murdering LCP on mobile.
This guide is the audit framework I actually use. No fluff, no tool worship. Just the checks that separate sites that rank from sites that don’t.
Step 1: Crawlability and Indexation—The Foundation Everything Else Rests On
Before you optimize a single page, you need to confirm Google can actually find and store it. This isn’t a formality. A single misplaced Disallow rule in robots.txt can wipe an entire category from the index, and the site owner won’t notice until organic traffic drops 40%.
Robots.txt: The First Gate
Your robots.txt file tells crawlers what they’re not allowed to access. The problem? Most sites treat it as a security tool, which it isn’t. It’s a polite request, not a firewall, and it’s remarkably easy to break.
What to check:
- Blocked CSS/JS resources. If you’re disallowing
/wp-content/or/assets/, you’re blocking Google from rendering your pages properly. This is one of the most common self-inflicted crawl wounds I see. - Disallowed directories containing money pages. Audit every
Disallowrule against your sitemap. If there’s overlap, you’ve found a problem. - Conflicts with XML sitemaps. A URL in your sitemap should never be blocked in robots.txt. This sends contradictory signals.
The fix: Use Google’s robots.txt Tester in Search Console. It shows exactly which URLs are blocked and lets you test against live rules.
XML Sitemaps: Your Indexation Roadmap
Your sitemap should contain only canonical, indexable URLs. Nothing else. Every non-canonical variant, every noindexed tag page, every paginated archive that shouldn’t rank—all of it dilutes the signal.
Red flags:
- Sitemap contains URLs that return 404, 301, or 302
- Sitemap includes parameter-based duplicates (
?utm_source=,?color=blue) - Indexed page count in GSC is 2x+ your sitemap count (index bloat)
The fix: Generate your sitemap from your canonical URL list, not your crawl. Screaming Frog’s sitemap generation tool lets you filter by indexability status before export.
Status Codes: Where Crawl Budget Goes to Die
Every 404 and 302 you leave lying around is a small tax on your crawl budget. Google will keep recrawling that broken URL, hoping it comes back, while your new product pages sit in the queue.
| Status Code | What It Means | Action |
|---|---|---|
| 404 | Page not found | Fix the link or 301 to relevant content |
| 301 | Permanent redirect | Fine if intentional; chain them to avoid |
| 302 | Temporary redirect | Change to 301 if the move is permanent |
| 5xx | Server error | Immediate priority; these kill crawl efficiency |
| 200 (noindex) | Page exists but shouldn’t rank | Confirm intent; remove from sitemap |
Redirect chains deserve special attention. A 301 → 301 → 200 chain wastes crawl budget and dilutes link equity. Use Screaming Frog’s “Redirect Chains” report to find them.
Step 2: Core Web Vitals—The Metrics That Actually Matter in 2026
Google retired First Input Delay (FID) in March 2024. If your audit checklist still references it, update your documentation. The current three metrics are LCP, INP, and CLS.
The Thresholds (Memorize These)
| Metric | Good | Needs Improvement | Poor |
|---|---|---|---|
| LCP (loading) | < 2.5s | 2.5s – 4.0s | > 4.0s |
| INP (responsiveness) | < 200ms | 200ms – 500ms | > 500ms |
| CLS (visual stability) | < 0.1 | 0.1 – 0.25 | > 0.25 |
Fixing LCP: The Three Culprits
1. Unoptimized hero images. This is the #1 cause of slow LCP on content sites. A 2MB PNG hero image will never load in 2.5 seconds on mobile.
Fix: Convert to WebP or AVIF, compress aggressively, and serve responsive sizes. If the image is above the fold, add fetchpriority="high" to tell the browser to prioritize it.
2. Render-blocking CSS and JS. Every <link rel="stylesheet"> and <script> in the <head> blocks rendering until it’s downloaded and parsed.
Fix: Inline critical CSS for above-the-fold content. Defer non-critical JavaScript with defer or async.
3. Slow server response (TTFB). If your Time to First Byte is over 600ms, no amount of front-end optimization will save your LCP.
Fix: Upgrade hosting, enable full-page caching, or put a CDN in front of your origin server.
Fixing INP: Less JavaScript, Fewer Third-Parties
INP measures how quickly your page responds to clicks and taps. The villain is almost always JavaScript—specifically, too much of it running at once.
Fix:
- Audit third-party scripts (chat widgets, analytics, ad tags). Each one competes for the main thread.
- Break up long tasks using
setTimeoutorrequestIdleCallback. - Remove scripts you’re not actively using. That old heatmap tool from 2022? Kill it.
Fixing CLS: Define Your Dimensions
CLS happens when the page visually jumps as it loads. The fix is almost always the same: reserve space for everything.
Fix:
- Add explicit
widthandheightattributes to every image. - Reserve space for ad slots and embeds with CSS
min-height. - Use
font-display: swapcarefully; preload critical fonts to avoid layout shift when they swap in.
Step 3: Duplicate Content and Canonical Issues—Consolidating the Signal
Duplicate content isn’t a penalty. It’s a dilution problem. When Google sees multiple URLs with the same content, it has to choose which one to rank. If it picks wrong, your preferred page loses.
The Usual Suspects
Common sources of duplication I see in every audit:
- URL parameters:
?utm_source=,?session_id=, sorting and filtering parameters - HTTP vs. HTTPS: Still present on sites with incomplete migrations
- www vs. non-www: Both versions resolving to the same content
- Trailing slash vs. no trailing slash:
/shoesand/shoes/are different URLs to Google - CMS-generated duplicates: WordPress tag pages, category archives, date archives
- Faceted navigation: E-commerce filtering combinations creating thousands of near-duplicates
The Canonical Tag: Your Preferred URL Declaration
The rel="canonical" tag tells Google which URL is the master version. Best practice in 2026 is to include a self-referencing canonical on every indexable page, even ones without duplicates.
html
<link rel="canonical" href="https://example.com/shoes" />
Critical rule: Always use absolute URLs, not relative ones. Relative canonicals break when content gets syndicated or proxied.
When Canonical Fails: Google Chose Differently
Search Console’s “Duplicate, Google chose different canonical” status means Google disagreed with your declaration. This usually happens when:
- Your canonical target isn’t indexed
- Your canonical target is blocked in robots.txt
- Your content signals are contradictory (internal links point to a different version)
Fix: Strengthen the signal. Point internal links to the canonical URL. Include only the canonical in your sitemap. Ensure the canonical target is itself indexable and returns a 200.
301 vs. Canonical: When to Use Which
Use a 301 redirect when the old URL should never be accessed directly. Use a canonical tag when both URLs legitimately need to remain accessible (for users or tracking).
If a URL has backlinks and should no longer exist, 301 it. Don’t canonical it and leave it live.
Step 4: Structured Data and Schema Markup—Speaking Google’s Language
Schema markup isn’t optional anymore. It’s how you tell search engines—and increasingly, AI answer engines—what your content actually is.
The Schema You Actually Need
| Page Type | Schema Type | Why It Matters |
|---|---|---|
| Blog posts | Article or BlogPosting | Rich results, author attribution |
| Product pages | Product + Offer | Price, availability, reviews in SERPs |
| Local business | LocalBusiness | NAP data, hours, location |
| FAQ sections | FAQPage | Expandable Q&A in search results |
| How-to guides | HowTo | Step-by-step rich results |
The Implementation Reality
You don’t need to hand-code JSON-LD. Most SEO plugins (Yoast, Rank Math, AIOSEO) handle the basics automatically. For custom schema, use Google’s Structured Data Markup Helper to generate the code visually.
The critical step: Validate. Every schema implementation needs to pass the Rich Results Test before you consider it done. A single typo in a property name will silently break the entire markup.
Schema and AI Overviews
Schema helps AI systems extract facts accurately, reducing the risk of incorrect information in AI-generated answers. If your site appears in AI Overviews, clean schema improves the odds that the citation is accurate.
One caution: Microsoft’s documentation explicitly states that structured data must be embedded in the server-returned HTML, not generated by JavaScript after page load. If your schema relies on client-side rendering, it may be invisible to crawlers.
The Complete Technical Audit Checklist
Run this in order. Fix critical items before moving down the list.
Crawlability & Indexation
- □ Robots.txt doesn’t block CSS/JS or important directories
- □ XML sitemap contains only canonical, indexable, 200-status URLs
- □ No 5xx errors on any crawlable URL
- □ Redirect chains resolved (301 → 301 → 200 chains eliminated)
- □
noindextags reviewed—nothing important is accidentally excluded - □ Indexed page count in GSC within 1.5x of sitemap count
Core Web Vitals
- □ LCP < 2.5s on mobile for top 10 pages
- □ INP < 200ms for interactive elements
- □ CLS < 0.1 (images have dimensions, no late-loading embeds)
- □ Hero images compressed and served in WebP/AVIF
- □ Render-blocking resources eliminated or deferred
- □ Third-party scripts audited and trimmed
Canonicalization & Duplicates
- □ Self-referencing canonicals on all indexable pages
- □ Canonical tags use absolute URLs
- □ HTTP/HTTPS and www/non-www versions consolidated
- □ Parameter-based duplicates handled (canonical or robots.txt)
- □ Internal links point to canonical versions
Structured Data
- □ Schema validates in Rich Results Test
- □ Product, Article, LocalBusiness, and FAQ schema implemented where relevant
- □ Schema is in server-returned HTML (not JS-generated)
- □ No errors in Google Search Console’s Enhancements report
Rendering & JavaScript
- □ Critical content renders without JavaScript (view page source)
- □ Server-side rendering (SSR) or dynamic rendering used for JS-heavy sites
- □ AI crawlers (which don’t execute JS) can access core content
Technical SEO & Audit Insights
Direct, actionable answers to advanced search optimization, crawl efficiency, and architecture challenges.
How often should a technical SEO audit be conducted?
For large enterprise platforms or dynamic e-commerce sites, technical audits should be run quarterly, backed by automated weekly crawl error monitoring. For small-to-mid business websites, a thorough audit twice a year—plus post-redesign checks—is typically optimal.
Why does Google choose a different canonical URL than specified?
Google frequently overrides rel="canonical" tags when conflicting technical signals exist. Common triggers include the target URL returning a non-200 status (404/301), being blocked via robots.txt, or featuring internal links that point toward an alternate version.
What is the impact of Core Web Vitals on search rankings?
Core Web Vitals (LCP, INP, and CLS) operate primarily as page experience signals. While content relevance remains paramount, meeting these thresholds serves as a vital tie-breaker between competing pages sharing comparable authority.
How do you systematically fix site index bloat?
To resolve index bloat: audit low-value programmatic pages, apply strategic noindex tags or 410 codes to thin content, consolidate parameter variations using clean canonicalization, and refresh your XML sitemap in Search Console.
Does Schema Markup directly boost organic rankings?
Schema markup is not a direct ranking weight, but it functions as a powerful entity translator. It unlocks Rich Results for higher CTR and ensures AI answer engines accurately extract facts for citations in generative search overviews.