SEO & Growth No. 444Editor's pick

Technical SEO skill for Claude Code

Every page you want indexed, provably indexable. Turns Claude into a methodical technical SEO auditor. It maps every URL template to an intended index state, probes live responses with a bundled script, reads Search Console indexing reasons correctly, and fixes conflicting signals in impact order, from accidental noindex to redirect chains. It also sets AI-crawler rules deliberately instead of blocking bots by accident.

nametechnical-seo
size222 lines · ≈2,850 tokens
files2
licenseMIT
updatedOct 20, 2026

What Claude sees before loading it

This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.

description: Audits and fixes crawlability, indexation, canonicalization, robots.txt, XML sitemaps, redirects, hreflang, JavaScript rendering, and AI-crawler access, then hands Core Web Vitals work off with field data attached. Use when pages are missing from Google, Search Console shows "Crawled - currently not indexed" or duplicate-canonical errors, during a site migration or domain move, when adding a sitemap or robots.txt, or when an SPA is not getting indexed.

Try asking

Half our product pages show 'Crawled - currently not indexed' in Search Console. Figure out why
We're moving from shop.example.com to example.com next month. Plan the SEO side of the migration
Our React SPA blog isn't showing up in Google at all. Audit what Googlebot actually sees

Use it when

Pages are missing from Google or Search Console reports indexing errors
Planning or recovering from a site migration, redesign, or domain move
Adding or rewriting robots.txt, sitemaps, canonicals, or hreflang
A JavaScript-rendered site is not getting indexed

Skip it when

The pages are indexed but slow — use core-web-vitals instead
You need rich results markup — use schema-markup instead

How to check the result

The bundled seo_probe.py exits 0 for every URL class, and URL Inspection's live test shows the rendered content, canonical, and index status you intended.

The common mistake it prevents

Disallowing a URL in robots.txt to remove it from Google hides the noindex tag from the crawler, so the URL often stays indexed with no snippet.

~/.claude/skills/technical-seo/SKILL.md

--- name: technical-seo description: Audits and fixes crawlability, indexation, canonicalization, robots.txt, XML sitemaps, redirects, hreflang, JavaScript rendering, and… license: MIT ---

Done well means every URL you want ranked returns 200, is crawlable, renders its primary content in HTML Googlebot can see, declares a self-consistent canonical, and appears in a sitemap — and every URL you do not want indexed is excluded by exactly one deliberate, working mechanism. Prove each claim with a fetch, not with a reading of the config.

Ground rules

Diagnose from evidence: Search Console (Page indexing, URL Inspection, Crawl stats), server logs, and live HTTP responses. Never “fix” from a checklist alone.
robots.txt controls crawling, not indexing. A disallowed URL can still be indexed from links; a noindex on a disallowed URL is never seen. Pick one per URL class.
Canonicals, sitemaps, internal links, hreflang, and redirects must all point at the same preferred URL. Mixed signals make Google choose for you.
Treat any change to robots.txt, sitewide canonicals, redirects, or noindex templates as a production-risk change: show the diff and get explicit user confirmation before deploy.
Rankings are not the goal metric for technical work. Indexed-valid URL count, crawl of important templates, and zero accidental exclusions are.

Workflow

- [ ] 1. Inventory URL classes and intended index status
- [ ] 2. Probe live responses for each class
- [ ] 3. Read Search Console indexing data
- [ ] 4. Check rendering (JS) for each template
- [ ] 5. Fix signals in priority order
- [ ] 6. Validate fixes and request recrawl
- [ ] 7. Hand off Core Web Vitals with field data
- [ ] 8. Report

1. Inventory URL classes

List templates, not pages: home, category, product/article, search results, filters and sort params, pagination, tag archives, user profiles, cart/account, staging hosts, PDFs, media attachment pages. For each, write the intended state: index, noindex, canonical → X, blocked, or 410. This table is the spec every later step checks against.

Find the stack’s SEO surface before editing:

grep -rniE "robots|canonical|noindex|sitemap|hreflang|x-robots-tag" \
  --include=*.{ts,tsx,js,jsx,php,py,rb,html,toml,yml,json} . | grep -v node_modules | head -50

2. Probe live responses

Run the bundled probe on one representative URL per class:

python3 scripts/seo_probe.py https://example.com/ https://example.com/blog/post \
  "https://example.com/shop?sort=price"

It prints the redirect chain, final status, X-Robots-Tag, meta robots, canonical, hreflang count, title, and whether robots.txt allows Googlebot. It exits non-zero when a URL has a chain longer than one hop, a non-200 final status, a cross-host canonical, or a noindex/robots conflict. (On macOS python.org builds, CERTIFICATE_VERIFY_FAILED means the bundled “Install Certificates” step was never run.) Equivalent manual checks:

curl -sIL https://example.com/page | grep -iE "^(HTTP|location|x-robots-tag|link)"
curl -s https://example.com/page | grep -ioE '<link[^>]+rel="?canonical"?[^>]*>|<meta[^>]+name="?robots"?[^>]*>'
curl -s https://example.com/robots.txt

3. Read Search Console

Page indexing report reasons and what they usually mean:

Reason	Usual cause	Action
Discovered – currently not indexed	Crawl budget or low perceived value; often a new site or huge faceted space	Cut low-value URLs, strengthen internal links to the important ones
Crawled – currently not indexed	Quality/duplication judgment after fetching	Improve or consolidate the content; do not just resubmit
Duplicate, Google chose different canonical than user	Conflicting signals (links, sitemap, redirects disagree)	Align all signals to one URL
Alternate page with proper canonical tag	Working as intended	None, unless the canonical is wrong
Excluded by ‘noindex’ tag	Intended or a template leak	Diff against the step-1 spec
Blocked by robots.txt / Indexed, though blocked	Disallow used where noindex was intended	Allow crawl, add noindex, remove later if desired
Soft 404	Thin/empty page returns 200 (empty search, out-of-stock shell, SPA error)	Return 404/410 or add real content
Redirect error	Loops, chains > 10 hops, or redirect to an oversized URL	Collapse to a single hop
Server error (5xx)	Origin or WAF throttling Googlebot	Check logs and bot-protection rules

Use URL Inspection’s “Test live URL” → “View tested page” to see rendered HTML and screenshot as Googlebot sees it. Crawl stats show response codes and host status over time.

4. Check JavaScript rendering

Google renders with an evergreen Chromium, but rendering is deferred and fragile.

Primary content, title, meta robots, canonical, and internal links must be in the server HTML (SSR, SSG, or prerender). Compare curl output to the rendered DOM.
Links must be <a href="/path">. onClick navigation and href="#" are not crawled.
Do not inject noindex in server HTML and remove it with JS; Google may skip rendering once it sees noindex.
SPA routes that do not exist must return a real 404 status from the server, not a 200 with “Not found” text (soft 404).
Fragment URLs (/#/products) are not indexed as separate pages.
Dynamic rendering (serving bots a different version) is a deprecated workaround; prefer SSR.
Lazy-loaded content must load without scroll or interaction events; use IntersectionObserver or native loading="lazy".

5. Fix signals in priority order

Accidental blocking — sitewide noindex, Disallow: / left from staging, auth walls, WAF challenges served to Googlebot. Highest impact, fix first.
Status codes — real 404/410 for gone pages, 301/308 for moved, no 302 for permanent moves, no 200 on error pages.
Canonicals — absolute URL, self-referencing on indexable pages, one per page, in <head> (or Link: header for PDFs), pointing at a 200 indexable URL. Never canonical all paginated pages to page 1; each page canonicals to itself.
Parameters and facets — decide per parameter: canonical to clean URL (sort, tracking), noindex,follow (low-value filter combos), or indexable (filters with search demand, given unique titles and content). Google Search Console’s URL Parameters tool is gone; control this on-site.
Sitemaps — only canonical, indexable, 200 URLs. Max 50,000 URLs or 50 MB uncompressed per file; use a sitemap index beyond that. Accurate <lastmod> only (Google uses it when it is reliably correct); Google ignores <changefreq> and <priority>. Reference it in robots.txt with Sitemap: https://… and submit in Search Console. The sitemap ping endpoint is retired; do not build on it.
Redirects — single hop, preserve path mapping, keep for at least a year; update internal links to the final URL instead of relying on redirects.
hreflang — every variant lists all variants including itself, return links are reciprocal, values are valid ISO codes (en-GB, not en-UK), x-default for the selector or fallback, and each hreflang URL is the canonical of its page.
Internal linking — important pages within 3 clicks of home; no orphan pages (in sitemap but no internal links).

robots.txt baseline:

User-agent: *
Disallow: /cart
Disallow: /account
Disallow: /search
Disallow: /*?*sessionid=

Sitemap: https://example.com/sitemap.xml

Google ignores crawl-delay, and processes only the first 500 KiB of robots.txt.

AI crawlers and answer engines

AI Overviews and AI Mode use the normal Googlebot index and snippet controls; there is no separate opt-in and no special markup that gets you cited. Controls that exist:

Goal	Mechanism
Keep content out of snippets and AI Overviews	`nosnippet`, `max-snippet:N`, or `data-nosnippet` on elements
Stay in Search but out of Gemini training/grounding	`User-agent: Google-Extended` + `Disallow: /`
Control OpenAI	`GPTBot` (training) separately from `OAI-SearchBot` (ChatGPT search)
Control others	Check each vendor’s documented user-agent tokens (e.g. `PerplexityBot`, `ClaudeBot`) and decide training vs. search access separately

Blocking a search-retrieval bot removes you from that engine’s answers; blocking a training bot does not. Confirm the user’s intent per bot. llms.txt is an unadopted proposal; Google does not use it — do not present it as a ranking or citation lever.

6. Validate and recrawl

Re-run scripts/seo_probe.py on every changed class; it must exit 0.
URL Inspection → “Test live URL” on one URL per fixed template, then “Request indexing” for a handful of key URLs (it is rate-limited; do not script it at scale).
For a migration: crawl the old URL list and assert each 301s in one hop to the mapped new URL; submit the new sitemap; keep the old sitemap briefly so Google recrawls redirects; use the Change of Address tool for domain moves.
Watch Page indexing and Crawl stats for 2–4 weeks; indexing changes are not instant.

7. Hand off Core Web Vitals

Page experience is a tie-breaker, not a substitute for relevance. Record field data at p75 from the CrUX report or Search Console’s Core Web Vitals report — LCP ≤ 2.5 s, INP ≤ 200 ms, CLS ≤ 0.1 — and list failing URL groups by template. Pass that to the core-web-vitals skill rather than optimizing here.

Migration checklist

- [ ] Full crawl of old site saved (URLs, titles, canonicals, status)
- [ ] Top URLs by clicks/links exported from Search Console and backlink tool
- [ ] 1:1 redirect map (old → new), no mass redirects to home
- [ ] Staging blocked by auth, not by robots.txt or noindex that could ship
- [ ] Launch: remove staging blocks, deploy redirects, new sitemap, verify new property
- [ ] Post-launch crawl: zero chains, zero 404s for mapped URLs, canonicals on new host
- [ ] Monitor indexing, impressions, and 404s daily for two weeks

Gotchas

noindex in robots.txt is not supported by Google. Use the meta tag or X-Robots-Tag.
Trailing-slash and case variants are different URLs. Pick one and 301 the other.
http→https and www→apex should be one combined hop, not two.
Canonical pointing at a URL that redirects, 404s, or is noindex is ignored.
CDNs and WAFs often challenge Googlebot; verify real Googlebot by reverse DNS (googlebot.com / google.com) before allow-listing, never by user-agent alone.
Geo-redirecting by IP hides content from Googlebot (crawls mostly from the US). Show a banner instead of forcing redirects.
Infinite spaces (calendars, faceted combinations, session IDs) burn crawl budget; cap them with robots.txt disallow patterns and stop linking to them.
Next.js and similar frameworks: confirm generateMetadata/head output is in the initial HTML response, and that robots.ts/sitemap.ts emit production hostnames.

Report template

## Technical SEO findings — example.com (YYYY-MM-DD)

| # | Severity | Issue | URLs affected | Evidence | Fix | Owner |
|---|---|---|---|---|---|---|
| 1 | Critical | Product template emits noindex | ~4,200 | probe output, GSC "Excluded by noindex" | Remove env-flag default in layout.tsx | |

**Index status vs. intent:** <table from step 1 with pass/fail>
**Core Web Vitals hand-off:** <failing groups with p75 values>
**Validation plan:** <what gets re-probed, what is watched in GSC, for how long>

Bundled files

Loaded only when a step in the skill needs them. Review them here before installing.

scripts/seo_probe.py 152 lines · 5.1 KB

#!/usr/bin/env python3
"""Probe URLs for core technical-SEO signals (stdlib only).

Usage: python3 seo_probe.py URL [URL ...]

For each URL prints: redirect chain, final status, X-Robots-Tag, meta robots,
canonical, hreflang count, <title>, and whether robots.txt allows Googlebot.
Exits 1 if any URL has a problem (chain > 1 hop, non-200 final, noindex while
blocked by robots.txt, canonical on another host, missing canonical).
"""
from __future__ import annotations

import sys
import urllib.error
import urllib.request
import urllib.robotparser
from html.parser import HTMLParser
from urllib.parse import urljoin, urlparse

UA = "Mozilla/5.0 (compatible; seo-probe/1.0)"
MAX_HOPS = 10
TIMEOUT = 15


class NoRedirect(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):  # noqa: D401
        return None


OPENER = urllib.request.build_opener(NoRedirect)


class HeadParser(HTMLParser):
    def __init__(self) -> None:
        super().__init__()
        self.canonicals: list[str] = []
        self.robots: list[str] = []
        self.hreflang = 0
        self.title = ""
        self._in_title = False

    def handle_starttag(self, tag, attrs):
        a = {k.lower(): (v or "") for k, v in attrs}
        if tag == "link":
            rel = a.get("rel", "").lower().split()
            if "canonical" in rel:
                self.canonicals.append(a.get("href", ""))
            if "alternate" in rel and a.get("hreflang"):
                self.hreflang += 1
        elif tag == "meta" and a.get("name", "").lower() in ("robots", "googlebot"):
            self.robots.append(a.get("content", "").lower())
        elif tag == "title":
            self._in_title = True

    def handle_endtag(self, tag):
        if tag == "title":
            self._in_title = False

    def handle_data(self, data):
        if self._in_title:
            self.title += data


def fetch(url: str):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    try:
        resp = OPENER.open(req, timeout=TIMEOUT)
        return resp.status, resp.headers, resp.read(2_000_000)
    except urllib.error.HTTPError as e:
        return e.code, e.headers, e.read(2_000_000) if e.fp else b""


def robots_allows(url: str) -> bool | None:
    p = urlparse(url)
    rp = urllib.robotparser.RobotFileParser(f"{p.scheme}://{p.netloc}/robots.txt")
    try:
        rp.read()
    except Exception:
        return None
    return rp.can_fetch("Googlebot", url)


def probe(url: str) -> list[str]:
    problems: list[str] = []
    chain = []
    current = url
    status, headers, body = 0, {}, b""
    for _ in range(MAX_HOPS + 1):
        status, headers, body = fetch(current)
        chain.append(f"{status} {current}")
        if status in (301, 302, 303, 307, 308) and headers.get("Location"):
            current = urljoin(current, headers["Location"])
            continue
        break
    hops = len(chain) - 1
    print(f"\n== {url}")
    for step in chain:
        print(f"   {step}")
    if hops > 1:
        problems.append(f"redirect chain of {hops} hops (collapse to 1)")
    if any(c.startswith(("302", "307")) for c in chain[:-1]):
        problems.append("temporary redirect in chain (use 301/308 for permanent moves)")
    if status != 200:
        problems.append(f"final status {status}")

    xrt = headers.get("X-Robots-Tag", "") if headers else ""
    parser = HeadParser()
    ctype = headers.get("Content-Type", "") if headers else ""
    if "html" in ctype:
        parser.feed(body.decode("utf-8", errors="replace"))
    robots_meta = ", ".join(parser.robots)
    noindex = "noindex" in xrt.lower() or "noindex" in robots_meta
    allowed = robots_allows(current)

    print(f"   title:        {parser.title.strip()[:90]!r}")
    print(f"   X-Robots-Tag: {xrt or '-'}")
    print(f"   meta robots:  {robots_meta or '-'}")
    print(f"   canonical:    {', '.join(parser.canonicals) or '-'}")
    print(f"   hreflang:     {parser.hreflang} alternates")
    print(f"   robots.txt:   {'allowed' if allowed else 'BLOCKED' if allowed is False else 'unreadable'} for Googlebot")

    if len(parser.canonicals) > 1:
        problems.append("multiple canonical tags")
    if "html" in ctype and status == 200 and not parser.canonicals and not noindex:
        problems.append("no canonical on indexable page")
    for c in parser.canonicals:
        if urlparse(urljoin(current, c)).netloc != urlparse(current).netloc:
            problems.append(f"canonical points to another host: {c}")
    if noindex and allowed is False:
        problems.append("noindex but blocked by robots.txt (Google cannot see the noindex)")
    for p in problems:
        print(f"   PROBLEM: {p}")
    return problems


def main(argv: list[str]) -> int:
    if not argv:
        print(__doc__)
        return 2
    total = 0
    for url in argv:
        try:
            total += len(probe(url))
        except Exception as exc:  # network errors, bad URLs
            print(f"\n== {url}\n   PROBLEM: fetch failed: {exc}")
            total += 1
    print(f"\n{total} problem(s) found across {len(argv)} URL(s)")
    return 1 if total else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

About this skill

Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published Oct 20, 2026, last updated Oct 20, 2026. MIT licensed.

Install

no restart needed

curl -fsSL https://claudeskills.page/install.sh | sh -s -- technical-seo

Installs to ~/.claude/skills/ — available in every project.

curl -fsSL https://claudeskills.page/install.sh | sh -s -- --project technical-seo

Installs to ./.claude/skills/ — commit it to share with your team.

curl -fsSLO https://claudeskills.page/downloads/technical-seo.zip
unzip -o technical-seo.zip -d ~/.claude/skills/

Download and unzip yourself. Swap the path for .claude/skills/ to scope it to a repo.

& ([scriptblock]::Create((irm https://claudeskills.page/install.ps1))) technical-seo

PowerShell 5.1+. Installs to %USERPROFILE%\.claude\skills\.

.zip

On Claude.ai, upload the zip under Settings → Skills. All methods

Archive contents

technical-seo/
SKILL.md
scripts/seo_probe.py

What Claude sees before loading it

Try asking

Use it when

Skip it when

How to check the result

The common mistake it prevents

~/.claude/skills/technical-seo/SKILL.md

Ground rules

Workflow

1. Inventory URL classes

2. Probe live responses

3. Read Search Console

4. Check JavaScript rendering

5. Fix signals in priority order

AI crawlers and answer engines

6. Validate and recrawl

7. Hand off Core Web Vitals

Migration checklist

Gotchas

Report template

Bundled files

About this skill

Related skills

Schema Markup

Core Web Vitals

Programmatic SEO

Content Refresh

International SEO

Keyword Research