AI Search Optimization skill for Claude Code
Get cited in AI answers without gaming anything. Claude separates what actually drives citations in AI Overviews, ChatGPT, Perplexity, Copilot, and Claude from the folklore. It checks robots.txt and server-rendered HTML per bot with a bundled probe, rewrites target pages into self-contained, sourced answer passages, cleans up how third-party sources describe the brand, and measures progress with a fixed prompt panel and AI referral traffic.
- name
ai-search-optimization - size191 lines · ≈2,700 tokens
- files2
- licenseMIT
- updated
What Claude sees before loading it
This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.
description: Improves how often a brand and its pages are cited or recommended in AI answers, including Google AI Overviews and AI Mode, ChatGPT search, Perplexity, Copilot, Gemini, and Claude. Covers crawler access per bot, server-rendered and quotable answer passages, original data, entity consistency across third-party sources, structured data that matches visible content, honest notes on llms.txt, and measurement with prompt panels, referral traffic, and logs. Use when asked about GEO, AEO, LLM visibility, or why AI assistants recommend competitors.Try asking
- ChatGPT and Perplexity recommend three competitors when people ask for the best tool in our category, never us. Fix that
- Check whether AI crawlers can actually read our docs and pricing pages
- Set up a monthly way to measure how often AI assistants mention and cite our brand
Use it when
- AI assistants recommend competitors or describe your product incorrectly
- Deciding which AI crawlers to allow or block in robots.txt
- Rewriting key pages so AI Overviews and answer engines can quote them
- Measuring brand visibility across ChatGPT, Perplexity, Gemini, Copilot, and Claude
Skip it when
- Pages are not indexed by Google at all — use technical-seo first
- You only need JSON-LD added or validated — use schema-markup
How to check the result
ai_bot_access.py exits 0 for key URLs, logs show 200s to the allowed AI agents, and the monthly prompt panel shows mention and citation rates per engine against the baseline.
The common mistake it prevents
Blocking every AI user agent to stop model training also blocks the retrieval bots, which removes the site from those engines' answers entirely.
Done well means the retrieval bots you want can fetch your pages, the pages contain specific, self-contained, verifiable answers worth quoting, the rest of the web describes your brand consistently, and visibility is measured with a fixed prompt panel and referral data rather than anecdotes from one chat session.
Ground rules
- AI answer engines mostly retrieve from a search index, then summarize. Classic SEO (indexable, crawlable, helpful pages) is the foundation; there is no separate trick layer.
- Google’s AI Overviews and AI Mode use the regular Google index and snippet controls. No special markup or file is required or rewarded; a page must be indexed and snippet-eligible.
- Never use hidden text, instructions aimed at language models, cloaking (different content for AI bots), fabricated statistics, or fake reviews. They violate search spam policies, get pages dropped, and can count as prompt injection.
- Bot access decisions (blocking or allowing training versus retrieval crawlers) are business
decisions. Present the trade-off and get explicit user confirmation before changing
robots.txtor WAF rules. - Results are probabilistic. One answer from one session proves nothing; measure in samples.
How each engine finds sources
| Engine | Retrieval source | Crawler that matters | Opt-out without leaving search |
|---|---|---|---|
| Google AI Overviews, AI Mode | Google index | Googlebot | nosnippet, max-snippet, data-nosnippet limit use; Google-Extended covers Gemini training and grounding, not Search |
| ChatGPT search | Own index plus third-party search data | OAI-SearchBot (search), ChatGPT-User (user-triggered fetches) | Block GPTBot for training only |
| Perplexity | Own index | PerplexityBot, Perplexity-User | No separate training token documented; check current docs |
| Microsoft Copilot | Bing index | Bingbot | Bing’s snippet controls |
| Claude (web search) | Third-party search plus fetches | Claude-SearchBot, Claude-User | Block ClaudeBot for training only |
| Gemini app | Google index | Googlebot | Google-Extended |
Vendors rename and add user agents; confirm each token against the vendor’s current crawler documentation before writing rules. Blocking a retrieval or user-fetch bot removes you from that engine’s answers; blocking a training bot does not.
Workflow
- [ ] 1. Baseline: prompt panel results, AI referral traffic, bot hits in logs
- [ ] 2. Access: robots.txt, WAF, and rendering for each bot that matters
- [ ] 3. Pick the pages and questions that should earn citations
- [ ] 4. Make those pages quotable: answer-first passages, data, sources, dates
- [ ] 5. Strengthen entity signals and third-party coverage
- [ ] 6. Align structured data with visible content
- [ ] 7. Re-run the panel monthly and report deltas1. Baseline
- Prompt panel: write 30-100 prompts real buyers would ask (category questions, comparisons, “best X for Y”, problem descriptions, your brand plus “reviews” or “alternatives”). Run each in every engine that matters, at least 3 times per engine per cycle in fresh sessions, logged out where possible. Record: brand mentioned (yes/no), position in list, cited URL (yours or third-party), sentiment, and factual errors.
- Referral traffic: in GA4, build a segment for sessions whose source matches
chatgpt.com|perplexity.ai|gemini.google.com|copilot.microsoft.com|claude.ai. ChatGPT appendsutm_source=chatgpt.comto many links. Much AI-driven traffic arrives without a referrer, so treat this as a floor. - Search Console: AI Overview and AI Mode impressions and clicks are counted inside the Web search type and cannot be isolated there; watch for queries where impressions rise and CTR falls, a common AI Overview footprint.
- Logs: count requests per AI user agent per day and which URLs they fetch. User-fetch agents hitting a page means someone’s assistant read it during a conversation.
2. Access and rendering
Run the bundled probe against key URLs:
python3 scripts/ai_bot_access.py https://example.com/ https://example.com/pricing https://example.com/guide/xIt reads robots.txt once per host and reports allow or block for Googlebot, Bingbot, and the
main AI search, user-fetch, and training agents, then fetches each URL without JavaScript and
counts visible words in the raw HTML. It exits 1 when a retrieval or user-fetch bot is blocked
or a page has under 150 words of server-rendered text. Python’s robots parser does not handle
* wildcards inside paths, so verify wildcard rules by hand.
Then check:
- WAF and bot-protection products often challenge AI agents by default; review their bot settings and logs for 403s and challenge pages served to the agents you allowed.
- Most AI crawlers do not execute JavaScript. Main content, headings, prices, and specs must
be in the server HTML (hand rendering fixes to
technical-seo). - Pages behind logins, cookie walls that hide content, or interstitials will not be quoted.
3. Choose target questions and pages
Map each panel prompt to the page that should be cited. If no page directly answers it, that is the gap. Prioritize prompts with commercial intent where competitors are cited and you are not. Comparison and “best for” prompts are usually won by third-party pages (reviews, listicles, forums, video), so plan step 5 for those rather than another page on your own site.
4. Make pages quotable
Answer engines lift passages, not pages. For each target page:
- Open with a direct answer of 40-80 words under a heading phrased like the question. It must make sense if quoted alone: name the subject instead of “it” or “this tool”.
- One idea per section, with descriptive H2 and H3 headings; keep paragraphs to 2-4 sentences.
- Use specifics models can repeat: numbers with units, named versions, prices with dates, thresholds, step counts. Vague claims (“fast”, “affordable”) are rarely cited.
- Publish original information: benchmarks, survey results, pricing data, teardown findings, with methodology. Original data is the most reliable way to earn citations from others too.
- Cite primary sources with links; attribute expert quotes to named people with credentials.
- Show who wrote or reviewed it and when it was last updated, with a real update history.
- Use tables for comparisons and specs, and ordered lists for procedures; both extract cleanly.
- Cover the follow-up questions a person would ask next (AI Mode expands one query into many related sub-queries); link to deeper pages rather than bloating one page.
- Keep product facts consistent across the site: one canonical price, plan list, and feature list, updated everywhere at once. Contradictions get repeated as errors.
5. Entity signals and third-party coverage
Models describe a brand from everything written about it, weighted toward sources they trust.
- Write one canonical entity description (what you are, for whom, category, founding year, headquarters) and use it on the About page, social profiles, directory listings, app store pages, and press kit.
- Organization schema with
sameAslinks to official profiles (hand details toschema-markup). - Wikipedia and Wikidata only if the brand meets notability rules; never write your own promotional article.
- Earn mentions on the pages engines cite for your category: independent reviews, “best X” roundups, analyst reports, industry publications, podcasts, YouTube reviews, and community threads. Contribute genuinely in communities; astroturfing on Reddit or forums is detected, removed, and remembered.
- Keep review profiles (G2, Capterra, Trustpilot, app stores) current; they are frequent sources for “best for” answers.
- When panel answers contain factual errors about you, fix the source they likely came from (outdated page, old press release, stale directory listing) rather than arguing with the model.
6. Structured data
Structured data helps engines parse pages and must match what users see. Use Organization,
Product with offers, Article with author and dates, FAQPage only where a real FAQ exists
on the page (Google limits FAQ rich results to authoritative government and health sites, but
the markup still clarifies content). It is not a citation switch. Validate everything.
llms.txt
llms.txt is a proposed convention for listing key pages in Markdown for language models.
As of 2026 no major answer engine has committed to using it for ranking or citation, and
Google has said it does not use it. Adding one is cheap and harmless for documentation sites
whose users paste docs into coding assistants; never present it as a visibility lever or
prioritize it above steps 2-5.
Gotchas
- Blocking
GPTBotin a panic over training also tempts teams to blockOAI-SearchBot; the second one removes the site from ChatGPT search results. - AI referral traffic converts well but is small; report it next to organic, not instead of it.
- Rankings in Google and citations in AI Overviews correlate but are not the same; a page ranking 6th can be cited while the 1st is not, because its passage answers more directly.
- Prompt panels drift as models update. Keep prompts fixed for a quarter and version changes.
- Personalization and memory skew results; use clean sessions and note the account state.
- Mass-generating “X vs Y” and “best X” pages to feed answer engines is scaled content abuse
in Google’s spam policies. Hand legitimate template work to
programmatic-seo.
Verification
ai_bot_access.pyexits 0 for every key URL, and logs show 200 responses to the allowed AI agents.- Every target page opens with a self-contained answer, has a named author and update date, and contains at least one specific, sourced fact or original data point.
- Monthly panel re-run: report mention rate and citation rate per engine against baseline. If a target prompt shows no change after two cycles, check whether the cited sources are third-party pages and shift effort to step 5.
Report template
## AI search visibility — <brand> (YYYY-MM-DD, cycle <n>)
Panel: <n> prompts x <engines> x 3 runs. Account state: logged out, fresh sessions.
| Engine | Mention rate | Citation rate (own URLs) | Avg list position | Factual errors |
|---|---|---|---|---|
**Access:** bots allowed/blocked, WAF findings, pages failing the server-HTML check
**Pages improved this cycle:** URL, prompt targeted, change made
**Third-party coverage:** sources cited for category prompts, outreach status
**AI referral sessions:** <n> (vs <n> last cycle), conversion rate
**Next cycle:** ...Bundled files
Loaded only when a step in the skill needs them. Review them here before installing.
scripts/ai_bot_access.py
#!/usr/bin/env python3
"""Check robots.txt access for search and AI crawlers, and server-rendered text (stdlib only).
Usage:
python3 ai_bot_access.py URL [URL ...] [--min-words 150] [--timeout 15]
For each host, robots.txt is fetched once and evaluated for each user agent below.
For each URL, the raw HTML (no JavaScript) is fetched and visible words are counted.
Exit 1 when a search, retrieval, or user-fetch agent is blocked for a URL, or a page
has fewer than --min-words words of server-rendered text. Training-only agents are
reported but never fail the run (blocking them is a policy choice).
Note: urllib.robotparser does not support '*' or '$' wildcards inside paths, so
verify wildcard rules by hand. Agent tokens change; confirm against vendor docs.
"""
from __future__ import annotations
import argparse
import re
import sys
import urllib.error
import urllib.request
import urllib.robotparser
from html.parser import HTMLParser
from urllib.parse import urlsplit
AGENTS = [
# (token, role) role: search | retrieval | user-fetch | training
("Googlebot", "search"),
("Bingbot", "search"),
("OAI-SearchBot", "retrieval"),
("ChatGPT-User", "user-fetch"),
("PerplexityBot", "retrieval"),
("Perplexity-User", "user-fetch"),
("Claude-SearchBot", "retrieval"),
("Claude-User", "user-fetch"),
("GPTBot", "training"),
("ClaudeBot", "training"),
("Google-Extended", "training"),
("Applebot-Extended", "training"),
("CCBot", "training"),
]
UA = "Mozilla/5.0 (compatible; ai-bot-access-check/1.0)"
class TextCounter(HTMLParser):
def __init__(self) -> None:
super().__init__()
self.skip = 0
self.words = 0
def handle_starttag(self, tag, attrs):
if tag in ("script", "style", "noscript", "template", "svg"):
self.skip += 1
def handle_endtag(self, tag):
if tag in ("script", "style", "noscript", "template", "svg") and self.skip:
self.skip -= 1
def handle_data(self, data):
if not self.skip:
self.words += len(re.findall(r"\w+", data))
def get(url: str, timeout: float) -> tuple[int, str]:
req = urllib.request.Request(url, headers={"User-Agent": UA})
try:
with urllib.request.urlopen(req, timeout=timeout) as r:
return r.status, r.read(3_000_000).decode(r.headers.get_content_charset() or "utf-8", "replace")
except urllib.error.HTTPError as e:
return e.code, ""
def load_robots(host_url: str, timeout: float) -> tuple[urllib.robotparser.RobotFileParser, str]:
rp = urllib.robotparser.RobotFileParser()
try:
status, body = get(host_url + "/robots.txt", timeout)
except Exception as e:
rp.parse([])
return rp, f"robots.txt fetch error ({type(e).__name__}); treating as allow-all"
if status >= 500:
# Google treats persistent 5xx on robots.txt as disallow-all.
rp.parse(["User-agent: *", "Disallow: /"])
return rp, f"robots.txt returned {status}; crawlers may treat the site as blocked"
if status >= 400:
rp.parse([])
return rp, f"robots.txt returned {status}; treated as allow-all"
rp.parse(body.splitlines())
return rp, "robots.txt ok"
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("urls", nargs="+")
ap.add_argument("--min-words", type=int, default=150)
ap.add_argument("--timeout", type=float, default=15)
a = ap.parse_args()
robots: dict[str, tuple[urllib.robotparser.RobotFileParser, str]] = {}
failed = False
for url in a.urls:
p = urlsplit(url)
if p.scheme not in ("http", "https") or not p.netloc:
print(f"error: not an absolute URL: {url}", file=sys.stderr)
return 1
base = f"{p.scheme}://{p.netloc}"
if base not in robots:
robots[base] = load_robots(base, a.timeout)
print(f"\n[{base}] {robots[base][1]}")
rp = robots[base][0]
print(f"\n{url}")
for token, role in AGENTS:
ok = rp.can_fetch(token, url)
flag = ""
if not ok and role != "training":
flag = " <-- removes this page from that engine's answers"
failed = True
print(f" {token:<18} {role:<10} {'allow' if ok else 'BLOCK'}{flag}")
try:
status, html = get(url, a.timeout)
except Exception as e:
print(f" fetch error: {type(e).__name__}: {e}")
failed = True
continue
tc = TextCounter()
tc.feed(html)
thin = tc.words < a.min_words
print(f" HTTP {status}, server-rendered words: {tc.words}{' <-- thin or JS-rendered' if thin else ''}")
if status != 200 or thin:
failed = True
print("\nRESULT:", "issues found" if failed else "ok")
return 1 if failed else 0
if __name__ == "__main__":
sys.exit(main())
About this skill
Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published , last updated . MIT licensed.