Web Scraping skill for Claude Code

A scraper is done when it collects only data you are permitted to collect, at a rate the site would not notice, survives restarts without re-fetching everything, detects when the page layout changes, and writes validated records with their source URL and fetch time.

Ground rules

• Check permission first: robots.txt, the site’s terms, and whether an official API or data export exists. If terms forbid automated access, stop and tell the user.
• Never bypass authentication, paywalls, CAPTCHAs, or bot protection, and never scrape behind a login without the account owner’s and site’s permission.
• Personal data (names, emails, profiles) triggers privacy law (GDPR, CCPA); collect only what the user has a lawful basis for, and say so when unclear.
• Identify yourself: a descriptive User-Agent with contact info.
• Default rate: at most 1 request/second per host, concurrency 1–2 per host, unless robots.txt or the owner allows more. Honor Crawl-delay and Retry-After.
• This is general information, not legal advice; for commercial or large-scale scraping, recommend the user get legal review.

Workflow

- [ ] 1. Permission check (robots.txt script, terms, API availability)
- [ ] 2. Find the cleanest data source (API > embedded JSON > HTML > browser)
- [ ] 3. Prototype extraction on 3–5 saved pages
- [ ] 4. Build the fetcher: rate limit, retries, caching, checkpointing
- [ ] 5. Validate records; track field null rates
- [ ] 6. Run small, inspect, then scale
- [ ] 7. Monitor for breakage on scheduled runs

1. Permission check

python3 scripts/robots_check.py "AcmePriceBot/1.0 (+mailto:[email protected])" \
  https://example.com/products/123 https://example.com/search?q=shoes

The script reports allow/disallow per URL, Crawl-delay, request rate, and sitemaps, and exits 1 if anything is disallowed or robots.txt is unreachable.

2. Find the cleanest source

Check in this order (open DevTools Network tab, filter Fetch/XHR):

Source	How to spot it	Why
Official API / bulk export	Developer docs, /api/ links	Stable, permitted, structured
Sitemaps	sitemap.xml from robots.txt	Complete URL list without crawling
JSON endpoints the page calls	XHR responses in DevTools	Structured, no HTML parsing
Embedded JSON	<script type="application/ld+json">, __NEXT_DATA__, window.__INITIAL_STATE__	Full data in the first response
Server-rendered HTML	Data visible in “View Source”	Simple HTTP + parser
Client-rendered only	Data absent from View Source	Needs a browser (Playwright)

3. HTTP + parser (default)

import json, time, random
import httpx
from selectolax.parser import HTMLParser

HEADERS = {"User-Agent": "AcmePriceBot/1.0 (+mailto:[email protected])", "Accept-Language": "en"}

def fetch(client: httpx.Client, url: str, max_tries: int = 5) -> httpx.Response:
    for attempt in range(max_tries):
        r = client.get(url)
        if r.status_code in (429, 503):
            wait = float(r.headers.get("Retry-After", 2 ** attempt)) + random.random()
            time.sleep(min(wait, 120))
            continue
        r.raise_for_status()
        return r
    raise RuntimeError(f"gave up on {url} after {max_tries} tries")

def parse_product(html: str, url: str) -> dict:
    tree = HTMLParser(html)
    ld = tree.css_first('script[type="application/ld+json"]')
    if ld:                                            # structured data first
        data = json.loads(ld.text())
        if isinstance(data, dict) and data.get("@type") == "Product":
            offer = data.get("offers") or {}
            return {"url": url, "name": data.get("name"), "price": offer.get("price"),
                    "currency": offer.get("priceCurrency")}
    title = tree.css_first("h1")
    price = tree.css_first("[data-testid='price'], .price")
    return {"url": url, "name": title.text(strip=True) if title else None,
            "price": price.text(strip=True) if price else None, "currency": None}

with httpx.Client(headers=HEADERS, timeout=20, follow_redirects=True, http2=False) as client:
    for url in urls:
        rec = parse_product(fetch(client, url).text, url)
        time.sleep(1.0)                               # per-host politeness delay

Selectors: prefer stable attributes (data-testid, itemprop, semantic tags) over generated class names (.css-1x2y3z). Resolve relative links with urllib.parse.urljoin. BeautifulSoup (bs4 with lxml) is fine when speed is not a concern.

Browser (only when required)

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page(user_agent="AcmePriceBot/1.0 (+mailto:[email protected])")
    page.route("**/*.{png,jpg,jpeg,webp,gif,woff2,mp4}", lambda route: route.abort())
    captured = []
    page.on("response", lambda r: captured.append(r) if "/api/products" in r.url and r.ok else None)
    page.goto("https://example.com/category/shoes", wait_until="domcontentloaded")
    page.wait_for_selector("[data-testid='product-card']")
    rows = [r.json() for r in captured]               # prefer the JSON the page fetched
    browser.close()

Capture the page’s own API responses instead of scraping the rendered DOM when possible. Reuse one browser and context; a new browser per page is slow and memory hungry.

4. Scale safely

Concern	Approach
Politeness	Per-host delay or token bucket; asyncio.Semaphore(2) per host for async
Retries	Only on timeouts, 429, 5xx; exponential backoff with jitter; cap attempts
403 / bot challenge	Stop and reassess permission; do not rotate proxies or spoof to evade
Caching	Store raw responses (HTML/JSON) keyed by URL + date; re-parse without re-fetching
Conditional requests	Send If-None-Match / If-Modified-Since when re-crawling
Checkpointing	Record completed URLs (SQLite table or JSONL); resume skips them
Dedupe	Canonical URL (strip tracking params, fragments) as the key
Pagination	Stop on empty page, repeated page, or max page; never loop unbounded

Scrapy, for larger crawls, with politeness built in:

# settings.py
ROBOTSTXT_OBEY = True
USER_AGENT = "AcmePriceBot/1.0 (+mailto:[email protected])"
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
DOWNLOAD_DELAY = 1.0
CONCURRENT_REQUESTS_PER_DOMAIN = 2
HTTPCACHE_ENABLED = True
RETRY_HTTP_CODES = [429, 500, 502, 503, 504]

5. Validate output

Every record carries url, fetched_at (UTC), and the extracted fields. Validate types (price parses to a number, currency is ISO 4217) and track per-field null rates per run. A field going from 0% to 80% null means the layout changed: fail the run instead of writing garbage.

Gotchas

• Prices and numbers come localized (1.234,56 €); parse with the page’s locale.
• Sites serve different content by geography, language, cookie consent, or A/B test; pin Accept-Language and note it.
• wait_until="networkidle" hangs on pages with long-polling or analytics beacons; wait for a specific selector instead.
• Infinite scroll usually calls a paginated JSON endpoint; use that directly.
• Character encoding: trust response.encoding only if the header sets it; otherwise let the parser detect from <meta charset>.
• Storing full HTML of pages with personal data creates a privacy liability; store only needed fields and set retention.

Output

**Target:** example.com product pages (robots.txt allows /products/, crawl-delay 2 s)
**Source used:** JSON-LD Product blocks (fallback CSS for 3% of pages)
**Run:** 4,812 URLs from sitemap, 1 req/2 s, 0 disallowed, 14 retries (429), 0 failures
**Output:** data/products_2026-10-19.parquet (url, name, price, currency, fetched_at)
**Quality:** price null rate 0.4%; name null rate 0.0%
**Notes:** terms permit non-commercial use; no personal data collected