Python Profiling skill for Claude Code

Done well means: a timed, repeatable reproduction exists, a profile names the functions (or lines) that dominate, the fix targets them, and the same benchmark shows the speedup with identical output.

Ground rules

• Measure before optimizing. Intuition about Python hot spots is wrong often enough (I/O, imports, logging, and ORMs surprise people) to never skip the profile.
• Pin a reproducible benchmark: fixed input, warm vs. cold stated, run 3–5 times, report median wall time. Use time.perf_counter() or hyperfine for whole commands.
• Check the output is identical before and after (hash or diff it). Fast and wrong is a regression.
• Algorithmic and data-structure fixes beat micro-optimizations by orders of magnitude. Do those first.
• Do not attach profilers (py-spy needs ptrace privileges) to production processes without the owner’s approval.

Choose the profiler

Situation	Tool	Command
Script or test, find hot functions	cProfile	python -m cProfile -o out.prof script.py
Long-running process, cannot restart	py-spy	py-spy top --pid PID / py-spy record -o prof.svg --pid PID
Readable call tree, async code	pyinstrument	pyinstrument script.py (--async_mode=enabled by default for asyncio)
Which line inside a known hot function	line_profiler	decorate with @profile, run kernprof -l -v script.py
CPU vs. native vs. memory, per line	Scalene	scalene script.py
Memory growth / peak allocations	memray	memray run script.py then memray flamegraph memray-*.bin
Quick memory diff in code	tracemalloc	snapshots before/after, compare_to
Microbenchmark of a snippet	timeit	python -m timeit -s "setup" "stmt"
Where is a hung process stuck?	py-spy dump	py-spy dump --pid PID

Workflow

- [ ] 1. Reproduce with a fixed input; record median wall time and output hash
- [ ] 2. Function-level profile (cProfile or py-spy) — find the top 3 by cumulative and self time
- [ ] 3. Classify: CPU in Python, CPU in native code, I/O wait, memory/GC, import time
- [ ] 4. Drill into the hot function (line_profiler or Scalene) if needed
- [ ] 5. Fix: algorithm/data structure > vectorize > cache > parallelize > compile
- [ ] 6. Re-run the benchmark; confirm output hash unchanged

2. Read a cProfile

python -m cProfile -o out.prof -m mypkg.job --input sample.csv
python - <<'EOF'
import pstats
s = pstats.Stats("out.prof").strip_dirs()
s.sort_stats("cumulative").print_stats(15)   # where time is spent including callees
s.sort_stats("tottime").print_stats(15)      # where time is spent in the function itself
s.print_callers("expensive_function")        # who calls it, how often
EOF

• ncalls far higher than expected (a function called 2 million times) usually means an algorithmic problem — a loop inside a loop or a query per row.
• High tottime in built-ins like {method 'append'} or list.index points at data structure choice.
• Visualize with snakeviz out.prof or convert py-spy output to speedscope (py-spy record -f speedscope -o prof.json -- python script.py).

py-spy on a live service (add --native to include C extension frames, --idle to include waiting threads, --subprocesses for workers):

py-spy record -o api.svg --pid "$(pgrep -f 'gunicorn.*app:app' | head -1)" --duration 30

3. Classify

Profile shows	Meaning	Direction
Time in your Python functions	Interpreter-bound	Algorithm, vectorize, cache, compile
Time in NumPy/pandas/C extensions	Native-bound	Fewer, larger operations; better dtypes; avoid copies
Time in socket, ssl, select, DB driver	I/O wait	Batch, concurrency, fewer round trips
Time in <frozen importlib>	Import cost	python -X importtime -c "import mypkg" and lazy imports
Time in gc / memory climbing	Allocation pressure	Generators, __slots__, smaller dtypes, memray

5. Fixes in order of payoff

Data structures and algorithms

Slow pattern	Fix
x in some_list inside a loop	Build a set / dict once
Nested loops joining two lists	Index one side in a dict: O(n·m) to O(n+m)
list.pop(0) / insert(0, x)	collections.deque
String building with += in a loop	"".join(parts) or io.StringIO
Repeated sorting to get top-k	heapq.nlargest(k, items, key=...)
Recomputing pure results	functools.cache / lru_cache(maxsize=...)
Reading a huge file into memory	Iterate lines / chunks; csv reader streaming
Regex compiled per call in a loop	Compile once at module level

Vectorize numeric and tabular work

# Slow: Python-level loop over rows
df["total"] = df.apply(lambda r: r["qty"] * r["price"] * (1 - r["discount"]), axis=1)

# Fast: column operations run in C
df["total"] = df["qty"] * df["price"] * (1 - df["discount"])

# Conditional logic without apply
import numpy as np
df["tier"] = np.select([df["total"] > 1000, df["total"] > 100], ["gold", "silver"], default="bronze")

• Never iterrows() for computation; use vectorized ops, merge, groupby().agg.
• Use categorical dtypes for repeated strings, int32/float32 where precision allows, and pyarrow-backed dtypes in pandas 2.x.
• For large or repeated pipelines, Polars lazy queries (pl.scan_csv(...).filter(...).collect()) are often 5–20x faster than pandas and use all cores.

Parallelize the right way

Workload	Use
CPU-bound pure Python	concurrent.futures.ProcessPoolExecutor (pickling cost: send chunks, not items)
Blocking I/O (HTTP, files), moderate fan-out	ThreadPoolExecutor
Many concurrent network calls	asyncio with a bounded semaphore and an async client (httpx, aiohttp)
NumPy-heavy	Often already multithreaded in BLAS; check before adding processes
Free-threaded build (3.13t/3.14t)	Threads can run Python in parallel, but verify extension support and measure

Compile the remaining hot loop (after the above): Numba @njit for numeric loops over arrays, Cython, or a Rust extension via PyO3/maturin. Keep the pure-Python version as the test oracle.

Asyncio bottlenecks

• Turn on debug mode to log callbacks that block the loop longer than 100 ms: asyncio.run(main(), debug=True) or PYTHONASYNCIODEBUG=1.
• Blocking calls inside async def (requests, time.sleep, sync DB drivers, heavy JSON, CPU work) stall every task. Replace with async libraries or wrap with await asyncio.to_thread(fn, ...); CPU work goes to a process pool via loop.run_in_executor.
• Sequential await in a loop over independent calls: use asyncio.TaskGroup or gather with a Semaphore to bound concurrency.
• pyinstrument and py-spy (--idle off) show where the loop thread burns CPU; time spent awaiting I/O will not appear as CPU.

Memory

memray run -o run.bin -m mypkg.job --input big.csv
memray flamegraph run.bin          # HTML: which call stacks hold the peak
memray summary run.bin

Typical wins: stream instead of materialize (generators, chunked read_csv(chunksize=)), drop intermediate DataFrames (del + avoid chained copies), __slots__ or dataclass slots=True for millions of small objects, numeric arrays instead of lists of floats.

Gotchas

• cProfile adds overhead per function call, inflating code with many tiny calls; confirm big findings with py-spy (sampling, low overhead).
• First-run effects: imports, disk cache, JIT warm-up in Numba, connection setup. Separate cold and warm timings.
• Profiling inside Jupyter: %prun, %lprun (line_profiler extension), %timeit — but notebook state can hide costs present in the real job.
• functools.cache on methods keeps self alive (memory leak); use it on functions or cached_property.
• Multiprocessing on macOS/Windows uses spawn: guard with if __name__ == "__main__": and expect start-up cost per worker.
• Logging f-strings are evaluated even when the level is disabled; use log.debug("x=%s", x) in hot paths.

Verification

1. Same benchmark command, same input: median wall time before and after (3–5 runs).
2. Output identical: compare hashes (sha256sum out_before.csv out_after.csv) or the test suite’s golden files.
3. A second profile shows the targeted function no longer dominates.
4. For memory work, memray summary or peak RSS (/usr/bin/time -v on Linux, /usr/bin/time -l on macOS) shows the reduction.

Report template

**Job:** nightly_rollup.py on 2.1 M rows (fixed sample, warm cache)
**Before:** 412 s median; cProfile: 78% in `match_customer` (2.1 M calls, list membership)
**Fix:** customer lookup list -> dict keyed by email; vectorized discount calc (removed df.apply)
**After:** 19 s median (21x); output sha256 identical
**Not done:** multiprocessing (no longer needed); possible further 2x with Polars