Computer Vision skill for Claude Code
Detectors that hold up on your real cameras. Claude approaches vision work data-first. It picks the simplest task type that answers the question, writes labeling guidelines, splits by camera or video so frames cannot leak, validates YOLO datasets with a bundled checker, trains from pretrained weights, chooses per-class thresholds from precision-recall targets, and verifies exported ONNX or TensorRT models still match on the target hardware.
- name
computer-vision - size185 lines · ≈2,050 tokens
- files2
- licenseMIT
- updated
What Claude sees before loading it
This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.
description: Builds computer vision features with OpenCV, Ultralytics YOLO, and torchvision - choosing classification, detection, segmentation, or pose; dataset collection and labeling in CVAT or Label Studio with written guidelines; YOLO-format validation and leakage checks; training from pretrained weights; mAP, per-class recall, and threshold tuning; error analysis; and export to ONNX or TensorRT for deployment. Use when detecting or counting objects, segmenting images, inspecting defects, reading camera feeds, preparing or auditing a labeled dataset, or when a vision model scores well in validation but fails in production. Try asking
- Train a detector that counts forklifts and people in our warehouse camera footage
- Our defect model has 0.97 mAP in validation but misses defects on the line. Why?
- Audit this YOLO dataset for label errors and train/val leakage before we train
Use it when
- Detecting, counting, segmenting, or classifying objects in images or video
- Preparing, labeling, or auditing a vision dataset
- A vision model validates well but fails in production
- Exporting a model for edge or GPU deployment
Skip it when
- You need text from scanned pages — use ocr-scans
- You want to generate images rather than analyze them — use image-generation
How to check the result
Run scripts/check_yolo_dataset.py until it passes, then confirm per-class precision and recall on a test split from separate cameras or days, and re-check metrics after export.
The common mistake it prevents
Splitting video frames randomly into train and validation, so near-identical frames leak and validation mAP looks far better than production.
A vision model is ready when it meets per-class targets on a test set drawn from the deployment cameras and conditions, the test set shares no near-duplicate frames with training, the confidence threshold was chosen from a precision-recall trade-off the business signed off on, and inference meets the latency budget on the target hardware. A high mAP on a random split of video frames is usually leakage, not skill.
Ground rules
- Start from the simplest task type that answers the question: classification before detection, detection before segmentation.
- Data beats architecture. Most gains come from fixing labels and adding the hard cases.
- Split by source (camera, site, day, video), never by random frame.
- Write labeling guidelines before labeling; inconsistent boxes cap accuracy.
- Check licenses: Ultralytics YOLO code and weights are AGPL-3.0 unless you hold an enterprise license; torchvision models are BSD-style but their weights carry the dataset’s terms.
Choose the task and tool
| Question | Task | Default tool |
|---|---|---|
| Is there a defect / what kind of scene is this? | Classification | timm or torchvision backbone fine-tune; YOLO -cls |
| Where and how many objects? | Detection | Ultralytics YOLO (YOLO11 or newer, e.g. YOLO26); torchvision Faster R-CNN when AGPL is a problem |
| Exact object shape or area? | Instance / semantic segmentation | YOLO -seg; SAM-family models for label assistance |
| Body or object keypoints? | Pose | YOLO -pose |
| Measure, align, threshold, simple shapes in controlled lighting | Classical CV | OpenCV (thresholding, contours, template matching) |
| Open-vocabulary “find any X” without training | Zero-shot detection or a vision LLM | Prototype only; measure before shipping |
Classical OpenCV beats deep learning when lighting, camera, and parts are fixed (factory inspection with a jig). Try it first in those settings.
Workflow
- [ ] 1. Define classes, metrics, and the latency budget
- [ ] 2. Collect images from deployment conditions
- [ ] 3. Write labeling guidelines; label; review a sample
- [ ] 4. Split by source; validate the dataset
- [ ] 5. Train a baseline from pretrained weights
- [ ] 6. Evaluate per class; pick thresholds
- [ ] 7. Error analysis; add data; retrain
- [ ] 8. Export, benchmark on target hardware, monitor
1. Definition
- Classes: name, definition, and edge cases (partially visible? reflections? toy versions?).
- Metric per class: recall matters for safety (missed person), precision for alerts that page humans. Write targets, e.g. “person recall >= 0.95 at precision >= 0.85”.
- Latency and hardware: “20 FPS per stream on Jetson Orin” or “200 ms per image on CPU”.
2. Collect
- Capture from the real cameras, resolutions, angles, lighting (night, glare, rain), and backgrounds. Public datasets help pretraining but rarely match deployment.
- Sample video frames sparsely (e.g. one frame per second or on scene change); adjacent frames are near-duplicates.
- Include negatives: images with no objects, and look-alikes that should not trigger.
- Typical starting volume: 100-300 labeled instances per class for fine-tuning a pretrained detector; more for rare, small, or variable classes.
3. Label
- Tools: CVAT or Label Studio; use model-assisted pre-labeling once a baseline exists, with humans correcting.
- Guidelines with pictures: tight boxes, include occluded parts or not, minimum object size to label, what to do with crowds, truncated objects at the image border.
- Review 10% of labels by a second person; measure box agreement (IoU >= 0.7 counts as agreement) and fix the guideline where they disagree.
4. Split and validate
YOLO layout:
dataset/
data.yaml # path, train, val, test, names
images/{train,val,test}/*.jpg
labels/{train,val,test}/*.txt # one line per object: class cx cy w h (normalized 0-1)
Run the bundled checker before every training run:
python3 scripts/check_yolo_dataset.py dataset/
It reads class names from data.yaml (simple YAML subset, no PyYAML needed), pairs
images with labels, flags missing or orphan label files, class IDs out of range,
coordinates outside (0, 1], zero-area or tiny boxes, duplicate boxes, prints instances
per class per split, and finds identical images across splits by content hash. It exits
1 on errors. Content-hash matching only catches exact copies; for video, split by video
or camera so near-duplicates cannot cross splits.
5. Train a baseline
from ultralytics import YOLO
model = YOLO("yolo11s.pt") # or a newer release; s/m sizes are good baselines
model.train(data="dataset/data.yaml", epochs=100, imgsz=640, batch=16,
patience=20, seed=0, project="runs", name="baseline")
metrics = model.val(data="dataset/data.yaml", split="test")
print(metrics.box.map50, metrics.box.map) # [email protected] and [email protected]:0.95
CLI equivalent: yolo detect train data=dataset/data.yaml model=yolo11s.pt epochs=100 imgsz=640.
| Lever | When |
|---|---|
Larger imgsz (960-1280) | Small objects |
| Larger model size | Underfitting: train and val metrics both low |
| More augmentation or more data | Overfitting: train high, val low |
| Class weights or oversampling | Rare classes with low recall |
| Tiling large images | Very high-resolution inputs (aerial, documents, PCB) |
Track runs, configs, and dataset versions (see mlops-tracking).
6. Evaluate and choose thresholds
- Report per-class precision, recall, [email protected], and [email protected]:0.95 on the test split from deployment conditions; also slice by camera, time of day, and object size.
- Choose the confidence threshold per class from the PR curve to hit the written targets; the default 0.25 is arbitrary.
- For counting, also measure count error per image; for tracking, use MOT metrics.
7. Error analysis
Look at the 50 worst images: false positives, false negatives, and mislocalized boxes.
| Pattern | Fix |
|---|---|
| Misses at night or in glare | Collect and label those conditions |
| Look-alike false positives | Add hard negatives |
| Same error on mislabeled examples | Fix labels; re-review the class guideline |
| Small objects missed | Higher resolution, tiling, or a model with better small-object performance |
| Confusion between two classes | Merge them or sharpen definitions with examples |
Retrain, re-evaluate on the same test set, and keep a changelog of dataset versions.
8. Deploy
yolo export model=runs/baseline/weights/best.pt format=onnx imgsz=640
- Export to ONNX (CPU, ONNX Runtime), TensorRT (NVIDIA), CoreML, or OpenVINO; verify that exported-model metrics match PyTorch on the test set (FP16/INT8 can shift results).
- Benchmark end to end: decode, resize, inference, post-processing, and drawing. Preprocessing often costs as much as inference.
- OpenCV reads images as BGR; most models expect RGB. Mismatched channel order quietly drops accuracy.
- Monitor in production: sample frames for human review, track detection-rate drift per camera, and alert on cameras that go dark or move.
Gotchas
- Random splits of video frames leak; models memorize backgrounds and score near-perfect on validation.
- EXIF orientation: phone photos may be rotated relative to labels; normalize orientation before labeling and inference.
- Letterboxing vs stretching at inference must match training preprocessing.
- [email protected] hides poor localization; check [email protected]:0.95 when box precision matters (measurements, cropping).
- Faces and license plates are personal data in many jurisdictions; blur or restrict storage and document the legal basis.
Output contract
**Task:** <cls/det/seg/pose>, classes <list>, targets <per-class>
**Data:** <n images>, sources <cameras/sites>, split by <source>, checker clean <yes>
**Model:** <name/size>, imgsz <n>, run <id>, license <AGPL/enterprise/BSD>
**Test metrics:** mAP50 <x>, mAP50-95 <y>, per-class P/R <table>, thresholds <per class>
**Deployment:** <format> on <hardware>, <ms or FPS> end to end
**Next data to collect:** <from error analysis> Bundled files
Loaded only when a step in the skill needs them. Review them here before installing.
scripts/check_yolo_dataset.py
#!/usr/bin/env python3
"""Validate a YOLO-format detection/segmentation dataset (stdlib only).
Usage:
python3 check_yolo_dataset.py <dataset_dir> [--data data.yaml] [--min-size 0.0005]
[--splits train,val,test] [--show 10]
Expected layout (Ultralytics convention):
<dataset_dir>/data.yaml
<dataset_dir>/images/<split>/*.jpg|png|...
<dataset_dir>/labels/<split>/*.txt one object per line: class cx cy w h (normalized)
segmentation lines (class x1 y1 x2 y2 ...) are also checked
data.yaml: reads `names:` as a list ([a, b] or "- a" lines) or a mapping ("0: a" lines),
or `nc:` alone. Only this simple subset of YAML is parsed.
Errors (exit 1): label without image, class id out of range, malformed line,
coordinates outside [0, 1], zero-area boxes, identical image in more than one split.
Warnings: image without label file (allowed for negatives but reported), empty label files,
tiny boxes (w*h < --min-size), duplicate boxes within one file, classes with no instances.
Exits 2 if the dataset cannot be read.
"""
import argparse
import hashlib
import os
import re
import sys
from collections import Counter, defaultdict
IMG_EXT = {".jpg", ".jpeg", ".png", ".bmp", ".webp", ".tif", ".tiff"}
def parse_names(path):
if not os.path.exists(path):
return None
lines = open(path, encoding="utf-8").read().splitlines()
names, nc, in_names = None, None, False
for raw in lines:
line = raw.split("#", 1)[0].rstrip()
if not line.strip():
continue
m = re.match(r"^nc:\s*(\d+)", line)
if m:
nc = int(m.group(1))
in_names = False
continue
m = re.match(r"^names:\s*(.*)$", line)
if m:
rest = m.group(1).strip()
if rest.startswith("["):
names = [x.strip().strip("'\"") for x in rest.strip("[]").split(",") if x.strip()]
in_names = False
else:
names, in_names = {}, True
continue
if in_names and raw.startswith((" ", "\t")):
item = line.strip()
m = re.match(r"^(\d+):\s*(.+)$", item)
if m:
names[int(m.group(1))] = m.group(2).strip().strip("'\"")
elif item.startswith("- "):
names[len(names)] = item[2:].strip().strip("'\"")
continue
in_names = False
if isinstance(names, dict):
names = [names[k] for k in sorted(names)]
if names is None and nc is not None:
names = [str(i) for i in range(nc)]
return names
def sha1(path):
h = hashlib.sha1()
with open(path, "rb") as fh:
for block in iter(lambda: fh.read(1 << 20), b""):
h.update(block)
return h.hexdigest()
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("dataset")
ap.add_argument("--data", default="data.yaml")
ap.add_argument("--min-size", type=float, default=0.0005, help="warn when w*h is below this")
ap.add_argument("--splits", default="train,val,test")
ap.add_argument("--show", type=int, default=10)
args = ap.parse_args()
root = args.dataset
if not os.path.isdir(os.path.join(root, "images")):
print(f"error: {root}/images not found", file=sys.stderr)
sys.exit(2)
names = parse_names(os.path.join(root, args.data))
if names is None:
print(f"warning: could not read class names from {args.data}; class range not checked")
nclass = len(names) if names else None
errors, warnings = defaultdict(list), defaultdict(list)
per_split = {}
hashes = defaultdict(set)
for split in [s for s in args.splits.split(",") if s]:
img_dir = os.path.join(root, "images", split)
lbl_dir = os.path.join(root, "labels", split)
if not os.path.isdir(img_dir):
continue
imgs = {os.path.splitext(f)[0]: f for f in os.listdir(img_dir)
if os.path.splitext(f)[1].lower() in IMG_EXT}
lbls = {os.path.splitext(f)[0]: f for f in os.listdir(lbl_dir)
if f.endswith(".txt")} if os.path.isdir(lbl_dir) else {}
counts = Counter()
n_boxes = 0
for stem, f in imgs.items():
hashes[sha1(os.path.join(img_dir, f))].add(f"{split}/{f}")
if stem not in lbls:
warnings["image without label (negative?)"].append(f"{split}/{f}")
for stem, f in lbls.items():
rel = f"{split}/{f}"
if stem not in imgs:
errors["label without image"].append(rel)
lines = [l for l in open(os.path.join(lbl_dir, f), encoding="utf-8").read().splitlines() if l.strip()]
if not lines:
warnings["empty label file"].append(rel)
seen = set()
for n, line in enumerate(lines, 1):
parts = line.split()
where = f"{rel}:{n}"
try:
cls = int(parts[0])
vals = [float(x) for x in parts[1:]]
except (ValueError, IndexError):
errors["malformed line"].append(where)
continue
if len(vals) < 4 or (len(vals) > 4 and len(vals) % 2):
errors["malformed line"].append(where)
continue
if cls < 0 or (nclass is not None and cls >= nclass):
errors["class id out of range"].append(f"{where} (class {cls})")
if any(v < 0 or v > 1 for v in vals):
errors["coordinate outside [0,1]"].append(where)
if len(vals) == 4:
w, h = vals[2], vals[3]
if w <= 0 or h <= 0:
errors["zero-area box"].append(where)
elif w * h < args.min_size:
warnings["tiny box"].append(where)
key = (cls, tuple(round(v, 4) for v in vals))
if key in seen:
warnings["duplicate box in file"].append(where)
seen.add(key)
counts[cls] += 1
n_boxes += 1
per_split[split] = (len(imgs), len(lbls), n_boxes, counts)
for h, files in hashes.items():
splits = {f.split("/", 1)[0] for f in files}
if len(splits) > 1:
errors["identical image in multiple splits"].append(" == ".join(sorted(files)))
if not per_split:
print("error: no split directories found under images/", file=sys.stderr)
sys.exit(2)
print(f"classes: {nclass if nclass is not None else '?'}" + (f" {names}" if names and len(names) <= 20 else ""))
print(f"\n{'split':8} {'images':>7} {'labels':>7} {'boxes':>7}")
for split, (ni, nl, nb, _) in per_split.items():
print(f"{split:8} {ni:7} {nl:7} {nb:7}")
all_classes = range(nclass) if nclass else sorted({c for *_, cnt in per_split.values() for c in cnt})
print("\ninstances per class:")
header = " " + f"{'class':20}" + "".join(f"{s:>8}" for s in per_split)
print(header)
for c in all_classes:
label = names[c] if names and c < len(names) else str(c)
row = [per_split[s][3].get(c, 0) for s in per_split]
print(" " + f"{label[:20]:20}" + "".join(f"{v:8}" for v in row))
if per_split.get("train") and per_split["train"][3].get(c, 0) == 0:
warnings["class with no training instances"].append(label)
for title, bucket in (("ERRORS", errors), ("WARNINGS", warnings)):
if bucket:
print(f"\n{title}:")
for kind, items in bucket.items():
print(f" {kind}: {len(items)}")
for it in items[: args.show]:
print(f" - {it}")
if errors:
print(f"\nFAIL: {sum(len(v) for v in errors.values())} error(s)")
sys.exit(1)
print("\nPASS: no errors")
if __name__ == "__main__":
main()
About this skill
Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published , last updated . MIT licensed.