AI Engineering No. 239

Computer Vision skill for Claude Code

Detectors that hold up on your real cameras. Claude approaches vision work data-first. It picks the simplest task type that answers the question, writes labeling guidelines, splits by camera or video so frames cannot leak, validates YOLO datasets with a bundled checker, trains from pretrained weights, chooses per-class thresholds from precision-recall targets, and verifies exported ONNX or TensorRT models still match on the target hardware.

  • namecomputer-vision
  • size185 lines · ≈2,050 tokens
  • files2
  • licenseMIT
  • updated

What Claude sees before loading it

This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.

description: Builds computer vision features with OpenCV, Ultralytics YOLO, and torchvision - choosing classification, detection, segmentation, or pose; dataset collection and labeling in CVAT or Label Studio with written guidelines; YOLO-format validation and leakage checks; training from pretrained weights; mAP, per-class recall, and threshold tuning; error analysis; and export to ONNX or TensorRT for deployment. Use when detecting or counting objects, segmenting images, inspecting defects, reading camera feeds, preparing or auditing a labeled dataset, or when a vision model scores well in validation but fails in production.

Try asking

  • Train a detector that counts forklifts and people in our warehouse camera footage
  • Our defect model has 0.97 mAP in validation but misses defects on the line. Why?
  • Audit this YOLO dataset for label errors and train/val leakage before we train

Use it when

  • Detecting, counting, segmenting, or classifying objects in images or video
  • Preparing, labeling, or auditing a vision dataset
  • A vision model validates well but fails in production
  • Exporting a model for edge or GPU deployment

Skip it when

  • You need text from scanned pages — use ocr-scans
  • You want to generate images rather than analyze them — use image-generation

How to check the result

Run scripts/check_yolo_dataset.py until it passes, then confirm per-class precision and recall on a test split from separate cameras or days, and re-check metrics after export.

The common mistake it prevents

Splitting video frames randomly into train and validation, so near-identical frames leak and validation mAP looks far better than production.

~/.claude/skills/computer-vision/SKILL.md

--- name: computer-vision description: Builds computer vision features with OpenCV, Ultralytics YOLO, and torchvision - choosing classification, detection, segmentation, or pose;… license: MIT ---

A vision model is ready when it meets per-class targets on a test set drawn from the deployment cameras and conditions, the test set shares no near-duplicate frames with training, the confidence threshold was chosen from a precision-recall trade-off the business signed off on, and inference meets the latency budget on the target hardware. A high mAP on a random split of video frames is usually leakage, not skill.

Ground rules

  • Start from the simplest task type that answers the question: classification before detection, detection before segmentation.
  • Data beats architecture. Most gains come from fixing labels and adding the hard cases.
  • Split by source (camera, site, day, video), never by random frame.
  • Write labeling guidelines before labeling; inconsistent boxes cap accuracy.
  • Check licenses: Ultralytics YOLO code and weights are AGPL-3.0 unless you hold an enterprise license; torchvision models are BSD-style but their weights carry the dataset’s terms.

Choose the task and tool

QuestionTaskDefault tool
Is there a defect / what kind of scene is this?Classificationtimm or torchvision backbone fine-tune; YOLO -cls
Where and how many objects?DetectionUltralytics YOLO (YOLO11 or newer, e.g. YOLO26); torchvision Faster R-CNN when AGPL is a problem
Exact object shape or area?Instance / semantic segmentationYOLO -seg; SAM-family models for label assistance
Body or object keypoints?PoseYOLO -pose
Measure, align, threshold, simple shapes in controlled lightingClassical CVOpenCV (thresholding, contours, template matching)
Open-vocabulary “find any X” without trainingZero-shot detection or a vision LLMPrototype only; measure before shipping

Classical OpenCV beats deep learning when lighting, camera, and parts are fixed (factory inspection with a jig). Try it first in those settings.

Workflow

- [ ] 1. Define classes, metrics, and the latency budget
- [ ] 2. Collect images from deployment conditions
- [ ] 3. Write labeling guidelines; label; review a sample
- [ ] 4. Split by source; validate the dataset
- [ ] 5. Train a baseline from pretrained weights
- [ ] 6. Evaluate per class; pick thresholds
- [ ] 7. Error analysis; add data; retrain
- [ ] 8. Export, benchmark on target hardware, monitor

1. Definition

  • Classes: name, definition, and edge cases (partially visible? reflections? toy versions?).
  • Metric per class: recall matters for safety (missed person), precision for alerts that page humans. Write targets, e.g. “person recall >= 0.95 at precision >= 0.85”.
  • Latency and hardware: “20 FPS per stream on Jetson Orin” or “200 ms per image on CPU”.

2. Collect

  • Capture from the real cameras, resolutions, angles, lighting (night, glare, rain), and backgrounds. Public datasets help pretraining but rarely match deployment.
  • Sample video frames sparsely (e.g. one frame per second or on scene change); adjacent frames are near-duplicates.
  • Include negatives: images with no objects, and look-alikes that should not trigger.
  • Typical starting volume: 100-300 labeled instances per class for fine-tuning a pretrained detector; more for rare, small, or variable classes.

3. Label

  • Tools: CVAT or Label Studio; use model-assisted pre-labeling once a baseline exists, with humans correcting.
  • Guidelines with pictures: tight boxes, include occluded parts or not, minimum object size to label, what to do with crowds, truncated objects at the image border.
  • Review 10% of labels by a second person; measure box agreement (IoU >= 0.7 counts as agreement) and fix the guideline where they disagree.

4. Split and validate

YOLO layout:

dataset/
  data.yaml          # path, train, val, test, names
  images/{train,val,test}/*.jpg
  labels/{train,val,test}/*.txt   # one line per object: class cx cy w h (normalized 0-1)

Run the bundled checker before every training run:

python3 scripts/check_yolo_dataset.py dataset/

It reads class names from data.yaml (simple YAML subset, no PyYAML needed), pairs images with labels, flags missing or orphan label files, class IDs out of range, coordinates outside (0, 1], zero-area or tiny boxes, duplicate boxes, prints instances per class per split, and finds identical images across splits by content hash. It exits 1 on errors. Content-hash matching only catches exact copies; for video, split by video or camera so near-duplicates cannot cross splits.

5. Train a baseline

from ultralytics import YOLO

model = YOLO("yolo11s.pt")                  # or a newer release; s/m sizes are good baselines
model.train(data="dataset/data.yaml", epochs=100, imgsz=640, batch=16,
            patience=20, seed=0, project="runs", name="baseline")
metrics = model.val(data="dataset/data.yaml", split="test")
print(metrics.box.map50, metrics.box.map)   # [email protected] and [email protected]:0.95

CLI equivalent: yolo detect train data=dataset/data.yaml model=yolo11s.pt epochs=100 imgsz=640.

LeverWhen
Larger imgsz (960-1280)Small objects
Larger model sizeUnderfitting: train and val metrics both low
More augmentation or more dataOverfitting: train high, val low
Class weights or oversamplingRare classes with low recall
Tiling large imagesVery high-resolution inputs (aerial, documents, PCB)

Track runs, configs, and dataset versions (see mlops-tracking).

6. Evaluate and choose thresholds

  • Report per-class precision, recall, [email protected], and [email protected]:0.95 on the test split from deployment conditions; also slice by camera, time of day, and object size.
  • Choose the confidence threshold per class from the PR curve to hit the written targets; the default 0.25 is arbitrary.
  • For counting, also measure count error per image; for tracking, use MOT metrics.

7. Error analysis

Look at the 50 worst images: false positives, false negatives, and mislocalized boxes.

PatternFix
Misses at night or in glareCollect and label those conditions
Look-alike false positivesAdd hard negatives
Same error on mislabeled examplesFix labels; re-review the class guideline
Small objects missedHigher resolution, tiling, or a model with better small-object performance
Confusion between two classesMerge them or sharpen definitions with examples

Retrain, re-evaluate on the same test set, and keep a changelog of dataset versions.

8. Deploy

yolo export model=runs/baseline/weights/best.pt format=onnx imgsz=640
  • Export to ONNX (CPU, ONNX Runtime), TensorRT (NVIDIA), CoreML, or OpenVINO; verify that exported-model metrics match PyTorch on the test set (FP16/INT8 can shift results).
  • Benchmark end to end: decode, resize, inference, post-processing, and drawing. Preprocessing often costs as much as inference.
  • OpenCV reads images as BGR; most models expect RGB. Mismatched channel order quietly drops accuracy.
  • Monitor in production: sample frames for human review, track detection-rate drift per camera, and alert on cameras that go dark or move.

Gotchas

  • Random splits of video frames leak; models memorize backgrounds and score near-perfect on validation.
  • EXIF orientation: phone photos may be rotated relative to labels; normalize orientation before labeling and inference.
  • Letterboxing vs stretching at inference must match training preprocessing.
  • [email protected] hides poor localization; check [email protected]:0.95 when box precision matters (measurements, cropping).
  • Faces and license plates are personal data in many jurisdictions; blur or restrict storage and document the legal basis.

Output contract

**Task:** <cls/det/seg/pose>, classes <list>, targets <per-class>
**Data:** <n images>, sources <cameras/sites>, split by <source>, checker clean <yes>
**Model:** <name/size>, imgsz <n>, run <id>, license <AGPL/enterprise/BSD>
**Test metrics:** mAP50 <x>, mAP50-95 <y>, per-class P/R <table>, thresholds <per class>
**Deployment:** <format> on <hardware>, <ms or FPS> end to end
**Next data to collect:** <from error analysis>

Bundled files

Loaded only when a step in the skill needs them. Review them here before installing.

scripts/check_yolo_dataset.py 192 lines · 7.9 KB
#!/usr/bin/env python3
"""Validate a YOLO-format detection/segmentation dataset (stdlib only).

Usage:
    python3 check_yolo_dataset.py <dataset_dir> [--data data.yaml] [--min-size 0.0005]
                                  [--splits train,val,test] [--show 10]

Expected layout (Ultralytics convention):
    <dataset_dir>/data.yaml
    <dataset_dir>/images/<split>/*.jpg|png|...
    <dataset_dir>/labels/<split>/*.txt     one object per line: class cx cy w h  (normalized)
                                            segmentation lines (class x1 y1 x2 y2 ...) are also checked
data.yaml: reads `names:` as a list ([a, b] or "- a" lines) or a mapping ("0: a" lines),
or `nc:` alone. Only this simple subset of YAML is parsed.

Errors (exit 1): label without image, class id out of range, malformed line,
coordinates outside [0, 1], zero-area boxes, identical image in more than one split.
Warnings: image without label file (allowed for negatives but reported), empty label files,
tiny boxes (w*h < --min-size), duplicate boxes within one file, classes with no instances.
Exits 2 if the dataset cannot be read.
"""
import argparse
import hashlib
import os
import re
import sys
from collections import Counter, defaultdict

IMG_EXT = {".jpg", ".jpeg", ".png", ".bmp", ".webp", ".tif", ".tiff"}


def parse_names(path):
    if not os.path.exists(path):
        return None
    lines = open(path, encoding="utf-8").read().splitlines()
    names, nc, in_names = None, None, False
    for raw in lines:
        line = raw.split("#", 1)[0].rstrip()
        if not line.strip():
            continue
        m = re.match(r"^nc:\s*(\d+)", line)
        if m:
            nc = int(m.group(1))
            in_names = False
            continue
        m = re.match(r"^names:\s*(.*)$", line)
        if m:
            rest = m.group(1).strip()
            if rest.startswith("["):
                names = [x.strip().strip("'\"") for x in rest.strip("[]").split(",") if x.strip()]
                in_names = False
            else:
                names, in_names = {}, True
            continue
        if in_names and raw.startswith((" ", "\t")):
            item = line.strip()
            m = re.match(r"^(\d+):\s*(.+)$", item)
            if m:
                names[int(m.group(1))] = m.group(2).strip().strip("'\"")
            elif item.startswith("- "):
                names[len(names)] = item[2:].strip().strip("'\"")
            continue
        in_names = False
    if isinstance(names, dict):
        names = [names[k] for k in sorted(names)]
    if names is None and nc is not None:
        names = [str(i) for i in range(nc)]
    return names


def sha1(path):
    h = hashlib.sha1()
    with open(path, "rb") as fh:
        for block in iter(lambda: fh.read(1 << 20), b""):
            h.update(block)
    return h.hexdigest()


def main():
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("dataset")
    ap.add_argument("--data", default="data.yaml")
    ap.add_argument("--min-size", type=float, default=0.0005, help="warn when w*h is below this")
    ap.add_argument("--splits", default="train,val,test")
    ap.add_argument("--show", type=int, default=10)
    args = ap.parse_args()

    root = args.dataset
    if not os.path.isdir(os.path.join(root, "images")):
        print(f"error: {root}/images not found", file=sys.stderr)
        sys.exit(2)
    names = parse_names(os.path.join(root, args.data))
    if names is None:
        print(f"warning: could not read class names from {args.data}; class range not checked")
    nclass = len(names) if names else None

    errors, warnings = defaultdict(list), defaultdict(list)
    per_split = {}
    hashes = defaultdict(set)

    for split in [s for s in args.splits.split(",") if s]:
        img_dir = os.path.join(root, "images", split)
        lbl_dir = os.path.join(root, "labels", split)
        if not os.path.isdir(img_dir):
            continue
        imgs = {os.path.splitext(f)[0]: f for f in os.listdir(img_dir)
                if os.path.splitext(f)[1].lower() in IMG_EXT}
        lbls = {os.path.splitext(f)[0]: f for f in os.listdir(lbl_dir)
                if f.endswith(".txt")} if os.path.isdir(lbl_dir) else {}
        counts = Counter()
        n_boxes = 0
        for stem, f in imgs.items():
            hashes[sha1(os.path.join(img_dir, f))].add(f"{split}/{f}")
            if stem not in lbls:
                warnings["image without label (negative?)"].append(f"{split}/{f}")
        for stem, f in lbls.items():
            rel = f"{split}/{f}"
            if stem not in imgs:
                errors["label without image"].append(rel)
            lines = [l for l in open(os.path.join(lbl_dir, f), encoding="utf-8").read().splitlines() if l.strip()]
            if not lines:
                warnings["empty label file"].append(rel)
            seen = set()
            for n, line in enumerate(lines, 1):
                parts = line.split()
                where = f"{rel}:{n}"
                try:
                    cls = int(parts[0])
                    vals = [float(x) for x in parts[1:]]
                except (ValueError, IndexError):
                    errors["malformed line"].append(where)
                    continue
                if len(vals) < 4 or (len(vals) > 4 and len(vals) % 2):
                    errors["malformed line"].append(where)
                    continue
                if cls < 0 or (nclass is not None and cls >= nclass):
                    errors["class id out of range"].append(f"{where} (class {cls})")
                if any(v < 0 or v > 1 for v in vals):
                    errors["coordinate outside [0,1]"].append(where)
                if len(vals) == 4:
                    w, h = vals[2], vals[3]
                    if w <= 0 or h <= 0:
                        errors["zero-area box"].append(where)
                    elif w * h < args.min_size:
                        warnings["tiny box"].append(where)
                key = (cls, tuple(round(v, 4) for v in vals))
                if key in seen:
                    warnings["duplicate box in file"].append(where)
                seen.add(key)
                counts[cls] += 1
                n_boxes += 1
        per_split[split] = (len(imgs), len(lbls), n_boxes, counts)

    for h, files in hashes.items():
        splits = {f.split("/", 1)[0] for f in files}
        if len(splits) > 1:
            errors["identical image in multiple splits"].append(" == ".join(sorted(files)))

    if not per_split:
        print("error: no split directories found under images/", file=sys.stderr)
        sys.exit(2)

    print(f"classes: {nclass if nclass is not None else '?'}" + (f" {names}" if names and len(names) <= 20 else ""))
    print(f"\n{'split':8} {'images':>7} {'labels':>7} {'boxes':>7}")
    for split, (ni, nl, nb, _) in per_split.items():
        print(f"{split:8} {ni:7} {nl:7} {nb:7}")
    all_classes = range(nclass) if nclass else sorted({c for *_, cnt in per_split.values() for c in cnt})
    print("\ninstances per class:")
    header = "  " + f"{'class':20}" + "".join(f"{s:>8}" for s in per_split)
    print(header)
    for c in all_classes:
        label = names[c] if names and c < len(names) else str(c)
        row = [per_split[s][3].get(c, 0) for s in per_split]
        print("  " + f"{label[:20]:20}" + "".join(f"{v:8}" for v in row))
        if per_split.get("train") and per_split["train"][3].get(c, 0) == 0:
            warnings["class with no training instances"].append(label)

    for title, bucket in (("ERRORS", errors), ("WARNINGS", warnings)):
        if bucket:
            print(f"\n{title}:")
            for kind, items in bucket.items():
                print(f"  {kind}: {len(items)}")
                for it in items[: args.show]:
                    print(f"    - {it}")
    if errors:
        print(f"\nFAIL: {sum(len(v) for v in errors.values())} error(s)")
        sys.exit(1)
    print("\nPASS: no errors")


if __name__ == "__main__":
    main()

About this skill

Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published , last updated . MIT licensed.

esc